lucene based semantic-repository 0.5.2: major performance improvement now 24,000 items imported in 19 minutes R

when we started using semantic repository, we had only one lucene index to make our content search able,
later we came up with another integration with one php based service aawaj.

on aawaj service they had  more than 150,000 items to index. we tried with our current release 0.5.1 to index all contents but we ended with extremely performance outage. later we released another version 0.5.2, where we added queued request handling and threw index optimization over an restful service uri - /rest/service/optimize/

here is the simple benchmark report -

version - 0.5.1 - first 100 items  ended in - 13.611 seconds.
version - 0.5.2 - first 100 items  ended in - 5.6152 seconds.

the change is really different and significant, later today we had anoter import on our repository, interestingly it took 1 hour to index 150,000 items. which was bit surprising since we were unable to do it with 0.5.1

actually we added single thread executor which keeps everything in queue and execute one by one. so we could remove  synchronized method.

here is an example code -

private final Executor mIndexTaskExecutor =

Executors.newSingleThreadExecutor();

public void addDocument(final Document pDocument) {

mIndexTaskExecutor.execute(

new Runnable() {

public void run() {

getLuceneIndexTemplate().addDocument(pDocument);

}

}

);

}

[source]

simple fragment cache implementation on ruby on rails R

i was getting serious performance problem with one of my projects. so i came up with a simple fragment cache implementation on ruby on rails.

after implementing this stuff, i replaced “render(:partial => …)” with the following method -

render_from_cache_or_render(:cache_key =>”cache key”, :cache_expire_after => ConstantHelper::TAG_CLOUD_EXPIRED_IN, # minutes :partial => “….”)

let’s have a look on my implementation -

def render_from_cache_or_render(p_args)

return render(p_args) if true == p_args[:cache_off]

# check from cache
cache_key = p_args[:cache_key]
cached_content = CacheService.get_cache(cache_key)

if not cached_content.nil? and not cached_content.empty?
return cached_content
else
content = render(p_args)
# cache expire time if defined
cache_expire_time_in_minutes = p_args[:cache_expire_after] || 60
CacheService.add_cache(cache_key, cache_expire_time_in_minutes, content)
return content
end
end

actually, my implemented “CacheService” class is simply storing all cache in a hash map.
when some cache was requested for peek, cache expiry was checked before returning the cached value.

for CacheService implementation look at the bottom of my post.

anyway, after implementing and utilizing this stuff, i gained 70+ requests capability per second. fyi, before applying cache it was around 10 per second.

module Cache
class Item
attr_accessor :key, :expire_time, :content, :created_on

def initialize(p_key, p_expire_time, p_content)
@key = p_key
@expire_time = p_expire_time * 60 # in minutes
@content = p_content
@created_on = Time.now
end
end
end

class CacheService
@@CACHES = {}
@@CACHE_EXPIRE_TIMES = {}

def self.add_cache(p_key, p_expire_time, p_content)
cache_item = Cache::Item.new(p_key, p_expire_time, p_content)
@@CACHES[p_key.to_sym] = cache_item
end

def self.get_cache(p_key)
# load content from cache
cached_content = @@CACHES[p_key.to_sym]
return nil if cached_content.nil?

# verify cache validity
return cached_content.content if not expired?(cached_content)
return nil
end

private
def self.expired?(p_cache)
# find time difference
time_difference = Time.now - p_cache.created_on
return true if time_difference > p_cache.expire_time
end
end

best wishes,