hasan's blog (বল্গ)

when we started using semantic repository, we had only one lucene index to make our content search able,
later we came up with another integration with one php based service aawaj.

on aawaj service they had  more than 150,000 items to index. we tried with our current release 0.5.1 to index all contents but we ended with extremely performance outage. later we released another version 0.5.2, where we added queued request handling and threw index optimization over an restful service uri – /rest/service/optimize/

here is the simple benchmark report -

version – 0.5.1 – first 100 items  ended in – 13.611 seconds.
version – 0.5.2 – first 100 items  ended in – 5.6152 seconds.

the change is really different and significant, later today we had anoter import on our repository, interestingly it took 1 hour to index 150,000 items. which was bit surprising since we were unable to do it with 0.5.1

actually we added single thread executor which keeps everything in queue and execute one by one. so we could remove  synchronized method.

here is an example code -

private final Executor mIndexTaskExecutor =

Executors.newSingleThreadExecutor();

public void addDocument(final Document pDocument) {

mIndexTaskExecutor.execute(

new Runnable() {

public void run() {

getLuceneIndexTemplate().addDocument(pDocument);

}

}

);

}

i had a tough time to configure nginx on my debian production environment.
the recent stable release from nginx is 0.6.x but on debian repository it was 0.4.x, so i had to build it from the source and install it.

since i had an old 0.4.x instance of nginx, installation wasn’t as smooth as i was expecting. here i would try to show how i have resolved those broken issues and made my way to run nginx to reverse proxy my backend mongrel instances.

i took several attemtps to remove the existing 0.4.x instance of nginx but i failed.
i used “aptitude remove nginx” i ended with the following error -

Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages will be REMOVED:
nginx
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0B of archives.
After unpacking 582kB disk space will be freed.
Do you want to continue [Y/n]?
(Reading database … 78227 files and directories currently installed.)
Removing nginx …
Stopping nginx: nginx.
Stopping nginx: invoke-rc.d: initscript nginx, action “stop” failed.
dpkg: error processing nginx (–remove):
subprocess pre-removal script returned error exit status 1
Starting nginx: nginx.
Errors were encountered while processing:
nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

though this is not my real error code but it has similarity, i took it from the following url -

http://sudhanshuraheja.com/2007/09/remove-nginx-from-ubuntu-fiesty-fawn.html

this blog author had some suggestion, but that wasn’t working for me, so i tried in different way -
i executed “sudo apt-get build-dep nginx” i found this tips from one of the blog comments
the comment author explained in this way -

“this should install everything required to build the package (compiler, headers/libs, packaging tools). Usually on a fresh install I do this to get everything required to build zope.Then issue “apt-get source nginx” (you need deb-src sources in /etc/apt/sources.list). This will download nginx sources (original tarball, diff, and uncompressed sources with patches applied). Just cd in source dir, make your modifications and use “dpkg-buildpackage -rfakeroot -b” (this requires fakeroot package). In parent directory you should get new deb files ready to install, with start/stops scripts and your patches. Just take care of package update that will surely remove your nginx version.”

if you want service script to initiate nginx on startup follow the link -
http://blog.labratz.net/articles/2006/10/03/rails-deployment-apache-lighttpd-nginx-mongrel-cluster

best wishes,

those who doesn’t know about JETTY RUNNER:
JETTY RUNNER is a standalone swing based application which is used to bundle java ee based application along with jetty container. it comes with simple web app configuration xml file and global properties manager through a simple properties file.

actually i have been using this project for my own development solution, so i belief this project will become a great strengthen feature gradually.

JETTY RUNNER is now running on max OSX, i have removed system try support in new tag v-0.2, soon i will release *.dmg package for mac osx. here are few screen snaps -

figure – 1: server console main window

figure – 2: global properties editor

change logs -
1. removed system tray support
2. removed default jmx configuration
3. added “start.sh” to launch JETTY RUNNER on *nix based platform where ruby script is installed.
here are few screen snaps, which i have taken from the newly added ruby script! -

figure – 3: newly added jetty runner on ruby

figure – 4: newly added jetty runner implementation in ruby

this script really great , at least i like it

for last couple of days, i was struggle with rails deployment related issue, so far i was happy with mongrel backed and apache reverse proxy front configuration.

so last night, i found mongrel is surprisingly slow for serving static content, like javascripts, images, and styles files. so later i set apache for serving static files. now it seems out performing

anyway, i was wondering about mongrel and how does it works in production deployment. as we know (who use java backed application server in production environment), any java backed application container are configurable. where we can define the maximum limit of thread pool size.
but i didn’t find this type of thing over mongrel, later few of the following urls, really helped me to understand how mongrel  works in production environment. the concept is very simple.

every mongrel worker is a standalone instance of mongrel and every mongrel instances are listening to a specific port. so i have to setup a reverse proxy or a load balancer which will be used to pick up the right mongrel instance.  for all session affinity shake, there must be some solutions, this is what i have to figure out later. (hopefully after getting my laptop)
here are the resource links -

pound – used as load balancer and reverse proxy service.

jared’s guide how to install pound

Grails vs rails fight, though it doesn’t worth rather than few configuration learning.

hope these will be helpful for my future reminding

after a long while i got myself on linked in site, as i got “Server spy” installed on my firefox browser, so i was getting the server name from the browser status menu.

anyway, today i found something different, actually “Server spy” said, linked in is using “Apache-Coyote/1.1″, as far i can remind, linked in was using sun application server previously.

i think it is really interesting, what led linked in to move over tomcat backed or tomcat based servlet container instead of sun application server?

anyway, if i find any info i would love to feed my blog

one of our escenic colleagues (Simen L. Haagenrud) just noticed a bottleneck with tomcat 5.x.x series containers. so we had a details digging to let find the reason.

as he mentioned it was because of recurring invocation of “getAttribute”, and tomcat 5.x.x series is using synchornization block inside “getAttribute” method, where lock is kept on “attributes” variable.
so i had a quick look on the source code of the 3 recent series -

1. 5.0.28
http://svn.apache.org/repos/asf/tomcat/container/tags/tc5.0.x/TOMCAT_5_0_28/catalina/src/share/org/apache/catalina/core/ApplicationContext.java

2. 5.5.24 (the last version from 5.5.x series)
http://svn.apache.org/repos/asf/tomcat/container/tags/tc5.5.x/TOMCAT_5_5_24/catalina/src/share/org/apache/catalina/core/ApplicationContext.java

3. 6.0.x (trunk)
http://svn.apache.org/repos/asf/tomcat/tc6.0.x/trunk/java/org/apache/catalina/core/ApplicationContext.java

so the code says 6.0.x series came with the fixes, the fix is the replacement of HashMap with ConcurrentHashMap.

in his case, recurring invocation of “getAttribute” was nearly reduced 50% server request handling capability.
best wishes,

hi web crasher,
(it was prepared for http://groups.yahoo.com/group/jpgroup)
(this post is cross published over http://www.somewherein.net/blog/?p=592)
i belief many JPGroup members are utilizing cache in most of the case. in my short post i will show few caching plan and tools.

today, i will highlight on web cache related stuff, let’s imagine we can’t modify our source code, no application level cache. this cache will be maintained separately.

My intended readers are those who didn’t get in touch with web cache related stuffs and those who heard these terms, but didn’t give a try on their project yet.

scene:
Let’s assume “www.nai.com” a site which you have developed, it is a community driven portal, you are expecting 2000 users who will browse this site with in few couple of hours.

so one fine releasing date, you got kicked off, you didn’t expect that you will get around 5000 users with in couple of hours. god damn… you have two severs with distributed session. both are in low latency network. you found your server “A1″ is wasting 50% processor on each request. Obviously lot of tuning over your code is left.

let’s assume, your community site have many aggregation stuffs. for example, the recent user list, the most rated stuffs, most hated stuffs, tag cloud, most comment shit, many things.

obviously, you can’t reduce any feature otherwise it won’t have any difference with other community sites.

so what was happened then, let me continue the story. while your server “A1” was frequently utilizing 50% processor, and your another server “A2” was almost utilizing the same processor, you have heavy database processing and many calculation and many stuffs.

So you have hired a consultant to resolve this problem, now let me pretend what he will do with your project.

preconditions:
You had few preconditions:

1. you can’t change the source or anything won’t be happened over development related stuff.
2. you have been assigned a shortest time, so the consultant has to do it before you dead.
Solutions:
Your hired guy may be happy because of your preconditions, because few stuffs that already helping on this kind of problem.

# Reverse proxy:
[visitor] [proxy server with cache] [application server]
Now how it will help to solve your problem:

1. Visitor won’t get access to your application server,
2. All heavily functional pages will be served from proxy server’s cache
3. proxy server to application connection is in low latency network
What is the changes?
The following changes will boost your community site’s performance.

1. the heavily database or complex logic related functions won’t be executed every time.
2. visitor won’t get 100% up to date data on the fly, perhaps it will take bit longer time based on your cache expire rule.
3. it doesn’t require any code related change.
4. many products and open source stuffs are available on the shelf
5. not suitable for heavily dynamic site, where every second is a change.
6. proxy server can be utilized for load balancing, later you can add more servers.
7. common solution apache with mod_proxy or separate dedicated proxy server. Squid, varnished (which is a modified version of squid proxy server, introduced by linpro)

Have a look on the wiki page, for more proxy server product.
http://en.wikipedia.org/wiki/Reverse_proxy

# OSCache servlet filter
This is another alternative solution, it will require few changes on “web.xml” file.
You have to put a OSCache filter servlet, which will cache each and every unique request. For expiring all these cache you can set up the cache expire event.

OSCache servlet filter is easy to use, and you can write your cache expire rule on a java class. As it is not the change on your core system, so no conflict with your preconditions.

# Oracle web cache
Oracle has reverse proxy type solution which they have labeled as “Web cache”
It can scale your community site, it had coupling with oracle application server. So if your application server is oracle$ application server, you can give a try.
That’s all for today.
Best wishes for those who died 21^st February to bring our mother tongue.

just few minutes back i have configured hasan.we4tech.com my blog domain.

thank you NetNeuron for configuring DNS stuff..

following code where required on httpd.conf to map hasan.we4tech.com on somewhereindhaka.net server

ServerAdmin hasan@somewherein.net
DocumentRoot d:/apache2triad/htdocs/hasan_blog
ServerName    hasan.we4tech.com
ErrorLog    logs/hasan.we4tech.com-error_log
CustomLog    logs/hasan.we4tech.com-access_log common

ServerAdmin hasan@somewherein.net
DocumentRoot d:/apache2triad/htdocs/
ServerName    somewhereindhaka.net
ErrorLog    logs/hasan.we4tech.com-error_log
CustomLog    logs/hasan.we4tech.com-access_log common