hasan's blog (বল্গ)

hi,
as some of you know, semantic repository version 0.5.2 has some problem with IOException (too many files open)
which was stopping repository to perform further search. thats what was reason to get empty search result.
after searching a while, we found the problem (which was also mentioned in lucene FAQ document).

by default bash shell allow limited files to open, since we had many index files, lucene has to open them up during performing search.
so when it exceeds the limit of 1025 files (which is default on our production environment). our application threw the following exception -
Caused by: java.io.FileNotFoundException: /var/indexes/ads-index/_gm.tvd (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:506)
at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:536)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:445)
at org.apache.lucene.index.TermVectorsReader.(TermVectorsReader.java:70)

after increasing this limit to 10,000 we didn’t find such problem exists. we had to apply “ulimit -n 10000″ command on bash shell to make it works.

TODO: possible bug, probable our system is opening up several index searcher.

when we started using semantic repository, we had only one lucene index to make our content search able,
later we came up with another integration with one php based service aawaj.

on aawaj service they had  more than 150,000 items to index. we tried with our current release 0.5.1 to index all contents but we ended with extremely performance outage. later we released another version 0.5.2, where we added queued request handling and threw index optimization over an restful service uri – /rest/service/optimize/

here is the simple benchmark report -

version – 0.5.1 – first 100 items  ended in – 13.611 seconds.
version – 0.5.2 – first 100 items  ended in – 5.6152 seconds.

the change is really different and significant, later today we had anoter import on our repository, interestingly it took 1 hour to index 150,000 items. which was bit surprising since we were unable to do it with 0.5.1

actually we added single thread executor which keeps everything in queue and execute one by one. so we could remove  synchronized method.

here is an example code -

private final Executor mIndexTaskExecutor =

Executors.newSingleThreadExecutor();

public void addDocument(final Document pDocument) {

mIndexTaskExecutor.execute(

new Runnable() {

public void run() {

getLuceneIndexTemplate().addDocument(pDocument);

}

}

);

}

i had a problem with spring framework, as you know while we are defining dependency over the xml documents, it becomes less changeable unless you know about my recent post about ext-util namespace handler and if you know any other alternatives.

before digging inside the problem better i clarify my context -

i have been working with one of my projects, where i have a several configurations which suppose to be different in different deployment environment, for example – to make my service avail, user has to define his database, his content index directory, server port, ip address and few more.

all these configurations were kept within the spring bean declaration scope. though few of them were moved to the properties file through ext-util namespace handler.

previously to make those configuration changeable i kept those files under WEB-INF/classes so it overrides the default bundled resource.

but still it wasn’t user friendly, since my user has to change one xml document to add more index services also he has to edit different configurations from java properties file.

again when i started bundling jetty with my whole application i found it is tricker to keep those xml outside of application, since i don’t wanna make my user confused with lot of xml files. and i have to keep few xml out of  my classpath context, so spring has to load my configuration from my “config/index-configuration/” directory.

though by default spring can do it over “file:///absolute/path/to/file.xml” since spring is using java URL class to locate this resource.  but in my case i can’t put any aboslute path.
so i found a simple way out there to support relative path reference.

previously i had the following xml document -

now i have replaced them with the following change -

if you observe closely you will find “extfile:///” protocol reference, which is not standard url protocol. so to make it work i used the following bunch of code and configuration -

1. i have created a class “Handler” under the “com.ideabase.repository.core.protocols.extfile” package.
this package and class name is conventional, though you can change them lets sun explain about it.

2. i wrote the following code -

public class Handler extends URLStreamHandler {
protected URLConnection openConnection(final URL pURL) throws IOException {
final String path = pURL.getPath();
final URL resourceUrl;
if (path.startsWith(“/./”)) {
resourceUrl = new File(path.substring(1, path.length())).toURL();
} else {
throw new RuntimeException(“Unsupported url schema. currently you can ” +
“use only extfile:///./path/to/somewhere/in”);
}
return resourceUrl.openConnection();
}
}
3. i registered my protocol package through the system property -

System.setProperty(“java.protocol.handler.pkgs”, “com.ideabase.repository.core.protocols”);

now have a look on the main applicationContext.xml

if you observe closely again you could find the difference, on the normal case i have used relative path reference. so it must be from classloader or if it is loaded through file system based resource locator it will be located from the file system.

but now i can use cross context (relative file system and class loader path reference) resource file path reference. you can see i have imported “indexer-beans.xml” from the application relative path “config/index-configuration/…” where “data-access-layer-beans.xml” is loaded from the class loader.
now have a quick look on my configuration directory, to get an idea how i have improved configuration ability.

hope this will help you to solve similar problem.

within spring container i was getting problem with managing multi level configuration.
for example while i am developing i got a specific filesystem path, database configuration and so on.
this configuration are mostly get changed while i an deploying them on the test or production environment.

this type of scenario is common on most of the projects. managing different scoped configuration is one of the bothering stuff. so i was always thinking some simpler way with in spring container.

so i wrote “ext-util” namespace handler, which is locating property from multi scoped resource.
lets give you a more details use case -

in my test environment i have a property “index.directory=/Users/…/index”
in my test server i use “index.directory=/var/index”
in my production server i use “index.directory=/nfs/index”

here i have a bean which need this property.

you can see the hardcoded configuration. which is not easy to manage.
thats why here my “ext-util” goes with the simplest solution -

now my property resource is locating from “test” scope.

you might think, this “scope=’test’” is again hardcoded stuff, so i also support the following value -
scope=”sys:env”, now you have to define “-Denv=test” while you running your application, and spring bean will find the right property.

so what ever you put over your spring are no longer hardcoded as long as you locate them through “ext-util” which is stands for “extended utility”.

here is my attached source code – (i just extracted them from my on going project, thats why no build script or prebuilt package or maven artifact is bundled)

how to use -
include the source file with in your project
keep “META-INF” while you building your jar or keep them inside your classpath.
change your spring configuration -

xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xmlns:lang=”http://www.springframework.org/schema/lang”
xmlns:aop=”http://www.springframework.org/schema/aop”
xmlns:ext-util=”http://dev.somewherein.net/schema/ext-util”
xsi:schemaLocation=”http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://dev.somewherein.net/schema/ext-util http://dev.somewherein.net/resources/schemas/ext-util-0.1.xsd“/>
basic properties of ext-util -

1. from-property – which property you are looking for.

2. or-default – define default value if no property is found from default scope (system properties) this value is returned.

3. scope – define scope name, by default whatever you set except “sys:” prefixed string, will search for “classpath:”. also you can use “sys:abc” where abc is located from system properties.

4. path-prefix – you can define scope properties path prefix, this prefix can be “file:///, classpath:”. this value also can be prefixed by “sys:abc” where abc is located from system properties.

[download source code]
[download missing xsd file]

best wishes,

those who doesn’t know about JETTY RUNNER:
JETTY RUNNER is a standalone swing based application which is used to bundle java ee based application along with jetty container. it comes with simple web app configuration xml file and global properties manager through a simple properties file.

actually i have been using this project for my own development solution, so i belief this project will become a great strengthen feature gradually.

JETTY RUNNER is now running on max OSX, i have removed system try support in new tag v-0.2, soon i will release *.dmg package for mac osx. here are few screen snaps -

figure – 1: server console main window

figure – 2: global properties editor

change logs -
1. removed system tray support
2. removed default jmx configuration
3. added “start.sh” to launch JETTY RUNNER on *nix based platform where ruby script is installed.
here are few screen snaps, which i have taken from the newly added ruby script! -

figure – 3: newly added jetty runner on ruby

figure – 4: newly added jetty runner implementation in ruby

this script really great , at least i like it

one of our escenic colleagues (Simen L. Haagenrud) just noticed a bottleneck with tomcat 5.x.x series containers. so we had a details digging to let find the reason.

as he mentioned it was because of recurring invocation of “getAttribute”, and tomcat 5.x.x series is using synchornization block inside “getAttribute” method, where lock is kept on “attributes” variable.
so i had a quick look on the source code of the 3 recent series -

1. 5.0.28
http://svn.apache.org/repos/asf/tomcat/container/tags/tc5.0.x/TOMCAT_5_0_28/catalina/src/share/org/apache/catalina/core/ApplicationContext.java

2. 5.5.24 (the last version from 5.5.x series)
http://svn.apache.org/repos/asf/tomcat/container/tags/tc5.5.x/TOMCAT_5_5_24/catalina/src/share/org/apache/catalina/core/ApplicationContext.java

3. 6.0.x (trunk)
http://svn.apache.org/repos/asf/tomcat/tc6.0.x/trunk/java/org/apache/catalina/core/ApplicationContext.java

so the code says 6.0.x series came with the fixes, the fix is the replacement of HashMap with ConcurrentHashMap.

in his case, recurring invocation of “getAttribute” was nearly reduced 50% server request handling capability.
best wishes,

Here is open search specification – http://www.opensearch.org/Specifications/OpenSearch/1.1

Let’s enable your browser to detect your search engine automatically; using open search servlet you can generate open search configuration file on the fly.

Download information is here –

Binary library

Source maven project

Binary war file (includes example usages)

Usages:

1. Copy “open-search-x.jar” file on your WEB-INF/lib/ directory
2. Edit web.xml file.
3. add these entries -

< servlet >
    < servlet-name >openSearch< /servlet-name >
    < servlet-class >com.we4tech.openSearch.servlet.OpenSearchServlet< /servlet-class >
    < !-- URL matching pattern -- >
    < init-param >
      < param-name >url-match< /param-name >
      < param-value >.*/(.+).osd< /param-value >
      < !--< param-value >.*/search/(.+)< /param-value >-- >
    < /init-param >
    < !-- map open search configuration with url -- >
    < init-param >
      < param-name >open-search< /param-name >
      < param-value >open-search.properties< /param-value >
    < /init-param >
    < init-param >
      < param-name >somewherein-blog< /param-name >
      < param-value >somewherein-blog.properties< /param-value >
    < /init-param >
  < /servlet >

  < servlet-mapping >
    < servlet-name >openSearch< /servlet-name >
    < !--< url-pattern >/search/*< /url-pattern >-- >
    < url-pattern >*.osd< /url-pattern >
  < /servlet-mapping >

4. Now place configuration properties file with in your WEB-INF/classes/
5. for example: “open-search.properties”

# Default open-search. properties
shortName=colorful moment.
description=somewhere in blog
contact=hasan@somewherein.net

url.1=text/html,get,http://www.somewherein.net/blog/index.php?s={searchTerms}&submit=
image.1=16,16,image/x-icon,http://www.searchmash.com/favicon.ico

6. Congratulations you have completed all steps

Sample usages on your web site can be found here –

http://www.opensearch.org/Specifications/OpenSearch/1.1#Autodiscovery_in_HTML.2FXHTML

Best wishes,

hi,
here is a great collection on servlet jsp performance related stuffs.
http://www.javaperformancetuning.com/tips/j2ee_srvlt.shtml

best wishes,

hi,
on my server i found some strange problem. it was actually related with tag. the problem was with the url rewriting, which was performed by struts tag library. struts tag library was creating an invalid URL.
for example i wrote the following code:
< img src=”my-image.gif” />

as i have only wrapped “${urlPrefix}” with in html:rewrite tag. so struts tag library wrote the following output.

which is totally a wrong url.

so how i resolved it?
very simple just fixing my weired code. now it looks like the following -
< img src=”" />

the generated output looks like -

here is a regex which may help you to find out the error tags:
“(.+)”

yeap it is working now

RESTful url may contain the content type with in the url string. For example: http://host/action/xml/bookmark. Here “xml” is content type, this content type can be changed to “text, json” or etc…

So following urls are valid too. http://host/action/text/bookmark, http://host/action/json/bookmark etc…

these days I am playing with “Stripes” web framework a lot. Those who didn’t get in touch with “Stripes” have a look on the following URL – http://stripes.mc4j.org, hope you will enjoy it.

stripes has fantastic flexibility, it enables a lot of customization. Now my tutorial will demonstrate how you can enable this url pattern, where your single “ActionBean” will be invoked on the following url patterns:

http://host/action/xml/bookmark, http://host/action/text/bookmark, http://host/action/json/bookmark etc…

My target audience:
Who has already been introduced with Stripes or those who are interested on stripes framework.

Routing plan:
Request Dispatch
[visitor] —  > http://host/action/xml/bookmark — > [web server] — > [Stripe servlet] — > [Custom Action resolver] — > [remove user defined content type “xml” and retrieve action bean]

Response Dispatch
[clean url (after removing content type)] — > http://host/action/bookmark — > [BookmarkActionBean] — > [Resolution] — >

Pre requisite:
Stripes servlet is configured and web context is up and running.

ActionBean code:
*  create an ActionBean, for example “BookmarkActionBean”

@UrlBinding(“/action/bookmark”)

public class BookmarkAction implements ActionBean {

    private final Log LOG = LogFactory.getLog(getClass());

    private ActionBeanContext mActionBeanContext;

    private String mContentType = “text/xml”;

    public void setContext(ActionBeanContext actionBeanContext) {

        mActionBeanContext = actionBeanContext;

        // retrieve user requested content type constant for example “text,json, xml”

        // convert consts to mime type “text, text/xml, application/JSON”

        mContentType = URLHelper.getMimeType(URLHelper.getContentType(getContext().getRequest().getPathInfo()));

    }

    public ActionBeanContext getContext() {

        return mActionBeanContext;

    }

    @DefaultHandler

    public Resolution save() {

        if (LOG.isDebugEnabled())

            LOG.debug( “storing bookmark object on the data store, ” +

                       “response content type is requested – ” + mContentType );

        return new StreamingResolution( mContentType, new StringReader(“”) );

    }

}

Description:
Here i have set our action on “/action/bookmark” url, but our user requested url pattern is “/action//actionBean”

Look on the “setContext” method, I have added an extra line

mContentType = URLHelper.getMimeType(URLHelper.getContentType(getContext().getRequest().getPathInfo()));

now have a look on the URLHelper class.

public static String getContentType( String pRequestedUrl ) {

        String contentType = CACHED_URL_CONTENT_TYPE.get(pRequestedUrl);

        if (contentType == null) {

            contentType = DEFAULT_CONTENT_TYPE;

            Perl5Util util = new Perl5Util();

            if (util.match( REGEX_ACTION_CONTENT_TYPE, pRequestedUrl ))

                contentType = util.group(1)+”";

            CACHED_URL_CONTENT_TYPE.put( pRequestedUrl, contentType );

        }

        return contentType;

    }

This method is responsible to retrieve user requested content type constants.
for example “text, xml, json”

public static String getMimeType( String pContentType ) {

        // convert content type to MIME Type

        if (pContentType.equalsIgnoreCase(CONTENT_TYPE_CONST_XML))

            pContentType = CONTENT_TYPE_XML;

        else if (pContentType.equalsIgnoreCase(CONTENT_TYPE_CONST_JSON))

            pContentType = CONTENT_TYPE_JSON;

        return pContentType;

    }

This method is also converting constant to MIME content type.
For example “xml –> text/xml”

Now return to my base action bean. I have added the following method, which is defined as default handler.

@DefaultHandler
public Resolution save() {

        ….

        return new StreamingResolution( mContentType, new StringReader(“”) );

    }

Here “mContentType” is repopulate when “setContext” is invoked.

ActionResolver code:
This is my action resolver.

public class NBookmarkSystemActionResolver extends NameBasedActionResolver {

     // ………   

    @Override

    public ActionBean getActionBean(ActionBeanContext pActionBeanContext, String pRequestedUrl) throws StripesServletException {

        String cleanUrl = URLHelper.getCleanUrl(pRequestedUrl);

        if (LOG.isDebugEnabled())

            LOG.debug( “user requested url – ” + pRequestedUrl + ” clean url – ” + cleanUrl);

        return super.getActionBean(actionBeanContext, cleanUrl);

    }

}

Now let’s have a look on “getCleanUrl” method.

public static String getCleanUrl( String pRequestedUrl ) {

        return pRequestedUrl.replaceAll(“/”+getContentType(pRequestedUrl),”");

    }

Custom ActionResolver configuration:

        Stripes Filter

        StripesFilter

        net.sourceforge.stripes.controller.StripesFilter

       

            ActionResolver.UrlFilters

            /WEB-INF/classes/

       

       

            ActionResolver.PackageFilters

            com.we4tech.nBookmarkSystem.service.webService/*

       

       

            ActionResolver.Class

            com.we4tech.nBookmarkSystem.configuration.NBookmarkSystemActionResolver

       

   

That’s all for today,

Hope you will enjoy stripes. Best of luck…