OutOfMemory

From DBSight Full-Text Search Engine/Platform Wiki

Table of contents

Checking memory usage

Dashboard => "System Report"(upper right corner)=> "System Info", look for something like:

Total Memory 	11.02 MB
Free Memory 	1.40 MB
Max Memory 	63.56 MB

The "Max Memory" is the amount that's really available to java.

Searching Server

Setting Java Garbage Collection parameters like the following will help to better allocate memories for specific memory needs.

For DBSight 4.x

DBSight Search Process now is a separated operating system process. And is out side of the DBSight front end servlet.

You can adjust the memory setting under "Data Source"=>"Advanced Settings"=>"Serving Search Requests"=>"JVM Memory Size for Searching".

When the index is updating, there will be a short moment when a new search process is starting up, and the previous search process still serves search requests. So for better performance, be sure to have twice memory available for search process, to avoid memory swapped to disk.

For DBSight 3.x and lower

Resin

Set JVM parameters in this way

httpd.exe -J-Xmx256m -J-XX:NewRatio=1

To watch Garbage Collectoin at work, you can use

httpd.exe -J-Xmx256m -J-XX:NewRatio=1 -J-verbose:gc

Jetty

This is the Jetty server

java -Xmx256m -XX:NewRatio=1 -verbose:gc -jar start.jar

Genereate Java Heap Dump

You can add a JVM option as such:

-XX:+HeapDumpOnOutOfMemoryError

Some more details are here: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html

By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the -XX:HeapDumpPath= option. For example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.

You can also use jmap -dump option to obtain a heap dump at runtime, but that may not reveal the problem at that specific time. http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html

Indexing

Could be MySql JDBC driver memory bug

A MySQL's JDBC problem (see http://bugs.mysql.com/bug.php?id=7698). Even the latest release jdbc 3.1.18 has this problem. You may have to download the latest development version mysql-connector-java-3.2.0-alpha-bin.jar.

Just put the file to directory "WEB-INF/lib/ext/jdbc/".

  1. It's relative to the directory where dbsight is unpacked.
  2. It's OK to leave older versions of the same JDBC driver there. 
     DBSight will pick up the latest one automatically. 

JVM Memory in Advanced Setting

DBSight Indexing Process is a separated operating system process. And is out side of the DBSight front end servlet. It is also different from the DBSight Search Process.

You can adjust the memory setting under "Data Source"=>"Advanced Settings"=>"Serving Search Requests"=>"JVM Memory Size for Searching".

Lower "List Fetch Size"

DBSight first get the list of the documents. If the list is too large, OutOfMemory exception may appear. Lowering "List Fetch Size" will reduce the JDBC fetch batch size. So it will also help to reduce the required memory.

Breaking Main Query up into subsequent queries

Some users tried to get all contents by one single complicated query. Quite often, the query is inefficient. It's better to break the Main Query up into subsequent queries. This will ease database's resouces, JDBC connections, and memory requirements.