Investigating OOM and other JVM issues

When working with a large scale Lucene/Solr installation, users sometimes run into memory issues or garbage collection performance problems. Its not a frequent occurrence in my experience, but just like life, it happens. Whenever I run into this sort of thing, I’ve come to rely on a variety of free tools to investigate the problem. There are many other free tools I have tried, but over time, I have found the following most useful:

jmap http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jmap.html

This is a great little tool that lets you get live memory info from running java apps. jmap -histo is a very useful command. It shows a nice little histogram of the heap for a live process. Its often a lot easier to use this tool see whats taking up all that RAM than to hook up a profiler, especially since jmap comes with most JDKs these days.

VisualGC http://java.sun.com/performance/jvmstat/visualgc.html

Awesome tool that takes advantage of the jstat instrumentation thats part of all JVMs these days to show you a visual representation of a live applications memory footprint. This lets you track your heap and watch as the eden space fills up and overflows, etc. This is a very cool tool to play around with, and is very useful for getting a feel for how garbage collection in your application works.

jstatd http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstatd.html

You can use this to use most of these tools remotely. Somewhat legacy now, as I think the latest Java releases allow you to simply use a command line parameter when you start the java app for remote connections.

jconsole http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

jconsole is a powerful remote monitoring and management tool for the Java platform. If you havn’t used it yourself yet, you really owe it to yourself to spend some time with this tool. You can view a plethora of live stats and JVM information with this GUI tool. Java 1.5 and up comes with jconsole.

Netbeans Profiler http://www.netbeans.org/

The profiler that is included with Netbeans is top notch and very easy to use. I use eclipse, but I also generally have Netbeans installed as well for some of its little goodies, like the profiler. Netbeans itself has also been getting better and better, but it has not yet pulled me from eclipse. Netbeans has the best and easiest to use free profiler I have used though (Netbean’s profiler is also part of the next tool).

VisualVM https://visualvm.dev.java.net/

This is a great open source VM tool that sort of combines jconsole type capabilities with the Netbeans profiler and plugin framework. Its a very cool wrapper around a lot of existing Java tools, with practically unlimited extendability. Very interesting, and plenty useful.

Some of the jstat tools come with Java 1.5 distributions and up (depending on the dist), but if you are using Java 1.4, check out http://java.sun.com/performance/jvmstat/.

These tools are great for troubleshooting both RAM and Garbage Collection issues. To add to your garbage collection toolbox though, also check out: http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html

Trying the different garbage collection options available and monitoring with some of the above tools can be a very effective attack on poor GC performance.