Forensic tools, part 2

Have you ever been in the situation that an application behaves in strange ways and by the time you find out, you do not have access to your development environment. In this article we are discussing ways to pin- point the problem with tools available in the Sun SDK bin directory.

In the first part of this article, which can be found in the previous JayView, the monitoring tools of the Sun JDK 1.5 was presented. These monitoring tools can be a good start for looking at the current status, and behavior over time, for an application. While these tools can tell you some parts of the doings of an application, finding a problem is sometimes more effective using the troubleshooting tools. This second part of the article will present some of these tools.

Tools overview

The list below contains all tools presented in this article, where the first four can
be found in part one.


Note: A known issue for OSX require that some of the tools is run as root, which can be accomplished by putting ‘sudo ‘ before each command.

Troubleshooting tools

The troubleshooting tools provided by Sun Microsystems in the JDK bin directory are to be considered experimental. They come with no support and they might harm your system. Although being in an experimental state, the tools can be quite useful when looking for problems.

Some of the tools can connect to a locally or remotely running Java process, others operate on a core dump. This dump can be created by the JVM, by specifying Sun Hotspot specific flags such as -XX:+HeapDumpOnCtrlBreak or
-XX:+HeapDumpOnOutOfMemoryError. These flags should be appended to the java command just like system properties:

> java -XX:+HeapDumpOnOutOfMemoryError -jar application.jar

The core dump can also be created with jmap and the syntax is described below. Be aware that the jmap of JDK 1.5 have some problems creating this core dump, sometimes it creates a corrupted file. This specific issue with jmap is fixed in Java 6.


The jinfo tool connects to a running local process, a JVM core dump or a remote
process. The purpose of the tool is to display the VM flags used and the values of
the system properties for the process in question. The process id, or VMID, can be found using the jps command. Here is an example connecting to a running process with process id 1649:

> jinfo -flags 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87 

In the example above, you can see that the Java application was started using the flag ‘’. To see the system properties, use the -sysprops flag:

> jinfo -sysprops 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87 = Java(TM) 2 Runtime Environment, Standard Edition
sun.boot.library.path = /System/Library/Frameworks/JavaVM.framework/
java.vm.version = 1.5.0_07-87
java.vm.vendor = ”Apple Computer, Inc.”
java.vendor.url =
path.separator = :

The jinfo command can give a quick hint about what a java process can see from its environment.


The jstack command prints out the execution stack trace for threads for a process in the local JVM, a remote JVM or in a core dump. It also prints out thread deadlocks detected by the Hotspot VM, something that can come in handy for
solving threading issues. The command for connecting to a local process looks like:

> jstack 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87

Thread t@4099: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
 - java.lang.ref.Finalizer$ @bci=3, line=159
(Interpreted frame)
Thread t@4611: (state = IN_NATIVE)
 -[], int, int) @bci=0
 -[], int, int) @bci=4, line=194
 -[], int, int) @bci=39, line=254
 -[], int, int) @bci=49, line=313
 - sun.nio.cs.StreamDecoder$CharsetSD.readBytes() @bci=135, line=411
 - sun.nio.cs.StreamDecoder$CharsetSD.implRead(char[], int, int) @bci=112,
 -[], int, int) @bci=180, line=183
 -[], int, int) @bci=7, line=167
 - @bci=145, line=136
 - @bci=44, line=299
 - @bci=2, line=362
 - @bci=29, line=24
 - se.jayway.jconsole.test.Application.main(java.lang.String[]) @bci=9,

In the example above, we can see that the Application.main( ) method is spending time in FileInputStream.readBytes( ).


This versatile command prints out information using the following flags:

  • – heap prints out heap information, including the different memory pools
  • – heap:format=b dumps the heap in a binary format
  • – histo prints class histogram information of the object heap
  • – permstat prints information class loader-wise about data in the permanent generation memory pool. The permanent generation memory pool contains class meta data and the String literals.
> jmap -heap 1649
Attaching to process ID 1649, please wait...
using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize      = 67108864 (64.0MB)
NewSize          = 655360 (0.625MB)

In the example above only the first part of the heap information is printed, there’s a lot more to see.

> jmap -histo 1649
Object Histogram:
Size    Count   Class description
194408  1025    char[]
102840  160     byte[]
16032   668     java.lang.String
13208   297     java.lang.Object[]
8208    27      * ObjArrayKlassKlass

Above, the first part of the class histogram is printed for the process with VMID 1649.

> jmap -heap:format=b 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_11-b03
Unknown oop at 0x889ec790
Oop’s class is null
heap written to heap.bin 

The heap is dumped for the process with VMID 1649 in the example above, and the result is a file called heap.bin.

The jmap -heap and jstat with a suitable garbage collection flag can help with finding memory issues for object instances. Jmap -histo and jmap -permstat can be used for finding memory leaks in permanent generation memory space, e.g. when there are too many classes loaded by class loaders that never get garbage collected.

The jmap -heap:format=b flag has issues with the heap dump sometimes being corrupted, an issue which is fixed for Java 6. When the command is working the heap dump can be analyzed using tools described above and with a tool called jhat, which is bundled with Java 6 SDK. It is also possible to debug the heap dump using the java debugger, called jdb, which also can be found in the bin directory.

I recommend taking a look at jhat, which is a heap analysis tool that processes heap dumps. When invoking jhat it starts with analyzing all objects found in the heap core dump. After the analysis is finished a web server is launched. Use an ordinary web browser to connect to the web server to get access to the object browsing functions available.

One example of a function is the one that makes it possible to filter out weak
references between objects, so that objects that are not eligible for garbage collection can be found. It is also possible to use OQL (Object Query Language) where you can use an SQL-like syntax to get object set results, such as the number of instances for a specific object type, the value for all String objects etc.

The jhat tool is available in Java 6 but can be downloaded from Sun for Java 1.5.

To conclude this article, the monitoring and troubleshooting tools found in the bin directory for the JDK provided by Sun can be a good start when finding bugs in your java code. Use them, you might already have them on your computer.

Originally published in JayView.

Leave a Reply