In the first part of this article, which can be found in the previous JayView, the monitoring tools of the Sun JDK 1.5 was presented. These monitoring tools can be a good start for looking at the current status, and behavior over time, for an application. While these tools can tell you some parts of the doings of an application, finding a problem is sometimes more effective using the troubleshooting tools. This second part of the article will present some of these tools.

Tools overview

The list below contains all tools presented in this article, where the first four can
be found in part one.

Note: A known issue for OSX require that some of the tools is run as root, which can be accomplished by putting ‘sudo ‘ before each command.

Troubleshooting tools

The troubleshooting tools provided by Sun Microsystems in the JDK bin directory are to be considered experimental. They come with no support and they might harm your system. Although being in an experimental state, the tools can be quite useful when looking for problems.

Some of the tools can connect to a locally or remotely running Java process, others operate on a core dump. This dump can be created by the JVM, by specifying Sun Hotspot specific flags such as -XX:+HeapDumpOnCtrlBreak or
-XX:+HeapDumpOnOutOfMemoryError. These flags should be appended to the java command just like system properties:

> java -XX:+HeapDumpOnOutOfMemoryError -jar application.jar

The core dump can also be created with jmap and the syntax is described below. Be aware that the jmap of JDK 1.5 have some problems creating this core dump, sometimes it creates a corrupted file. This specific issue with jmap is fixed in Java 6.

jinfo

The jinfo tool connects to a running local process, a JVM core dump or a remote
process. The purpose of the tool is to display the VM flags used and the values of
the system properties for the process in question. The process id, or VMID, can be found using the jps command. Here is an example connecting to a running process with process id 1649:

> jinfo -flags 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87
-Duser.property=somevalue 

In the example above, you can see that the Java application was started using the flag ‘user.property’. To see the system properties, use the -sysprops flag:

> jinfo -sysprops 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
sun.boot.library.path = /System/Library/Frameworks/JavaVM.framework/
Versions/1.5.0/Libraries
java.vm.version = 1.5.0_07-87
java.vm.vendor = ”Apple Computer, Inc.”
java.vendor.url = http://apple.com/
path.separator = :
   ...

The jinfo command can give a quick hint about what a java process can see from its environment.

jstack

The jstack command prints out the execution stack trace for threads for a process in the local JVM, a remote JVM or in a core dump. It also prints out thread deadlocks detected by the Hotspot VM, something that can come in handy for
solving threading issues. The command for connecting to a local process looks like:

> jstack 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_07-87 

Thread t@4099: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
(Interpreted frame)
...
Thread t@4611: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=194
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=254
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=313
 - sun.nio.cs.StreamDecoder$CharsetSD.readBytes() @bci=135, line=411
 - sun.nio.cs.StreamDecoder$CharsetSD.implRead(char[], int, int) @bci=112,
line=453
 - sun.nio.cs.StreamDecoder.read(char[], int, int) @bci=180, line=183
 - java.io.InputStreamReader.read(char[], int, int) @bci=7, line=167
 - java.io.BufferedReader.fill() @bci=145, line=136
 - java.io.BufferedReader.readLine(boolean) @bci=44, line=299
 - java.io.BufferedReader.readLine() @bci=2, line=362
 - se.jayway.jconsole.test.Application.run() @bci=29, line=24
 - se.jayway.jconsole.test.Application.main(java.lang.String[]) @bci=9,
line=14

In the example above, we can see that the Application.main( ) method is spending time in FileInputStream.readBytes( ).

jmap

This versatile command prints out information using the following flags:

• - heap prints out heap information, including the different memory pools
• - heap:format=b dumps the heap in a binary format
• - histo prints class histogram information of the object heap
• - permstat prints information class loader-wise about data in the permanent generation memory pool. The permanent generation memory pool contains class meta data and the String literals.

> jmap -heap 1649
Attaching to process ID 1649, please wait...
...
using thread-local object allocation.
Mark Sweep Compact GC 

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize      = 67108864 (64.0MB)
NewSize          = 655360 (0.625MB)
...

In the example above only the first part of the heap information is printed, there’s a lot more to see.

> jmap -histo 1649
Object Histogram:
Size    Count   Class description
-------------------------------------------------------
194408  1025    char[]
102840  160     byte[]
16032   668     java.lang.String
13208   297     java.lang.Object[]
8208    27      * ObjArrayKlassKlass
...

Above, the first part of the class histogram is printed for the process with VMID 1649.

> jmap -heap:format=b 1649
Attaching to process ID 1649, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_11-b03
Unknown oop at 0x889ec790
Oop’s class is null
heap written to heap.bin 

The heap is dumped for the process with VMID 1649 in the example above, and the result is a file called heap.bin.

The jmap -heap and jstat with a suitable garbage collection flag can help with finding memory issues for object instances. Jmap -histo and jmap -permstat can be used for finding memory leaks in permanent generation memory space, e.g. when there are too many classes loaded by class loaders that never get garbage collected.

The jmap -heap:format=b flag has issues with the heap dump sometimes being corrupted, an issue which is fixed for Java 6. When the command is working the heap dump can be analyzed using tools described above and with a tool called jhat, which is bundled with Java 6 SDK. It is also possible to debug the heap dump using the java debugger, called jdb, which also can be found in the bin directory.

I recommend taking a look at jhat, which is a heap analysis tool that processes heap dumps. When invoking jhat it starts with analyzing all objects found in the heap core dump. After the analysis is finished a web server is launched. Use an ordinary web browser to connect to the web server to get access to the object browsing functions available.

One example of a function is the one that makes it possible to filter out weak
references between objects, so that objects that are not eligible for garbage collection can be found. It is also possible to use OQL (Object Query Language) where you can use an SQL-like syntax to get object set results, such as the number of instances for a specific object type, the value for all String objects etc.

The jhat tool is available in Java 6 but can be downloaded from Sun for Java 1.5.

To conclude this article, the monitoring and troubleshooting tools found in the bin directory for the JDK provided by Sun can be a good start when finding bugs in your java code. Use them, you might already have them on your computer.

Originally published in JayView.

A possible scenario

You have started up an application, let’s say it is a web application which is running in an application server. After a while you notice that the application is behaving oddly. You cannot really put your finger on the problem, but sometimes it looks like the web pages are not served as they should. The application also feels slower than you expect.
First thing could be to check out the log files, but as you all know, the debug level
is set to INFO or WARNING because the application is in production:

10:10:40.542 INFOStarting  application...
10:12:21.650 INFOAll  service started.
10:14:22.145 INFOApplication  booted successfully. 

What conclusion do you draw from this output? The application has started and it
looks like nothing serious has detected in the code that is worth mentioning in the
log. Is it time to start guessing or just ignore the gut feeling you have? Not quite yet,
there are some help to be found not far away, and it will not cost you a dime.

Bring out the tools

In your Java SDK 1.5 (or higher) “bin” directory, there are tools that can be quite us-
able for poking around in the Java processes. There are different number of tools for
different platforms and the number of available functions per tool is also depending
on the distribution. Sun Solaris and Linux Java distributions have been blessed with
the most number of tools and functions, Mac OSX and Windows have the least.
Also, the tools are getting constantly improved for newer releases of Java.

Here is a comprehensive list of the tools mentioned in this article and on which
platforms they exist:

This article consists of two parts, and this first part goes through the available moni-
toring tools in the Sun Java 1.5 SDK bin directory.

Monitoring tools

jps

To see which java processes that are currently running on a machine use the jps
command:

> jps
1649 Application
2533 Jps 

The resulting output lists the VMID (Virtual Machine ID) and the Java process
name. To get extended information, append the -v flag to the jps command:

> jps v
1649  Application
2534 Jps
-Dapplication.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/
Home
-Xms8m 

jstat

This command gives performance statistic information from the JVM for a Java
process. There are a number of statistic options available, which is displayed using
the -options switch:

> jstat -options
-class
-compiler
-gc
-gccapacity
...

To print the statistics for one of the options above for a process, the VMID has to
be specified after the option. The VMID is the same id as listed with the jps com-
mand:

> jstat -gcutil  1649
S0  S1  E  O  P  YGC  YGCT  FGC  FGCT  GCT
0,00  0,00  78,32  0,00  0,24  0  0,000  0  0,000  0,000

Using the jstat command, it is possible to create scripts that periodically checks the
status for e.g. the number of loaded classes or the garbage collection utilization. In
the example above, the garbage collection has taken place mostly in Eden memory
space. Eden space is Sun Hotspot specific memory space for shortlived objects.
Information about memory pools can be found at http://java.sun.com.

jstatd

This command starts up a jstat daemon on a machine and enables remote monitor-
ing. To startup the daemon on the computer which should be monitored, the port
number where the remote clients can connect to has to be specified:

> jstatd -p  12345

After that, it is possible to use both jps and jstat remotely by specifying the host-
name and port:

> jps example.computer.org:12345
2617 Jps
2611 Jstatd
1649 Application

The above example lists the processes on the ‘example.computer.org’ host using the
port 12345. Presented in the output, you can see the jstatd is also a java process.

> jstat  -gcutil  1649@example.computer.org:12345
S0  S1  E  O  P  YGC  YGCT  FGC  FGCT  GCT
0,00  0,00  78,32 0,00  0,24  0  0,000  0  0,000  0,000

Above we can see the same example for process 1649 as in the previous command,
but this time listed for host ‘example.computer.org’ using port 12345.

jconsole

On client computers, or servers with a graphical interface, it is possible to monitor
the JVM status by using a tool called jconsole. The jconsole tool visualizes services
that exposes a JMX (Java Management Extension) interface, and Java 1.5 and later
contains JMX beans for exposing the data defined in the java.lang.management
package.

To be able to monitor java applications using jconsole in Java 1.5, the application
has to be started with the system property -Dcom.sun.management.jmxremote.
Example:

> java -Dcom. sun.management.jmxremote jar  testapp.jar 

When the application has started, the jconsole can be initiated:

> jconsole 

The connection console displays all java processes that has JMX enabled. After con-
necting to the Java process, jconsole provides the following tabs:

• Summary Tab
• Memory Tab – The different memory pools can be inspected
• Thread Tab – The total, peak and currently live threads are displayed
• Classes Tab – The number of classes are displayed
• Mbeans Tab – Gives access to the JMX beans in the management package
• VM Tab – Displays information for the VM where the application is running in

Below are example screenshots for the memory and thread tab:

The jconsole application can be a tool to start with when looking at application
behavior over time, such as memory leaks or unusual thread behavior.

Using the Java monitoring and management tools you can get a better understanding
of the runtime behavior of your application. Sometimes this information is enough
for finding and fixing a problem.

This concludes the first part of the article. In the next issue of JayView we will look
into the troubleshooting tools.

Originally published in JayView.