We have a stand alone Java application (called MyApp here) that runs an embedded Jetty HTTP server that we wanted to monitor.

It was surprisingly easy to get basic information from New Relic. Simply download the new relic agent zip file and expand it and it is ready for use with:

java -javaagent:newrelic/newrelic.jar -cp newrelic/newrelic.jar -jar MyApp.jar

After running MyApp for a few minutes the following information was available under Monitoring / Transactions after logging in on newrelic.com:

Here we can see how much time the HTTP requests take. Looking at the image almost all time is spent in AsyncProcessing. Why is that? Well the MyApp application processes the requests asynchronously (as introduced in Servlet 3.0) leaving the actual work to dedicated threads. It would therefore make sense to monitor what the worker threads do. This can be achieved in the following way:

• First create an annotation that can be used to tell New Relic about custom metrics:

  package com.myapp.NewRelicTrace;
  
  import java.lang.annotation.*;
  
  @Target(ElementType.METHOD)
  @Retention(RetentionPolicy.RUNTIME)
  public @interface NewRelicTrace {
      String metricName() default "";
  
      boolean dispatcher() default false;
  
      String tracerFactoryName() default "";
  }
  

• Then tell New Relic to use the new annotation by adding it to the newrelic.yml configuration file (in the common section):

    class_transformer:
      trace_annotation_class_name: com.myapp.NewRelicTrace
  

• Now we start by adding the annotation to the place where the asynchronous processing of each request is dispatched (e.g. where the worker threads start processing the requests). We want these to be shown as transactions in New Relic:

    @NewRelicTrace(metricName = "Custom/Worker/request", dispatcher = true)
    public WorkResult process(WorkRequest request){
  	...
    }
  

  • metricName is the name used in New Relic and has to use the format: Custom/Category/Label. Note that you have to start with Custom/ or New Relic will not pick up the metric.
  • dispatcher needs to be true where a new Transaction should be started in New Relic.
• Add the annotation to the different steps that are interesting to measure. In our case looking up user information in DynamoDb:

    @NewRelicTrace(metricName = "Custom/UserDb/LookUp")
    public UserInfo lookUpUser(String id){
       ...
    }
  

  …and calling a 3rd party system:

    @NewRelicTrace(metricName = "Custom/ThirdParty/Request")
    String make3rdPartyRequest() throws IOException {
       ...
    }
  

When running the application we can now see a new Transaction type called Custom in New Relic:

Here we can see what happens during the AsyncProcessing we saw in the first figure. Looking at the breakdown we see how the time is spent:

• 64% – Calls to the 3rd Party system
• 19% – HTTP communication with DynamoDB (identified as External by New Relic)
• 10% – Unclassified parts of the Worker.process(…) method
• 7% – Handling of user look up (in addition to the time spent on DynamoDB communication).

As you can see it is easy to choose what parts to measure and you get a good overview of what is happening.