Asynchronous Spring Service

It is not unusual that your web service needs to communicate with another web service in order to serve its clients. In the old days, that would imply that an incoming request to your server would capture one servlet connection, and perform a blocking call to the remote service before it can send a response to the client. It works, but it does not scale very well if you have multiple concurrent clients. However, as of Servlet 3.0 we have support for asynchronous servlets (see my colleague Henrik’s blog post), and as of Servlet 3.1 it also supports non-blocking I/O which means that we can get a significantly increased throughput.

Problem

In this blog post, we will implement an asynchronous servlet that talks to a remote server using Spring. For demonstration purposes, GitHub’s repository search API will be used as the remote service. The problem is basically divided into three main classes:

  • A GitHubRepoListService has been implemented using the AsyncRestTemplate to communicate with a remote service.
  • The RepositoryListDtoAdapter is responsible for converting the response extracted from the AsyncRestTemplate from Pojos specific to the GitHub response to other Pojos more suitable for the clients of our service. As such, it also acts as an anti-corruption layer between GitHub and the rest of our service implementation.
  • Lastly, there is the AsyncController that is the entry point for our clients.

Of course, the service would not compile without some glue code and a build script. For reference, the project is available on GitHub.

Remote Server Communication

First, we implement a server that uses a AsyncRestTemplate to communicate with GitHub:

The response of the AsyncRestTemplate request is a ListenableFuture that eventually will reference a GitHubItems, which is then wrapped in a custom RepositoryListDtoAdapter that converts it to a ListenableFuture<RepoListDto>, see below.

Asynchronous Object Transformation

If you have not used the AsyncRestTemplate or if you are not used to asynchronous Java programming, you may find that transforming one object to another is more complex than one would expect. It may be tempting to just call gitHubItems.get() to fetch the response entity and then continue from there, but remember that we are dealing with a future which means that the get()method is a blocking call that waits for the future to complete before returning. A better solution is to use the ListenableFutureAdapter which can then do the transformation in an asynchronous way:

Controller

The frontend of our service is a controller that maps incoming requests, extracts the query parameter and delegate to the RepoListService to do the remote call as described earlier. We need to convert the ListableFuture to a DeferredResult before returning it to our clients in order to take advantage of Spring’s asynchronous request processing (see also Considerations). For this reason, we create a ListenableFutureCallback that we add to the ListenableFuture:

The ListenableFutureCallback interface has two methods that we need to implement. If all goes well, the onSuccess() method is called. Consequently, the response that is transformed by the RepositoryListDtoAdapter is wrapped in a ResponseEntity together with a success status, which in turn is passed to the DeferredResult instance. If something goes wrong, the onFailure() method is called, an empty response entity with a failure status code is created instead.

Glue Code

Since the project is implemented using Spring Boot, we use a simple Application class to configure Spring:

The rest of the code is basically plumbing to handle the serialization and deserialization of requests and responses. The JSON response from GitHub is mapped to GitHubItems by the AsyncRestTemplate:

Where GitHubItem is implemented like:

and the GitHubOwner is implemented as:

The response that is sent to the client consists of two Pojos. At the top level, there is a RepoListDto that contains information about the client’s search query, the total number of repositories at GitHub that matched the query and a list of RepoDtos that represents each repository:

Each individual GitHub repository is presented as a RepoDto:

Build script

The project was built using Maven and the following pom.xml:

Considerations

  • In this blog, I have showed how the AsyncRestTemplate to communicate with an external restful service. However, before using it in production I suggest that you configure the internals of the AsyncRestTemplate such as connection timeout, read timeout, max number of connections, max connections per route etc. The Apache HttpAsyncClient examples, this blog post and this question on Stack Overflow all provide good advice.
  • The asynchronous service will tolerate a much higher load than a synchronous solution would. However, from a client perspective the latency may be an issue since it is first calling your service, which in turn has to call the remote service and wait for its response, before a new response can be created and returned. To mitigate this problem, different cache solutions come to mind. Consider caching the responses from the remote services, caching the Pojos generated by our service and / or using HTTP headers to implement appropriate caching of the responses generated by our service in the HTTP layer.
  • Our asynchronous service is well prepared to handle big load, but is the remote service capable of handling the load you are delegating to it? The suggested cache solution may work if the remote service returns generic answers, but if the remote service returns unique responses for similar requests the cache solution will not do. And what happens if the remote service goes down? One way of handling such requirements, and to protect your server from cascading failures, is to wrap the remote call in a circuit breaker.
  • If you can migrate to Spring 4.1 that was released last week you do no longer need to convert your result to DeferredResult. Simply return the ListenableFuture from your controller (see the list of improvements).

Result

The response as returned from GitHub directly:

The response that is returned by our service:

Side note, jq is a convenient tool for JSON processing that you can install.

References

12 Comments

  1. Haytham

    Mattias, thank you for an interesting blog. In the Considerations section you say “However, from a client perspective the latency may be an issue since it is first calling your service, which in turn has to call the remote service and wait for its response, before a new response can be created and returned. ”

    You really lost me there! I thought the whole purpose of the AsyncService was to avoid that. To my understanding the client who called the server via the controller would receive the DeferredResult and call getResult on it and only then block until the result is available.

    • @Haytham: Sorry for being unclear, let me elaborate. First of all, in this blog by “client” I mean “end user” or “consumer of the service”.
      Now, consider a simplified example where a request from the client to the service takes 200ms and the request from the service to the remote service takes another 100ms during normal operation. If the service only receives a single request it does not matter if the service is single threaded and synchronous or multithreaded and asynchronous, the total response time for the client will be 300ms regardless of the service implementation. However, if we make 100 requests concurrently, the response time of all requests to the single threaded, synchronous service will be 10.2 seconds (200ms + 100 * 100ms) whereas for the asynchronous service it will still be 300ms. Consequently, the asynchronous implementation matters if your service has a high load.
      Secondly, consider another example where the request from the client to the service takes 200ms, but this time the request from the service to the remote service takes 5 seconds. If a single request is being sent, the response time from the clients perspective is 5.2 seconds and the following request will also take 5.2 seconds. However, if the response to the second request can be generated from a cached value rather than from the remote service the response time for that (and any subsequent request with the same cache key) will only be 200ms since the call to the remote service is avoided. In other words, both the synchronous and the asynchronous service may get significantly decreased response times if cache is used. As a side effect, a cache will also decrease the load on the remote service.
      These are just two staged and very simplified examples, but I hope they provide some more understanding to the problem.

  2. Haytham

    I think the link you provided to one of your Colleague’s blogs answers my question:
    http://blog.jayway.com/2014/05/16/async-servlets/

    Returning a DeferredResult signals to Spring that the request should be treated asynchronously. After invoking setResult the response will be sent back to the client.

  3. Bob

    From a testing perspective, does this work with MockMvc?

    • @Bob: Yes, MockMvc is supported, however you need to write some extra lines compared to a synchronous flow in order for it work. First, you start the async processing by calling the mockMvc.perform(...) as you would do normally, but make sure to keep a reference to the returned MvcResult. Next, you make an async dispatch by re-using the MvcResult. The following lines are copied from the JavaDoc of the MockMvcRequestBuilders.asyncDispatch(MvcResult mvcResult) method:

  4. Gustavo Orsi

    Also, since 4.1 you can use lambdas to set the callback methods (success and failure).

    repositoryListDto.addCallback(r -> deferredResult.setResult(new ResponseEntity(r, HttpStatus.OK)), t -> deferredResult.setResult(new ResponseEntity(HttpStatus.SERVICE_UNAVAILABLE)));

  5. Carl Moser

    You don’t really need the complex controller code to map the ListenableFuture into the DeferredResult manually. Why do you don’t return the ListenableFuture directly? Spring supports this as return type and is doing the rest for you, see http://docs.spring.io/spring/docs/current/spring-framework-reference/html/mvc.html#mvc-ann-return-types. The controller has nothing more to do than pass it through. That’s make the code easier to read and keep it clean. The result is exactly the same.

    1. The Response type of the server and controller method has to be the same. Instead of returning a ResponseEntity of RepoListDto you could simply return a ListenableFuture of RepoListDto.

    2. The exception handling can be implemented in a @ExceptionHandler annotated method.

    CHANGES TO THE CONTROLLER

    @RequestMapping(“/async”)
    ListenableFuture async(@RequestParam(“q”) String query) {
    return repoListService.search(query);
    }

    @ExceptionHandler(Exception.class)
    public ResponseEntity handleException(Exception e) {
    log.error(“Failed to fetch result from remote service”, e);
    return new ResponseEntity(HttpStatus.SERVICE_UNAVAILABLE);
    }

    • @Carl: Thank you for your comment. You are quite right, the implementation can be simplified in the ways that you suggest. The reason why I chose the implementation that I did was because I used Spring Boot version 1.1.6, i.e. Spring Framework version 4.0.7. Returning ListanableFuture from a controller or other service was not supported until Spring 4.1. I guess that I could have been more clear about which version was used in the blog post, but you can see the versions used in the associated project at GitHub. Moreover, I mentioned the possibility in one of considerations above as a suggestion when migrating to Spring 4.1.

  6. Sting

    Does it work if the restTemplate’s getForEntity timeout?

Leave a Reply