Testing hard to reach app states with a custom debug shell

I have worked in a few app projects that have been up and running for quite some
time. The apps often handle a bunch of error states or seldom visited states
that are hard to test manually. Often I find out that these states are not
tested at all, mainly because no-one knows how to put the app in the state to
test or that it is too bothersome.

These states are often relatively easy to test with unit tests if the code is
written in such a way. Unfortunately my experiences tell me that the lack of
proper testing and hard to test code often come hand in hand.

So what can you do if it is too difficult or too expensive to cover states
with unit tests or automated tests?

In comes the custom debug shell

In order to test specific app states it must be possible to put the app in the
states that we wish to test. One way to do this is to attach a debugger and
change the values of control variables. This approach will be difficult when
many variables are involved and is equally hard each time the test is performed.
It not only requires a developer, it also requires that the developer has a good
grasp on how the app works. Furthermore, using a debugger changes the timing of
the app which will affect timing sensitive functionality.

My approach is to add separate functionality that can both query and change the
app’s state from within the app itself. The state altering functionality is
accessed via telnet as a debug shell and is not present in release builds. It is
possible to do this in a way that all members of the team, not only developers,
can make use of the functionality.

Since the debug shell is part of the app it can gain access to run-time values
and other app private data which are often used by the app to determine state.
It is also extremely useful to be able to inspect the data that the app is using
from a terminal. The alternative is to log values or to find appropriate places
to place breakpoints.

The approach in this post can be used on a wide variety of platforms. I mostly
use it on Android but you can port it to whatever platform you want.

Examples of what the shell can be used for

  • Put the app in a specific state by modifying runtime values using
    direct object access or via Reflection
  • Modify or inspect (encrypted) database data
  • Pause/resume/inspect network message queues
  • Modify timestamps for user events. For instance to trigger a logout
    of the user due to inactivity
  • Fake incoming messages to an app
  • Check which authentication state the app is in
  • All main UI actions can be triggered from the shell. The shell can then be
    used by automated tests to run the application without having to interact with
    the ui
  • Load base64 coded jar files with custom code to execute within a running app.
    This makes it possible to run whatever code you want in an existing running
    app (in case you want a debug shell function that wasn’t implemented when the
    app was compiled)

These are just a few examples of what you can do. The possibilities are endless.

Basic implementation

The debug shell basically listens on a socket and reacts to input. Below is a
snippet of code that shows the basics of how a shell can be created in Java:

ServerSocket serverSocket = null;
try {

    serverSocket = new ServerSocket(12345);

    // Wait for a client to connect
    Socket clientSocket = serverSocket.accept();

    // A client has connected
    PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);
    BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
    String inputLine;
    String outputLine;

    // We listen for a line of input, handle it and repeat until one of the commands
    // we support returns a string indicating that the connection should be closed.
    final String goodbyeResponse = "Farewell!";

    while ((inputLine = in.readLine()) != null) {
        // parse and handle inputLine
        outputLine = mCommandHandler.handle(inputLine);

        // respond to client
        out.println(outputLine);
        out.flush();

        // Handle quit messages
        if (goodbyeResponse.equals(outputLine)) {
            clientSocket.close();
            break;
        }
    }
} catch (IOException e) {
    // Happens when the client has disconnected. And at other IO related events/errors
} finally {
    // Close server socket when done
    if (serverSocket != null && !serverSocket.isClosed()) {
        try {
            serverSocket.close();
        } catch (IOException e) {
        }
    }
}

The snippet above should run on a background thread so that it does not hang the
main thread when waiting for connections and when serving requests. Furthermore
it should be placed in another loop that will allow for new clients to connect. On
Android the app must have the INTERNET permission in order to open sockets.

The mCommandHandler is the handler of all your custom commands. It
receives an input string and provides an output string. Your custom functionality
is triggered before returning the output string from the handle method
in your mCommandHandler.

How to connect to the debug shell on an Android device

It is possible to access the debug shell in three different ways. The examples
use the port 12345.

From a computer on the same wifi network as the device

telnet ipOfDevice 12345

From a computer that is connected to the device via USB

Forward localhost requests on port 12345 to the device on the same port:

adb forward tcp:12345 tcp:12345

Connect using telnet:

telnet 127.0.0.1 12345

From a telnet app on the device itself

Download a telnet app from Google Play, start it on the device, and connect to
127.0.0.1 port 12345.

What to do next

I encourage you to experiment in order to see what a debug shell can be used for
in your project. Use the code example above, implement the suggested loops,
put everything on a background thread and don’t forget to create a beautiful
ASCII help text to send when clients connect. Good luck!

Leave a Reply

Close Menu