Visualizing AWS Mobile Analytics using Elastic Search and Kibana

Amazon Mobile Analytics can be used to collect a lot of information. It is very easy to create charts in a custom dashboard but the easy of use means that you are rather limited both in what you can show and how it is showed. To work around this I looked into exporting from Mobile Analytics to Elastic Search and presenting using Kibana. Since a few months back AWS provides an Elasticsearch Service which seemed like a natural fit.

Export to S3

First step is to enable export to S3 from Mobile Analytics. When you do this an S3 bucket is automatically created, unfortunately you cannot control the region that will be used, it is always placed in us-east-1. Looking in the bucket you see that compressed files with analytic events are generated. Each line holds a JSON document similar to:

Import to Elastic Search

Setting up Amazon Elastic Search Service is simply a matter of selecting domain name and instance type. Now we need to get the data from S3 to Elasticsearch. This can be done by creating a lambda that listens to S3 events, see for example awslabs sample code. Note that the same file on S3 can be written three times, this means that we need to create an id for each event to avoid duplicates. This can for example be achieved by hashing the event line and use it for the special Elastic search id field called _id. Since each exported file contains many events we use the Elastic Seach /_bulk endpoint to PUT data similar to:

Elastic Search mapping

To use Kibana for generating graphs over time a timestamp is needed for each event. Mobile Analytics automatically generates a timestamp when an event occurs and when it is reported. AWS uses Elasticsearch 1.5.2 which uses the special field _timestamp for indexing timestamp, so we need to map the event_timestamp to this field (see example below).

By default elastic search will analyze new string fields. This makes sense when the fields contain human readable sentences. However the data we collect using Mobile Analytics is mostly string constants that shouldn’t be processed further. To alter this behavior you can use dynamic templates to tell Elastic Search not to analyze new string fields. The following curl example creates the index MY_INDEX with mapping for the document type event:

Elastic Search indices

You will probably only want to have recent events in Elastic search. To do this you can simply create new indices every day (remember to set up the mapping) and remove old ones. This can easily be achieved by using a scheduled lambda.

Kibanan

Kibana needs to be told about the new indices and how to use them:

  • Open Settings / Advanced and add _timestamp to metaFields.
  • Open Indices and configure a new index with the name you chose (note that wildcards can be useful if you choose to autogenerate indices) and set the Time-field name to _timestamp.

Now you can use Kibana to view data collected by Mobile Analytics and generate lovely graphs like:
kibana_android_version

Leave a Reply

Close Menu