I’ve recently been deploying a service by launching EC2 instances, with the new version, and terminating the old instances, running the previous version of the service. I think this has been working pretty well for me, as I want the instances replaced at some point anyway. There has, however, been a big drawback of this approach – losing my service’s log files, which I might want to use for troubleshooting and analytics.
I decided to backup my log files by copying them to S3. It was, however, not obvious how I should do that as I’m using logrotate, rotating over a set of files. The problem is that the log filenames are static when run with default options, but the content will change with every rotation. At first I considered the
dateext directive, to get the current date added to the filenames, but in the end I figured it would be easier to just add a rotate-hook to logrotate.
To create a logrotate hook is very simple. Just add the
postrotate/endscript directive with the script or commands you want to run. You’d also want to add
sharedscripts in order to make sure the script is only executed once per rotation.
My logrotate file (
/etc/logrotate.d/my_service) for my service looks like this:
/bin/bash /etc/my_service/upload_log_to_s3.sh log.1
But what about the current log file that has not been rotated yet? If the instance would be terminated, then the latest log file would not be uploaded.
The solution is to use a shutdown-hook for the system. In Ubuntu it is very easy to add a shutdown-hook by using upstart.
The upstart job config file (
/etc/init/shutdown-hook.conf) should look something like this:
description "run at shutdown"
start on starting rc
exec /bin/bash /etc/my_service/upload_log_to_s3.sh log
Finally I just needed the script (
/etc/my_service/upload_log_to_s3.sh) for uploading a log file to S3:
gzip -c /var/log/my_service/*.$log_file_ext > /tmp/log.gz
aws s3 cp /tmp/log.gz s3://my_service_bucket/logs/`date +%Y-%m-%dT%H:%M:%SZ`.log.gz
The upload script will just gzip the log file (needed as I’m using
delaycompress), rename the log file to the current timestamp, and upload the file using aws-cli. The argument sets the file extension of the log file, which is necessary to be able to upload both the current (
.log) as well as the previous log file (
This is of course not the best solution for all use cases, but I think this is really simple to implement. I hope you find it useful.