AWS Glue Dev Endpoint Deleter

Development of AWS Glue scripts can potentially add unnecessary expenses to your invoice if you are not careful. This blog post shows one way to avoid some of the cost in an automated fashion by using AWS CloudFormation and AWS Lambda.


A while ago, I had the opportunity to explore AWS Glue, a serverless extract, transform and load (ETL) service from AWS. The AWS Glue service offering also includes an optional developer endpoint, a hosted Apache Zeppelin notebook, that facilitates the development and testing of AWS Glue scripts in an interactive manner. Typically, you only pay for the compute resources consumed while running your ETL job. However, in contrast to the rest of AWS Glue family, usage of developer endpoints are charged hourly, regardless if you actively use them or not. You can see this when launching a dev endpoint from the AWS Console:

Billing for a development endpoint is based on the Data Processing Unit (DPU) hours used during the entire time it remains in the READY state. To stop charges, delete the endpoint. To delete, choose the endpoint in the list, and then choose Action, Delete.

Is this something that we should care about? Each dev endpoint gets 5 DPUs by default (with a minimum of 2 DPUs), and they are currently priced at $0.44 per DPU-hour (billed per second, 10 minutes minimum). Put it differently, you pay $2.20 per hour or $52.80 per day for the default configuration (or more than $100 for a weekend if you forget to delete your endpoint before you go home for the weekend which I did). For me, this was expensive enough to motivate an automatic dev endpoint deleter.


The solution consists of the following components:

  • A Lambda function that lists all Glue developer endpoints and subsequently deletes them.
  • A CloudWatch Event that triggers the Lambda function at scheduled intervals (typically at the end of each workday).
  • A Lambda Permission that allows the CloudWatch Event to invoke the Lambda function.
  • An IAM Role that allows the Lambda function to get and delete the Glue developer endpoints.
  • A CloudFormation template that comprises all resources. Thus, the stack can be re-used across AWS accounts and AWS regions.


The entire solution is presented in the CloudFormation template below. By inlining the Lambda source code into the template a single file is enough for both the infrastructure as well as the application logic. I have chosen to declare the cron expression as a parameter. I have scheduled the CloudWatch Events to trigger the Lambda when I leave the office, typically at 5PM. Since the cron expression is given in UTC the actual time will depend on daylight saving. A cron expression of 0 16 * * ? * translates to 5PM CET (Central European Time) in the winter and 6PM CEST (Central European Summer Time) in the summer.


The AWS documentation is comprehensive, yet it can be hard to navigate. Here are some links that you may find useful:

Mattias Severson

Mattias is a senior software engineer specialized in backend architecture and development with experience of cloud based applications and scalable solutions. He is a clean code proponent who appreciates Agile methodologies and pragmatic Test Driven Development. Mattias has experience from many different environments, including everything between big international projects that last for years and solo, single day jobs. He is open-minded and curious about new technologies. Mattias believes in continuous improvement on a personal level as well as in the projects that he is working on. Additionally, Mattias is a frequent speaker at user groups, companies and conferences.

Leave a Reply

Close Menu