Error handling in API Gateway and AWS Lambda

Error handling is one of the most important things when building an application. The ability to give our clients comprehensive and well defined errors when something goes wrong is just as important as as our “happy paths”. With this in mind, error handling in API Gateway and AWS Lambda is not as straight forward as one would like. There are a lot of moving parts and somewhat quirky behaviour on both ends that makes it quite a challenge to manage. In this post I will show you what I have found to be the best strategy to manage your errors, although it is admittedly far from perfect.

For the impatient there is a short summary at the end of the post.

A simple use case

In this post I will present a very simple use case: Get a resource from a public API using API Gateway and AWS Lambda. The goal of the exercise is to receive a 200 OK HTTP Status code if it exists or a 404 Not Found HTTP Status code with a custom error object if the requested resource did not exist. The focus will be on 404 Not Found.

Lambda and Errors

The first thing we have to understand is the options we have when sending a response from a synchronous Lambda execution.

There are two alternatives available; we can either succeed(result)  or fail(error) . succeed implies a successful execution and fail an erroneous execution.

There is a third alternative named done(error, result)  but it is very important to read the fine print of the documentation here. The main take away is that done does not change the behaviour of succeed and fail, it merely wraps the two functions. This means that calling done with a non-null error value will give the error in the response just like fail would, the result value is ignored

So far this looks pretty good. If something goes wrong we just call context.done(error)  (or context.fail(error) ) and our Lambda function will respond with the error value.

Unfortunately it’s not that simple. What Lambda does here is a bit odd. It takes the error, regardless of what type it is, and essentially calls toString() on it and sticks it in a wrapper object. Consider this example:

We get the error but this is a standard NodeJS error and doesn’t provide a lot of information. It will actually expose some information that we may not want to give to the calling party: the stacktrace.

What if we add an additional attribute the error? Let’s add a “code” attribute to give it some context:

As you can see, the “code” attribute was ignored so it appears we are limited to manipulating the message of the Error if we want to pass custom data.

What if we want to send a custom object instead?

It got even worse! It just took the object a called toString(). Apparently errorMessage is just a string and cannot be something more complex but we want to give it something complex.

Let’s tweak the example above slightly and give Lambda that object as a string:

Ok, at least we got our data there. It’s no longer an object but it is at least a parseable JSON string. So far this looks like the best alternative for passing a custom error object.

At this point some of you probably think that we could simply pass the error as a successful response instead:

This works very well as you can see. We get the error object as JSON and everything looks good. However, this will not play well together with API Gateway. I will show you why in the next section.

API Gateway and Response Mapping

Now that we understand how Lambda errors are passed we will move on and take a look at how API Gateway receives them.

API Gateway have something called Method Response and in there we can specify the different HTTP Status codes that the endpoint can respond with. Once that is done we skip over to Integration Response and here is where the fun begins. We can map each status code to a regular expression called “Lambda Error Regex”. One status code is generally reserved as the default and does not have a RegExp. It will act as a catch-all status code.

The Lambda Error Regex is just what it sounds like; a regular expression that is evaluated against a Lambda error response. I’m emphasising error response because if the Lambda function succeeds API Gateway will always assume that it can use the default response. This is why returning the Lambda error as a successful response will not work: we won’t be able to map the response to a proper HTTP Status Code in API Gateway!

With this in mind we can define our API Gateway resource:

It’s a GET operation that expects an ID parameter and it can respond with 200 OK or 404 Not Found.

We have already defined that our Lambda will respond with a stringified error object with the attribute code: “NotFound” if the resource does not exist. This will help us when configuring our Integration Response Mapping:

API Gateway Integration Response Mapping

What does this configuration do?

First of all there is a default mapping that will yield a 200 OK. Since this is the default it will accept all successful Lambda responses as well as any error that does not match any of the Lambda Error Regex we supply which means that we have to take care not to return any undefined errors. In a real application we would probably want to add mappings for 500 and 400 status code as a minimum.

The second mapping will yield our 404 Not Found response. The Lambda Error RegExp will look for the string “NotFound” (with quotes) anywhere in the response if the Lambda function responded with fail.

Our stringified error contains this string and will map to this Integration Response and API Gateway will respond with a 404 HTTP Status code. In addition we have configured this Integration Response to let the response pass through which means that our stringified error message will be returned to the client in the body.

But wait. That’s a stringified error object? Yes, unfortunately API Gateway does not provide a helper function to parse this string back to JSON so we have to ship it all the way to the client as a string. This is the big downside to this approach as this means that the client has to do the parsing!

In a real world example using jQuery it could look something like this on the client side:

Update!

Amazon has just announced a parseJson function in the utility of the mapping templates.

This means that we can do the parsing on the API Gateway side and give a proper object back to the client. See the announcement for more information.

Demo

To tie this post together I have created a demo that implements this error handling:
Demo 200 OK
Demo 404 Not Found

TL;DR;

Lets summarize what we’ve learned:

  • The Lambda function error response is wrapped and stringified and to work around this we stringify our error before failing the Lambda function.
  • API Gateway will only evaluate our Integration Responses error regexp if the Lambda function fails.
  • Map the response to an HTTP Status code using a well defined string in the error, preferably something that is encapsulated to avoid false positives.
  • Have API Gateway pass the output from the Lambda right through to the client. On the client side we do  JSON.parse(error.errorMessage)  to get the error back.

Conclusion

Error handling in API Gateway and Lambda is currently awkward at best. If Amazon could provide a $util.JSONparse(x) helper in the response mapping we could at least hide these oddities from our clients. Until then this appears to be the best way to pass complex error structures back to the clients and at the same time map the errors to the appropriate HTTP Status code.

I have an ongoing discussion with Amazon regarding these issues and hopefully we’ll see some improvements coming.

This blog post is part of a series of posts on server-less deploys on AWS using API Gateway and AWS Lambda.

32 Comments

  1. Andy

    You were so close to figuring out how to pass the error message object. All you needed to do was create an Integration Response template (not “pass-through”) that looks like this:
    {$input.path(“$”).errorMessage}

    That will grab the ‘errorMessage’ object out of the lambda failure, and return it as the output.

    • Hi Andy,

      Thanks for you comment. I am not sure I understand what your suggestion would solve though.
      Getting the errorMessage data through to the client is not the issue. It’s the data type that is causing problems. The data type is not related to API Gateway but to Lambda. Mapping the errorMessage as you suggest will not change what Lambda gives back, i.e. toString() of what ever is provided in the error argument position, from the failed invokation and thus the transformation you suggest will only change the outlining structure of the response body, not the actual error contents which would still be either
      “[object Object]” or the stringified JSON.

      What’s missing is a way to convert from JSON string back to JSON object in the VTL response mapping templates. That would solve the problem or at least mitigate the effects of it.

  2. Thanks for this great post! One issue I’ve run into is that malformed user requests (e.g. invalid JSON) are still returning a 200 OK response with the following response body:

    200 OK
    {
    “Type”: “User”,
    “message”: “Could not parse request body into json.”
    }

    Have you found a way to get these types of requests to map to the appropriate HTTP error code?

    Josh

    • Hi Josh!
      Can you supply some more information on your use case. I’ve seen that error message quite a few times but mostly while I’ve been working out my request mappings. Once the mappings are in place I am unable to reproduce this error regardless of what the user input is. What does your request mapping template look like?

      / Carl

  3. Devin

    Hey Carl,

    Through some testing I found that you actually do have the ability to set the ‘code’ attribute of the error that is sent to API Gateway. You want to set the ‘name’ property of your Error object – that gets directly mapped to the ‘code’ attribute in API Gateway.

    This way I am able to use codes that my client application can parse and knows how to handle. The ‘message’ attribute is then used primarily for Integration Response mapping to a specific HTTP Status code.

    If you need more than just a specific code or message, however, API Gateway is not forgiving as you know. I know one team at my company is just making everything a 200 response (including errors), but in the case of an error they return an additional ‘error’ object within the response and clients simply check for that. I’m not a huge fan of that but I can see the merits if you want custom data in your error response.

    Let me know if you have any questions or concerns, I’ve been using this with my team for a couple months and it seems to be working well.

    – Devin

    • Hi Devin!

      Thanks for the tip with the name property, I’ll check that out.

      I considered the 200 OK solution as well but I want my APIs to be semantically correct and passing everything as a 200 goes against the principles of the protocol so I decided against it although I can also see the merits of that solution as you so eloquently put it :)
      The problem here isn’t really API Gateway but how Lambda treats errors. API Gateway could rectify this issue quite easily by providing a JSON.parse method in the VTL utilities, that is assuming that you use my approach and JSON.stringify the error.

      It’s nice to see that I’m not the only one who’s been struggling with this though, that means I haven’t completely misunderstood things :)

      / Carl

      • Devin

        Hey Carl,

        I think it’s actually the integration between Lambda and APIG where the issue lies. True, Lambda doesn’t behave well if you try to send a custom object as the error as you pointed out. Using actual Error objects, however, does provide the expected results, but APIG is specifically only using the ‘name’ and ‘message’ properties of the Error object when rendering responses.

        If we could set a certain property of an Error object with our own custom data, that would be enough. I never considered just sending a stringified JSON object as the error content though. I might utilize this strategy if I ever get to the point where I absolutely have to include custom data in my error response :)

        – Devin

        • Let’s call them equally evil in this case ;)
          One could argue that Lambda shouldn’t really care about what I send in my response and that API Gateway should parse for potential status code mapping from the entire response, error or not. This could introduce performance issues though which is probably why API Gateway is restricted to a selected few properties.

          On the other hand you are absolutely right. As long as we only work with actual Errors it works quite well. The problem as I see it is that a lot of times the Error object isn’t sufficient to properly communicate with the client. With the name addition, thank you for that, it does get a little bit better though.

          I’m really hoping for a JSON.parse function in the utilities in the response mappings so that we can hide the ugliness of the stringified JSON from the client. It’s a non-breaking change so it should be a no-brainer for AWS to introduce.

          If you stumble across a better way to handle these error, let me know! :)

          / Carl

        • Hi again Devin,

          I just tried adding a ‘name’ property to my error.
          I’m not sure if I misunderstood the way you meant for this to work but it’s not working for me.

          My Lambda:
          exports.handler = function(event, context) {
          var error = new Error(‘Access denied’);
          error.name = ‘Forbidden’;
          context.fail(error);
          };

          The result:
          {
          “errorMessage”: “Access denied”,
          “errorType”: “Forbidden”,
          “stackTrace”: “…”
          }

          In API Gateway I first mapped 403 to the regexp “Forbidden” (also tried “.*Forbiddien.*”) but this rendered a 200 OK response from API Gateway. I changed the regexp to “Access Denied” and that works and gives a 403 from API Gateway. It appears it doesn’t take the “typeError” property into consideration.

          Please point me in the right direction if I’m doing something wrong.

          / Carl

          • Devin

            Hey Carl,

            The regex matching is always against the ‘errorMessage’ property when you use an Error object, which is why you’re seeing this behavior.

            I use the ‘errorType’ property for my client application to handle the error the right way, i.e. show the right view.

            I prefix the ‘errorMessage’ with a specific string for regex matching, i.e. “BAD REQUEST” for 400s, “INTERNAL ERROR” for 500s.

            Hope that helps!
            -Devin

          • Ah, I misunderstood your first post. I thought you meant that Api Gateway would look at both the errorType, if supplied, and the message. If it only looks at the message We’re back to square one again :)

  4. Thanks, saved me a bunch of time.

  5. Ryan Green

    Thanks for the great post on error mapping with Lambda and API Gateway. This is an area where we get a lot of questions and we would love to improve this experience. Please get in touch if you’d like to discuss your use-cases in more detail.

    Thanks,
    Ryan
    API Gateway

    • Hi Ryan,
      It’s great that this post can provide you with more insight on the customer experience.
      I’ll get in touch with you through other channels for further discussion.

      / Carl

  6. Kevin Hornschemeier

    I spent the last couple days trying to figure this out and finally came up with a working solution, thanks in part to this post! It was all trial and error, so I can’t say I fully understand it, but it seems to be working by returning the proper http status code AND an object.

    In my Lambda when I want to return an error, say a 500, then I have an object that I stringify and send to context.done (context.fail works as well).

    var error = {
    code: 500,
    message: ‘500 message’
    };
    context.done(JSON.stringify(error));

    In the Gateway I have a 500 method, and under Integration Response I added the 500 response with a simple regex (.*500.*) to capture it (it’s looking at the string “\{code: 500, message: \”500 message\”\}”). And in the mapping template I have an application/json type with the following template:

    #set($inputRoot = $input.path(‘$.errorMessage’))
    $inputRoot

    My understanding of this – based solely on a bunch of tedious testing – is $input.json(‘$’) returns a JSON object with the result as a JSON string in the form of:

    { errorMessage: “\{code: 500, message: \”500 message\”\}” }

    And if you do $input.json(‘$.errorMessage’) then you get the string, but can’t parse this into JSON in any way. However, doing $input.path(‘$.errorMessage’) actually returns that item as a JSON object. So the response I get is an http status code of 500 with the following real JSON object:

    {
    “code”: 500,
    “message”: “500 message”
    }

  7. Neil

    Thanks for blogging about this point. I feel that the current error handling pain-points are due to limitations in API Gateway’s integration with AWS Lambda. Currently to return custom error info back to API Gateway and to be able to map it to a Method Response you have to report an unhandled error (exception in the Java world) back to AWS Lambda. I would rather just return a custom ‘failure’ response from the Lambda handler method, and have the ability to parse that in API Gateway. I’ve posted this feedback on the AWS forum https://forums.aws.amazon.com/thread.jspa?threadID=222078.

    • Hi Neil,

      It’s good to see that this issue is getting some traction within Amazon.
      I’ve received indications that they will add a custom property that we can use to trigger error codes in API Gateway but no confirmation on the details or when this feature could become available. Hopefully it’ll solve these issues altogether.

      / Carl

  8. Cam

    Carl, great post and ongoing discussion. I have hit the same issue in error responses but wanted to see if anyone have solved a related problem of attempting to return a 201 for a POST to create a resource. Ideally I would like to return to 201 status and also map a link to the resource path in the Location header. It seems from my testing you are not able to return any other 2xx response other than the default 200.
    Thanks Cam.

    • Hey Cam,

      Returning a 201 is possible by deleting the 200 method response (under Method Execution -> Method Response) and then adding back a 201 method response.
      Note that you can obviously only have one 2xx response code configured per method.

      Philipp

  9. Damon

    if you’re trying to do this in Java (and not in Node.js, which is what every example about this subject online is for), then this is the article you want:

    http://kennbrodhagen.net/2016/03/09/how-to-return-a-custom-error-object-and-status-code-from-api-gateway-with-lambda/

    however, make sure you read the comments at the bottom that i just posted, as there are some totally non-obvious gotchas that will cause you much pain and suffering.

    also, even if you are doing this in Node.js, you (probably) need to add the mapping template secret incantation that is in that article, which ironically, is also based on a Node.js implementation.

  10. Tony

    This was a super helpful post. Thank you!

  11. alex nishikawa

    Thanks for tutorial

  12. Peter

    This post was a life-saver. I agree with Andy though, if you use your response mapping right, you don’t have to parse your JSON at the client layer

  13. Orr

    Hi,
    My name is Orr Weinstein, and I am a senior Product Manager on the AWS Lambda team.

    Just wanted to update that today, three API GW features were launched that both simplify Lambda integration, and also make it much more powerful (depending on your needs).

    API GW What’s new post: https://aws.amazon.com/about-aws/whats-new/2016/09/amazon-api-gateway-introduces-3-new-features-to-simplify-api-configuration/
    Jeff Barr’s blog post: https://aws.amazon.com/blogs/aws/api-gateway-update-new-features-simplify-api-development/

    Thanks,
    Orr

  14. RAHUL BENIWAL

    Hi Carl,

    You have done almost right, just use context.fail(JSON.stringify(myErrorObj)) in case of error instead of callback(JSON.stringify(myErrorObj)); it will return json in response and with 400 HTTP status if defined in API gateway response integration.

  15. Vishwesh

    Thanks for the post Carl.

  16. Ravi

    I am using NodeJS for AWS Lambda + API Gateway APIs.

    I have multiple Lambda functions and each giving different response formats as it integrated multiple third party SDKs like Stripe/DynamoDB and all.

    Is there any way to get common response for all the functions like below?

    {
    “success” : true,
    “data” : { RESPONSEFROMLAMBDA },
    “messages” : null,
    “code” : 200,
    “description” : “OK”
    }

  17. I’m a bit late to the game, but thanks for this article. It helped me sort out the regular expression. Still a newbie but I’m getting there!

Leave a Reply