Sunday, January 31, 2021

Using JQ in AWS Lambda Custom Runtime via AWS Lambda Layer

For a situation wherein there's a need to use JQ on AWS Lambda custom runtime, an AWS Lambda layer can be created and used for your AWS Lambda function. This blog post explains how the aforementioned can be achieved but is not just limited to creating a layer for JQ, the instructions can be similarly used for building AWS Lambda layer for any Linux distribution.

For quick reference

AWS Lambda: is a serverless compute service that allows running code without provisioning or managing servers. It's a powerful tool backing for potentially implementing the serverless architecture.

JQ: it's is like sed for JSON data and is a fast and flexible CLI JSON processor tool written in portable C.

Steps to follow

The entire process of building an AWS Lambda layer can be broken down as follows:

  • Get the required distribution files
  • Build a zip archive of the required files 
  • Create a layer on AWS Lambda
  • Use created AWS Lambda layer for your Lambda function

Get required distribution files

Since AWS Lambda custom runtime is based on Amazon Linux AMI, let's first get the JQ files specific to Amazon Linux AMI. For this, create a new Amazon Linux EC2 instance (or use an Amazon Linux Docker image). 

Install JQ and locate the required files (installed on Amazon Linux EC2 instance) once the installation is completed. 

Installing JQ

sudo yum install jq
At the time of writing this post, for JQ version 1.5, the required files can be found at the following location: 
# executable

# dependencies

Build a zip archive with required files

Build a archive containing all these required files so that JQ is functional when used inside the AWS Lambda function. The jq executable file should be at the root of the zip file and dependencies should be inside the lib directory in the zip file. The reason being, when AWS Lambda layers are unpacked by AWS, the custom runtime dependency path of the Lambda function should be /opt and /opt/lib. To simplify it, /opt is where the executables should go, and /opt/lib is where the required dependencies should go.

Create a layer of AWS Lambda

Now let's create a JQ layer on AWS Lambda so that it can be used by the AWS LAmbda function. It can be directly created via AWS Console or use the following command to create it using the AWS CLI:

aws lambda publish-layer-version --layer-name jq --zip-file /PATH_TO_FILE/

Use created AWS Lambda layer for your Lambda function

Once the jq layer is ready to use from the above step, create an AWS Lambda function with custom runtime using AWS console. After creating a Lambda function with custom runtime, AWS automatically creates a bootstrap and files whose content (at the time of writing this post) is as follows:


set -euo pipefail

echo "##  Environment variables:"

# Handler format: <script_name>.<bash_function_name>
# The script file <script_name>.sh  must be located at the root of your
# function's deployment package, alongside this bootstrap executable.
source $(dirname "$0")/"$(echo $_HANDLER | cut -d. -f1).sh"

while true
    # Request the next event from the Lambda runtime
    EVENT_DATA=$(curl -v -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
    INVOCATION_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)

    # Execute the handler function from the script
    RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")

    # Send the response to Lambda runtime
    curl -v -sS -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$INVOCATION_ID/response" -d "$RESPONSE"
function handler () {

    RESPONSE="{\"statusCode\": 200, \"body\": \"Hello from Lambda!\"}"
    echo $RESPONSE

Update the function so at it prints the jq version to test if the jq layer is working properly.

function handler () {
    cd /opt
    ./jq --version   

Now run a quick test on the Lambda function and it should print the jq version of the AWS Lambda layer as follows:


Follow a similar process to create an AWS Lambda layer for any distribution.

No comments:

Post a Comment