Using JQ in AWS Lambda Custom Runtime via AWS Lambda Layer

For a situation wherein there's a need to use JQ on AWS Lambda custom runtime, an AWS Lambda layer can be created and used for your AWS Lambda function. This blog post explains how the aforementioned can be achieved but is not just limited to creating a layer for JQ, the instructions can be similarly used for building AWS Lambda layer for any Linux distribution.

For quick reference

AWS Lambda: is a serverless compute service that allows running code without provisioning or managing servers. It's a powerful tool backing for potentially implementing the serverless architecture.

JQ: it's is like sed for JSON data and is a fast and flexible CLI JSON processor tool written in portable C.

Steps to follow

The entire process of building an AWS Lambda layer can be broken down as follows:

• Get the required distribution files
• Build a zip archive of the required files 
• Create a layer on AWS Lambda
• Use created AWS Lambda layer for your Lambda function

Get required distribution files

Since AWS Lambda custom runtime is based on Amazon Linux AMI, let's first get the JQ files specific to Amazon Linux AMI. For this, create a new Amazon Linux EC2 instance (or use an Amazon Linux Docker image). 

Install JQ and locate the required files (installed on Amazon Linux EC2 instance) once the installation is completed. 

Installing JQ

sudo yum install jq

At the time of writing this post, for JQ version 1.5, the required files can be found at the following location: 

# executable
/usr/bin/jq

# dependencies
/usr/lib64/libjq.so
/usr/lib64/libjq.so.1.0.4
/usr/lib64/libonig.so.2

Build a zip archive with required files

Build a jq.zip archive containing all these required files so that JQ is functional when used inside the AWS Lambda function. The jq executable file should be at the root of the zip file and dependencies should be inside the lib directory in the zip file. The reason being, when AWS Lambda layers are unpacked by AWS, the custom runtime dependency path of the Lambda function should be /opt and /opt/lib. To simplify it, /opt is where the executables should go, and /opt/lib is where the required dependencies should go.

Create a layer of AWS Lambda

Now let's create a JQ layer on AWS Lambda so that it can be used by the AWS LAmbda function. It can be directly created via AWS Console or use the following command to create it using the AWS CLI:

aws lambda publish-layer-version --layer-name jq --zip-file /PATH_TO_FILE/jq.zip

Use created AWS Lambda layer for your Lambda function

Once the jq layer is ready to use from the above step, create an AWS Lambda function with custom runtime using AWS console. After creating a Lambda function with custom runtime, AWS automatically creates a bootstrap and hello.sh files whose content (at the time of writing this post) is as follows:

bootstrap:

#!/bin/sh
set -euo pipefail

echo "##  Environment variables:"
env

# Handler format: <script_name>.<bash_function_name>
#
# The script file <script_name>.sh  must be located at the root of your
# function's deployment package, alongside this bootstrap executable.
source $(dirname "$0")/"$(echo $_HANDLER | cut -d. -f1).sh"

while true
do
    # Request the next event from the Lambda runtime
    HEADERS="$(mktemp)"
    EVENT_DATA=$(curl -v -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
    INVOCATION_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)

    # Execute the handler function from the script
    RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")

    # Send the response to Lambda runtime
    curl -v -sS -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$INVOCATION_ID/response" -d "$RESPONSE"
done

hello.sh

function handler () {
    EVENT_DATA=$1

    RESPONSE="{\"statusCode\": 200, \"body\": \"Hello from Lambda!\"}"
    echo $RESPONSE
}

Update the hello.sh function so at it prints the jq version to test if the jq layer is working properly.

function handler () {
    EVENT_DATA=$1
    
    cd /opt
    ./jq --version   
}

Now run a quick test on the hello.sh Lambda function and it should print the jq version of the AWS Lambda layer as follows:

jq-1.5

Follow a similar process to create an AWS Lambda layer for any distribution.

Remote profiling using SSH Port Forwarding (SSH Tunneling) on Linux

In this blog post I’ll lay out few steps that are needed for remote profiling using SSH Port Forwarding (SSH Tunneling) using Yourkit profiler.

Steps to be followed on remote machine:

1) Download Yourkit profiler from official Yourkit website.

2) Extract the downloaded file anywhere.

3) What we need to do now is find the file named libyjpagent.so in the extracted folder corresponding to the system architecture of your remote machine. In my case it is located in yjp-2013-build-13086/bin/linux-x86-64.

4) Copy the libyjpagent.so file to any convenient location (if required), I copied it to /tmp/yjp folder.

5) Add the following VM option to the command line of your Java application:

-agentpath:/tmp/yjp/libyjpagent.so=port=7878

Here 7878 is the port that you will port forward to.

6) Your remote machine is all set for profiling.

Steps to be followed on local machine:

1) Port forward to remote machine by executing the following command on terminal:

ssh -N -f 1.2.3.4 -L 8085:1.2.3.4:7878

Explanation of what is going on in the above command:

-L 8085:1.2.3.4:7878 is forwarding port 7878 from 1.2.3.4 to localhost port 8085

-N -f parameters are forcing the ssh to go to background.

2) Download Yourkit profiler from official Yourkit website on you local machine as well.

3) Extract the downloaded file anywhere.

4) From terminal go to the following folder:

yjp-2013-build-13086/bin

and start Yourkit profiler using sh yjp.sh.

5) Now from yourkit profiler choose “Connect to remote application” option and enter localhost:8085 in the pop-up that asks for the link to your remote application.

6) Yourkit will start profiling your Java application on remote machine now.

This article was first published on the Knoldus blog.

How to Backup your Linux Computer

Ever wondered what will you do in case of a disk or system failure. No? Even Linux can't protect you from disk or system failure! But in order to get around this problem, it provides an option to make a compressed backup of all your data. Cloud storage is a good option to back up all your files but for many of us, it's not enough. Here's how to back up your data using Simple Backup Suite that's preinstalled in your Linux distribution (Note: I am using Ubuntu, so the details may slightly differ if you are using a different distribution):

1. From the search bar, just type "backup" to bring up preinstalled options. I'll be using Simple Backup Suite.
2. From the first tab "General", configure Linux how often you want to perform a complete back up and what format, if any, to use for compression.
3. To add files and directories for backup select the "Include" tab. The default is pretty good for most users, but you may have some important files that deserve to be backed up, so be thorough! You only have to do this once.
4. Select the Exclude tab if you want to keep some files or subfolders from being archived. There's no special reason to do this unless you're trying to save space.
5. Make sure to select the Destination tab to pick your storage location. As with any other backup, you'll want it to be as secure as possible. A network location is probably best, though your needs may vary.
6. Schedule tab lets you set a schedule to do incremental backups of files that have changed since the last backup. This happens in the background and shouldn't interrupt your day-to-day work.
7. Click the disk icon at the top to save your settings, then click the drive icon to back up your files. The first backup process will take a while.

Once you're done with it, the only reason you'd need to revisit the tool is to change the frequency, location, or contents of your backups.