Build lambda with a custom docker image

Introduction

In this tutorial, we will build lambda with a custom docker image.

Prerequisites

Before starting, you should have the following prerequisites configured

  • An AWS account
  • AWS CLI on your computer

Walkthrough

  • Create a Python virtual environment
  • Create a Python app
  • Create a lambda with a custom docker image of ECR
  • Create ECR repositories and push an image
  • Create a lamba from the ECR image
  • Test lambda function on local
  • Test lambda function on AWS

Create a Python virtual environment

Create a Python virtual environment with the name py_virtual_env

python3 -m venv py_virtual_env

Create a Python app

This Python source code will pull a JSON file from https://data.gharchive.org and put it into the S3 bucket.
Then, transform the uploaded file to parquet format.

Download the source code from here and put it into the py_virtual_env folder.

Build lambda with a custom docker image

Create a lambda with a custom docker image of ECR

#build docker image
source ./py_virtual_env/bin/activate
cd py_virtual_env
docker build -t test-image .
docker images
image 10

Create ECR repositories and push an image

#create an ecr repository
aws ecr create-repository --repository-name lambda-python-lab \
--query 'repository.repositoryUri' --output text

#authen 
aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com

#tag image
docker tag test-image:latest xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lambda-python-lab:latest
docker images

#push the image to ecr repository
docker push xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lambda-python-lab:latest

#check image
aws ecr list-images --repository-name lambda-python-lab
image 11

Create a lamba from the ECR image

Create a lambda function from the ECR image

image 12

General configuration

image 13

Environment variables

image 14

Test lambda function on local

To test the lambda function locally, you can follow

docker run --name test-lambda -v /Users/hieudang/.aws:/root/.aws \
  -e BUCKET_NAME=hieu320231129 \
  -e TARGET_FOLDER=TRANSFORMED \
  -e FILENAME=2022-06-05-0.json.gz \
  -d test-image
docker container list
docker exec -it test-lambda bash
python -c "import S3app;S3app.lambda_ingest(None, None)"
image 15

Test lambda function on AWS

image 16

Conclusion

These steps provide an example of creating a lambda function and running it on docker, then, we put the docker image in the ECR repository, and create a lambda function from the ECR repository. The specific configuration details may vary depending on your environment and setup. It’s recommended to consult the relevant documentation from AWS for detailed instructions on setting up. I hope will this be helpful. Thank you for reading the DevopsRoles page!

Refer

https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-no-dependencies

App source code: https://github.com/itversity/ghactivity-aws

About Dang Nhu Hieu

I'm Vietnamese. In the past, I'm a software developer, now working in Japan on an Infra team. Skill : AWS, VMware, HA architech,setting Database : Oracle DB, PostgresDB ,.. Programming language : Java, C#, Python, Bash linux, Batch windows, powershell ,... Hobbies: badminton, film photo, travel. https://www.linkedin.com/in/hieu-dang-15a0561a6/
View all posts by Dang Nhu Hieu →

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.