Table of Contents
Introduction
In this tutorial, we will build lambda with a custom docker image.
Prerequisites
Before starting, you should have the following prerequisites configured
- An AWS account
- AWS CLI on your computer
Walkthrough
- Create a Python virtual environment
- Create a Python app
- Create a lambda with a custom docker image of ECR
- Create ECR repositories and push an image
- Create a lamba from the ECR image
- Test lambda function on local
- Test lambda function on AWS
Create a Python virtual environment
Create a Python virtual environment with the name py_virtual_env
python3 -m venv py_virtual_env
Create a Python app
This Python source code will pull a JSON file from https://data.gharchive.org and put it into the S3 bucket.
Then, transform the uploaded file to parquet format.
Download the source code from here and put it into the py_virtual_env folder.
Create a lambda with a custom docker image of ECR
#build docker image
source ./py_virtual_env/bin/activate
cd py_virtual_env
docker build -t test-image .
docker images
Create ECR repositories and push an image
#create an ecr repository
aws ecr create-repository --repository-name lambda-python-lab \
--query 'repository.repositoryUri' --output text
#authen
aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com
#tag image
docker tag test-image:latest xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lambda-python-lab:latest
docker images
#push the image to ecr repository
docker push xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lambda-python-lab:latest
#check image
aws ecr list-images --repository-name lambda-python-lab
Create a lamba from the ECR image
Create a lambda function from the ECR image
General configuration
Environment variables
Test lambda function on local
To test the lambda function locally, you can follow
docker run --name test-lambda -v /Users/hieudang/.aws:/root/.aws \
-e BUCKET_NAME=hieu320231129 \
-e TARGET_FOLDER=TRANSFORMED \
-e FILENAME=2022-06-05-0.json.gz \
-d test-image
docker container list
docker exec -it test-lambda bash
python -c "import S3app;S3app.lambda_ingest(None, None)"
Test lambda function on AWS
Conclusion
These steps provide an example of creating a lambda function and running it on docker, then, we put the docker image in the ECR repository, and create a lambda function from the ECR repository. The specific configuration details may vary depending on your environment and setup. It’s recommended to consult the relevant documentation from AWS for detailed instructions on setting up. I hope will this be helpful. Thank you for reading the DevopsRoles page!
Refer
App source code: https://github.com/itversity/ghactivity-aws