# Streamlining AMIs using Packer, Vault & GitHub Actions Nowadays, if you want to minimize human errors and maintain a consistent process for how software is released then you are going to rely on Continuous integration and continuous deployment (CI/CD). It's really hard to imagine how much productivity they bring into the plate. In this tutorial, we are going to take entire AWS instance backup using tools like Packer and see how it solves our problem and make our life much easier. ## Amazon Machine Image (AMI) An **Amazon Machine Image (AMI)** is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2. -- *Wikipedia* An AMI includes the following: * A template for the root volume for the instance (for example, an operating system, an application server, and applications) * Launch permissions that control which AWS accounts can use the AMI to launch instances. * A block device mapping that specifies the volumes to attach to the instance when it's launched. --- ## What is Packer ? Packer is a tool for building identical machine images for multiple platforms from a single source configuration. ![Packer_HashiCorp](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726267174/501d40cf-4542-495e-b851-35580df0fd9b.jpeg align="left") Image Source : https://www.hashicorp.com/ Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer comes out of the box with support for many platforms. To know more about Packer, visit : https://developer.hashicorp.com/packer --- ## Project Structure GitHub Repository : https://github.com/mukulmantosh/Packer-Exercises ![project_structure](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726268600/45a00e45-c710-4007-afd9-e7550a8717db.png align="left") * **.github** - Workflow files for GitHub Actions * **packer** - Contains HCL2 Packer templates, Shell Scripts etc. * **Dockerfile** - Building Docker Image * **main.py** - FastAPI Routes handling two endpoints * **requirements.txt** - listing all the dependencies for a specific Python project --- ## Let's Begin ![project_structure_2](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726270556/03e4fdf2-530c-465e-bc5b-c1f2ee22a209.png align="left") I have used Amazon Linux 2 with arm64 architecture as our base AMI. ![ami](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ifcn28rcnw5xs8k1qkz.png align="left") The custom AMI name is `FastAPI_Base_Image`. It's a clean AMI without any OS/Software dependencies. ![custom_ami_fastapi](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726271931/26fdf4bc-b26d-4e8a-8e3f-706a3723aa02.png align="left") If you are not sure how to create an AMI, follow this link : https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html ### Dockerfile I will create a container from the Dockerfile which is taking Python 3.9 as the base image and followed with python dependencies installation and starting the uvicorn server. ![dockerfile](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726273412/e908b965-f0d3-410d-b75b-70accce9e92b.png align="left") The image is already hosted in DockerHub. URL : https://hub.docker.com/r/mukulmantosh/packerexercise ![dockerhub](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726274599/b735957c-9ea4-4bd9-92f7-6c367082c9e8.png align="left") We have compiled for three architectures. Thanks to [Docker Buildx](https://docs.docker.com/build/). * `amd64` * `arm64` * `arm/v7` --- ## Packer Template **packer/build.pkr.hcl** ```plaintext variable "ami_name" { type = string description = "The name of the newly created AMI" default = "fastapi-nginx-ami-{{timestamp}}" } variable "security_group" { type = string description = "SG specific for Packer" default = "sg-064ad8064cf203657" } variable "tags" { type = map(string) default = { "Name" : "FastAPI-NGINX-AMI-{{timestamp}}" "Environment" : "Production" "OS_Version" : "Amazon Linux 2" "Release" : "Latest" "Creator" : "Packer" } } source "amazon-ebs" "nginx-server-packer" { ami_name = var.ami_name ami_description = "AWS Instance Image Created by Packer on {{timestamp}}" instance_type = "c6g.medium" region = "ap-south-1" security_group_id = var.security_group tags = var.tags run_tags = var.tags run_volume_tags = var.tags snapshot_tags = var.tags source_ami_filter { filters = { name = "FastAPI_Base_Image" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["self"] } ssh_username = "ec2-user" } build { sources = [ "source.amazon-ebs.nginx-server-packer" ] provisioner "shell" { inline = [ "sudo yum update -y", ] } provisioner "shell" { script = "./scripts/build.sh" pause_before = "10s" timeout = "300s" } provisioner "file" { source = "./scripts/fastapi.conf" destination = "/tmp/fastapi.conf" } provisioner "shell" { inline = ["sudo mv /tmp/fastapi.conf /etc/nginx/conf.d/fastapi.conf"] } error-cleanup-provisioner "shell" { inline = ["echo 'update provisioner failed' > packer_log.txt"] } } ``` [**User Variables**](https://developer.hashicorp.com/packer/docs/templates/legacy_json_templates/user-variables) User variables allow your templates to be further configured with variables from the command-line, environment variables, Vault, or files. This lets you parameterize your templates so that you can keep secret tokens, environment-specific data, and other types of information out of your templates. This maximizes the portability of the template. [**Builders**](https://developer.hashicorp.com/packer/docs/builders) Builders create machines and generate images from those machines for various platforms (EC2, GCP, Azure, VMware, VirtualBox) etc. Packer also has some builders that perform helper tasks, like running provisioners. [**Provisioners**](https://developer.hashicorp.com/packer/docs/provisioners) Provisioners use built-in and third-party software to install and configure the machine image after booting. Provisioners prepare the system, so you may want to use them for the following use cases: * installing packages * patching the kernel * creating users * downloading application code [**Post-Processors**](https://developer.hashicorp.com/packer/docs/post-processors) Post-processors run after builders and provisioners. Post-processors are optional, and you can use them to upload artifacts, re-package files, and more. [**On Error Provisioner**](https://developer.hashicorp.com/packer/docs/templates/hcl_templates/blocks/build/provisioner#on-error-provisioner) You can optionally create a single specialized provisioner called an error-cleanup-provisioner. This provisioner will not run unless the normal provisioning run fails. If the normal provisioning run does fail, this special error provisioner will run before the instance is shut down. This allows you to make last minute changes and clean up behaviors that Packer may not be able to clean up on its own. The **amazon-ebs** Packer builder is able to create Amazon AMIs backed by EBS volumes for use in EC2. ```plaintext source "amazon-ebs" ``` This builder builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine. This is all done in your own AWS account. The builder will create temporary keypairs, security group rules, etc. that provide it temporary access to the instance while the image is being created. This simplifies configuration quite a bit. The builder does not manage AMIs. Once it creates an AMI and stores it in your account, it is up to you to use, delete, etc. the AMI. To know more, visit this link : https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs In the “source\_ami\_filter” section, We are filtering based on the base AMI which we created earlier. ```plaintext source_ami_filter { filters = { name = "FastAPI_Base_Image" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["self"] } ``` **most\_recent** - Selects the newest created image when true. **owners** - You may specify one or more AWS account IDs, "self" (which will use the account whose credentials you are using to run Packer) We are using a Packer function called **“timestamp”** to generate UNIX timestamp, which helps to get a unique AMI name on every build. By default the AMI’s you create will be private. If you want to share the AMI’s with other accounts you can make use of the **“ami\_users”** option in packer. If you want to build images in multi-region, you can specify the below code in the source section. ```plaintext ami_regions = ["us-west-2", "us-east-1", "eu-central-1"] ``` In the provisioner section we will be updating the OS along-with installing scripts and copy nginx configuration. **packer/scripts/build.sh** Installing Docker, NGINX, and pulling latest application image from DockerHub and starting the container. ```plaintext #!/bin/bash sudo yum install jq -y sudo yum install -y git sudo yum install -y docker sudo usermod -a -G docker ec2-user sudo systemctl enable docker.service sudo systemctl start docker.service sudo amazon-linux-extras install nginx1 -y sudo systemctl enable nginx.service sudo systemctl start nginx.service IMAGE_TAG=`curl -L -s 'https://hub.docker.com/v2/repositories/mukulmantosh/packerexercise/tags'|jq '."results"[0]["name"]' | bc` sudo docker pull mukulmantosh/packerexercise:$IMAGE_TAG sudo docker run -d --name fastapi --restart always -p 8080:8080 mukulmantosh/packerexercise:$IMAGE_TAG ``` **packer/scripts/fastapi.conf** Copy the configuration to NGINX configuration folder. So, NGINX will proxy the request to backend. ```plaintext upstream fastapi { server 127.0.0.1:8080; } server { listen 80; location / { proxy_pass http://fastapi; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_redirect off; } } ``` --- ## Building Template Before you begin to build, make sure you have setup the following keys in your system and **aws-cli** is installed in your machine. * `AWS_ACCESS_KEY_ID` * `AWS_SECRET_ACCESS_KEY` There are two commands which you need to run before you execute build. `packer fmt build.pkr.hcl` The packer [fmt](https://developer.hashicorp.com/packer/docs/commands/fmt) Packer command is used to format HCL2 configuration files to a canonical format and style `packer validate build.pkr.hcl` The packer [validate](https://developer.hashicorp.com/packer/docs/commands/validate) Packer command is used to validate the syntax and configuration of a template ### Starting the Build `packer build build.pkr.hcl` The packer [build](https://developer.hashicorp.com/packer/docs/commands/build) command takes a template and runs all the builds within it in order to generate a set of artifacts. ![animate_packer_build](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ahmfrd9vqq09emfxo6gy.gif align="left") You can see the new AMI has been successfully created and tag has been assigned. ![new_ami_1](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726277602/628fa052-a15c-4add-bfc2-482f2f6a0cbf.png align="left") ![new_ami_2](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726279585/8471b5a8-d38d-4e04-96f8-2166e6aee3c4.png align="left") You must have observed in the packer template, that we are using a custom security group. By default, Packer creates security group which access port 22 (0.0.0.0) from anywhere. This posses security risk and to minimize that, I created a custom security group (**Packer\_SG**) which allows only My IP. ```plaintext variable "security_group" { type = string description = "SG specific for Packer" default = "sg-064ad8064cf203657" } ``` ![custom_sg](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726280955/ff668246-f101-443a-bad4-84bd8c930d38.png align="left") You can add more security by taking leverage of Session Manager Connections. [**Session Manager Connections**](https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs#session-manager-connections) Support for the AWS Systems Manager session manager lets users manage EC2 instances without the need to open inbound ports, or maintain bastion hosts. --- ## GitHub Actions GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. ### Self-hosted runners For our setup we will be using self-hosted Github runners. Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud. Don't know how to setup ? Follow the below link : * [Deploying Self-hosted Runners for GitHub Actions](https://www.youtube.com/watch?v=G6nBM3NxBDc) * [About self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) As from security standpoint, we will make sure **"Packer\_SG"** security group allow inbound port 22 for Github Action IP. ![allow_sg_gh_action_1](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726282897/b75b113d-9294-448d-b552-eb3800fec073.png align="left") ![allow_sg_gh_action_2](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726284094/3a23d848-cb1e-4a73-b759-d03699ac8b8c.png align="left") ![allow_sg_gh_action_3](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726285443/380d653a-dd16-43f3-ac08-9ad4fd6c281e.png align="left") ### Execute Pipeline Before proceeding, make sure to create the secrets which will be required in the build process. `AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEY` ![action_secrets](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726287033/59f0de8e-afe7-41e3-a72d-22f404d26af2.png align="left") **.github/workflows/build-packer.yml** ```plaintext name: Packer on: push: branches: main jobs: packer: runs-on: self-hosted name: packer steps: - name: Checkout Repository uses: actions/checkout@v2 - name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v1-node16 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ap-south-1 # validate templates - name: Validate Template uses: hashicorp/packer-github-actions@master with: command: validate arguments: -syntax-only target: build.pkr.hcl working_directory: ./packer # build artifact - name: Build Artifact uses: hashicorp/packer-github-actions@master with: command: build arguments: "-color=false -on-error=abort" target: build.pkr.hcl working_directory: ./packer ``` On inspecting the YAML file, you can clearly observe that we will be validating packer templates and then followed by building the artifact. Let me make a small change in main branch. So, the pipeline will get triggered. ![packer_github_action_flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7gc791kcxld37dvnlqg0.gif align="left") You can see now, the new AMI is created. ![packer_github_action_ami](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726291987/3fbaa141-3bae-493e-84a1-bfcf847b2acd.png align="left") --- ## Vault ![hashicorp_vault](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726293299/b07d4915-0e9f-4b8c-960b-4ed8ec5ec9a5.png align="left") **HashiCorp Vault** tightly controls access to secrets and encryption keys by authenticating against trusted sources of identity such as Active Directory, LDAP, Kubernetes, Cloud Foundry, and cloud platforms. Vault enables fine grained authorization of which users and applications are permitted access to secrets and keys. To know more about Vault, visit this link : * https://developer.hashicorp.com/vault The reason we are using Vault over here is to create dynamic user credentials. This helps us to avoid setting up environment variables for `export AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXX"export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXX"` Putting this keys in local machine, might expose some risks. So, I would recommend trying out [**AWS Secrets Engine**](https://developer.hashicorp.com/vault/docs/secrets/aws). ### AWS Secrets Engine The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are **time-based** and are **automatically revoked** when the Vault lease expires. I have already setup Vault in my local machine. Follow the below link for setting up Vault. * https://developer.hashicorp.com/vault/downloads You can either setup in your local machine or use [HashiCorp Cloud](https://cloud.hashicorp.com/). Let's now begin by enabling the AWS secrets engine in our Vault server which is running locally. ![create_vault_aws_1](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726295082/adc3e4a0-ff71-47a5-a077-c2308abd4778.png align="left") ![create_vault_aws_2](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726297134/db2d2d12-9701-4aa5-a5d0-93db7b1b2498.png align="left") ![create_vault_aws_3](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726298396/ddbeb36c-eefc-4a67-b56a-43a5c9dc7d22.png align="left") Now, click on **Configuration** to setup our credentials. ![vault_configure](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726299662/965655af-40e3-498d-b9fe-6d49f70774b4.png align="left") Provide the AWS credentials and region which will be used to create user and attach role to them. ![create_vault_aws_4](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726301548/3af5d95b-194a-4240-b6b2-748daf2643fa.png align="left") Next, I will modify lease time to 15 minutes. So, once the user is created it will be deleted automatically after 15 minutes. ![create_vault_aws_5](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726303430/9e313f39-78fc-4293-af13-75c6dce137b1.png align="left") Click on Save. I have configured the AWS credential. Now, I will create the role which is going to be attached to the new user. ![create_vault_aws_6](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726304757/5d236424-d86b-4d5c-b7a0-e21648560c11.png align="left") **Policy Document** ```plaintext { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AttachVolume", "ec2:AuthorizeSecurityGroupIngress", "ec2:CopyImage", "ec2:CreateImage", "ec2:CreateKeypair", "ec2:CreateSecurityGroup", "ec2:CreateSnapshot", "ec2:CreateTags", "ec2:CreateVolume", "ec2:DeleteKeyPair", "ec2:DeleteSecurityGroup", "ec2:DeleteSnapshot", "ec2:DeleteVolume", "ec2:DeregisterImage", "ec2:DescribeImageAttribute", "ec2:DescribeImages", "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeRegions", "ec2:DescribeSecurityGroups", "ec2:DescribeSnapshots", "ec2:DescribeSubnets", "ec2:DescribeTags", "ec2:DescribeVolumes", "ec2:DetachVolume", "ec2:GetPasswordData", "ec2:ModifyImageAttribute", "ec2:ModifyInstanceAttribute", "ec2:ModifySnapshotAttribute", "ec2:RegisterImage", "ec2:RunInstances", "ec2:StopInstances", "ec2:TerminateInstances" ], "Resource": "*" } ] } ``` I would recommend follow defense in depth and principle of least privilege. Most of them don't encourage that policy document should contain delete permissions. I came across an interesting article for tightening your policy document and make it more secure. So, it won't interfere with other instances. Please checkout the below link : * [**Towards a more Restricted Set of AWS IAM Permissions for Packer**](https://blog.stefan-koch.name/2021/05/16/restricted-packer-aws-permissions) ![create_vault_aws_7](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726306695/644d69b3-4471-43e6-bb68-4ac4bc003153.png align="left") Now, I will click on **Generate Credentials**. ![create_vault_aws_8](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726308888/5a13c7c4-42cc-484f-8206-dc35b1519f20.png align="left") Now, it's going to create a IAM user which is valid for 15 minutes (900 seconds) You can see below, the new user is appearing in the IAM User section. ![create_vault_aws_9](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726310834/3a460117-d5a1-4c8d-a194-4fd417616ec9.png align="left") The **PackerRole** with assigned permissions are also being reflected. ![create_vault_aws_10](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726312113/cab4d464-32a0-44f2-9acf-1d185e3048d6.png align="left") Now, we are going to make sure that Packer should generate this credentials automatically. Let's begin by editing the **build.pkr.hcl** file. You need to add this line before closing of the source block. ```plaintext vault_aws_engine { name = "PackerRole" } ``` ![vault_aws_engine_packer_template](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726313593/d2d78f68-5fd5-4560-8169-af1efde2740d.png align="left") Next, you need to setup environment variables. Windows : `set VAULT_ADDR=http://127.0.0.1:8200set VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX` Linux : `export VAULT_ADDR=http://127.0.0.1:8200export VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX` Once, we are done setting up our environment variables. We need to validate everything is working as expected by running the `validate` command. ```plaintext packer validate build.pkr.hcl ``` If you receive this message `The configuration is valid` then you are good to proceed. To initiate the build run the below command : ```plaintext packer build build.pkr.hcl ``` * Note : Make sure before your begin build. The security group **Packer\_SG** allows inbound access to port 22 from MyIP, as you are running the build from local machine. ![Image description](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726315032/0d382b4c-7f29-4662-a611-a08e93820042.png align="left") Observe the message : `You're using Vault-generated AWS credentials` ![Image description](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726316361/5744e35b-4493-4732-aa9c-c6f4e2c8c4ca.png align="left") This is going to pick the credentials from Vault, which is going to dynamically create a new user and attach the role. The user will get automatically deleted based on the expiry specified. ![vault_iam_user_expire](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726318274/facb065a-8c3b-44ea-8a70-fda0d4a4e2b9.png align="left") Once, the build is complete. You will find the new image appearing in the AMI section. ![ami_images_list](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726319578/40c226f2-13e9-4825-9c95-d2f280a4be67.png align="left") ## Final Destination Congratulations !!! You did it 🏆🏆🏆 ![conclusion](https://cdn.hashnode.com/res/hashnode/image/upload/v1719726321059/83ff6301-58b8-47df-8ca9-59bbc5451d43.jpeg align="left") If you liked this tutorial 😊, make sure to share across your friends and colleagues. #### References * [Build a Golden Image Pipeline with HCP Packer](https://developer.hashicorp.com/packer/tutorials/cloud-production/golden-image-with-hcp-packer) * [How to Set Up Continuous Golden AMI Vulnerability Assessments with Amazon Inspector](https://aws.amazon.com/blogs/security/how-to-set-up-continuous-golden-ami-vulnerability-assessments-with-amazon-inspector/) * [Packer Docs](https://developer.hashicorp.com/packer/docs)