Blog

Introduction

In modern software development, Continuous Integration (CI) is essential for streamlining build and test automation, enabling seamless code integration, and maintaining high-quality standards. This project explores CI by leveraging AWS CodeBuild and AWS CodePipeline to efficiently automate the process of building, testing, and validating code changes.

To explore more of my projects, click here.

Developer Tools and Technologies Used

This project utilizes a range of tools and technologies to streamline Continuous Integration (CI) with AWS CodeBuild and AWS CodePipeline:

AWS CodeBuild: AWS’s fully managed build service compiles source code, runs tests, and produces software packages. It automates the entire build process.

AWS CodePipeline: A continuous integration and delivery (CI/CD) service that automates the steps required to release software changes. It orchestrates various stages, including source retrieval, building, testing, and deployment.

GitHub: A version control repository for storing and managing source code. It triggers automated builds in AWS CodeBuild upon code commits.

Docker: A platform for creating and running containers that package software and its dependencies into isolated environments, ensuring consistent performance across different systems.

Docker Hub: A cloud-based repository for storing Docker container images used in the application.

AWS Systems Manager: Facilitates the secure storage and management of Docker Hub credentials using Parameter Store, ensuring sensitive information is stored centrally and accessed securely.

Fork the GitHub Repository: Fork the GitHub repository used for this project here

Create IAM Role and Add Permissions Please navigate to the I AM console by typing I AM in the search bar then select I AM under services.

In the left-hand menu, click on RolesunderAccess Management.

Click on the Create role button.

Select the service that will use this role, and choose AWS service. Then under use case, select the drop-down button and select AWS code build.

Search for and select the following policies to attach to the role: AmazonEC2FullAccessand AmazonSSMFullAccess.Then click on the next button.

Provide a name and description for the role. Optionally, add tags to your IAM role for better organization and management.

Review everything then click on Create role.

Configure AWS Systems Manager Parameter Store

To securely manage and access Docker credentials (such as username, password, and registry URL) within AWS environments, the AWS Systems Manager Parameter Store offers a robust solution. By storing these sensitive values as SecureString parameters, AWS ensures they are encrypted using AWS Key Management Service (KMS), safeguarding the confidentiality and integrity of your credentials.

This setup is essential for integrating Docker-based workflows with AWS services like CodeBuild and CodePipeline, enabling seamless and secure Continuous Integration and Continuous Delivery (CI/CD) pipelines.

Navigate to the AWS Systems Manager Console.

Click on Parameter Storein the left-hand menu.

Click on the Create parameterbutton.

Enter Parameter Details:

Name: Enter a name for your parameter. Let’s start with the Docker registry URL. Use the following name: /myapp/docker-registry/url.

Description (Optional): Optionally, provide a description to document the purpose of the parameter.

Tier: For the tier, select Standard.

Type: For the type, select String.

Data Type: Under Data Type, select Text.

Value: For the value, enter docker.io. Finally, click Create Parameter.

Now, let’s create the parameter for the Docker registry password:

Name: Enter /myapp/docker-credentials/password as the name.

Type: Choose SecureString as the parameter type. This ensures that the parameter value is encrypted using an AWS Key Management Service (KMS) key for secure storage.

KMS Key (Optional): Optionally, you can select a specific AWS KMS key to encrypt the parameter value. If none is specified, AWS will use the default KMS key.

Value: Enter your Docker registry password. Finally, click Create Parameter to save.

Use the same process to create the parameter for the Docker registry username:

Name: Enter /myapp/docker-credentials/username.

Type: For the type, select Text.

Finally, click Create Parameter.

You should now have three parameters in total:

/myapp/docker-registry/url

/myapp/docker-credentials/password

/myapp/docker-credentials/username

These Docker credentials are securely stored in AWS Systems Manager Parameter Store.

AWS CodeBuild

Navigate to the AWS CodeBuild Console.

Click on the Create Build Projectbutton.

Enter a name for your CodeBuild project.

Choose the Source Provider where your build will fetch the code. This could be one of the following:

AWS CodeCommit, GitHub, Bitbucket, and Amazon S3 Next, specify the branch, tag, or revision ID that represents the source code version you want to build.

Define the Build Environment by specifying the following details:

Operating System: Choose the operating system for your build environment (e.g., Amazon Linux, Ubuntu,).Docker Configuration (if applicable): If your build requires Docker, configure it by selecting the appropriate Docker image or enabling Docker in your environment.

Choose an Existing Service Role: From the dropdown menu, select the IAM role you created in the previous steps.

This role should have the necessary permissions, such as SSM Full Access, to access the Docker credentials stored in AWS Systems Manager, as well as any other required resources.

Enable Privileged Mode: Ensure you check the Privileged box to allow the build environment to run Docker commands or perform elevated tasks.

Choose between the following options for your build specification:

Buildspec File in Source Repository: Select this option if your buildspec.yml file is stored within your source code repository. The build process will reference this file directly.

Buildspec in Console: Alternatively, you can provide the buildspec directly in the console by entering the required build

commands and configurations manually.

paste in the following build command in the buildspec editor section.

 version: 0.2

env:
  parameter-store:
    DOCKER_REGISTRY_USERNAME: /myapp/docker-credentials/username
    DOCKER_REGISTRY_PASSWORD: /myapp/docker-credentials/password
    DOCKER_REGISTRY_URL: /myapp/docker-registry/url
phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo "Installing dependencies..."
      - pip install -r day-14/simple-python-app/requirements.txt
  build:
    commands:
      - echo "Running tests..."
      - cd day-14/simple-python-app/
      - echo "Building Docker image..."
      - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin "$DOCKER_REGISTRY_URL"
      - docker build -t "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest" .
      - docker push "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest"
  post_build:
    commands:
      - echo "Build completed successfully!"
artifacts:
  files:
    - '**/*'
  base-directory: day-14/simple-python-app

Configure where build logs should be stored for monitoring and troubleshooting.

Specify the build artifacts to be generated and where they should be stored (e.g., S3 bucket).

Review the configuration details to ensure accuracy. Click on Create Build Project to create the CodeBuild project.

The CodeBuild project has now been created and is ready to execute builds based on the specified configuration.

Click Start Build to initiate the build process.

Use the Build Details section to monitor the status of the build.

By following these steps, you can create an AWS CodeBuild project that integrates with your source code repository and is configured to build Docker images or execute other build tasks according to your project requirements.

Verify the Image Was Pushed to the Docker Hub

Log in: Sign in to your Docker Hub account.

Navigate to Repository: Go to the repository where you expected the image to be pushed.

Verify Image: Check that the image is present, has the correct tag, and was pushed recently.

By following these steps, you can confirm that your Docker image was successfully pushed to Docker Hub, ensuring that the build process is completed correctly.

AWS CodePipeline

In this step, we will set up an AWS CodePipeline to automate the continuous integration (CI) process. AWS CodePipeline is a CI/CD service that automates the steps required to release software changes. It orchestrates various stages, including source retrieval, building, testing, and deployment.

For this article, we will focus on the CI aspect, ensuring that every code change is automatically built and tested.

Open the AWS Management Console.

Navigate to the AWS CodePipeline service.

Click on Create Pipeline to start the pipeline creation wizard.

For creation options select Build Custom Pipeline then click the next button.

Give your pipeline a name.

New Service Role: Select this option to allow AWS to create a new IAM role for the pipeline.

Existing Service Role: Choose this option if you have a preconfigured IAM role that you want to use for the pipeline.

Source Provider: Select the source provider for your code repository (e.g., AWS CodeCommit, GitHub, Bitbucket).

Repository Name: Enter the name of your repository. Specify the branch you want to use (e.g., main or master).

Choose AWS CodeBuild: Select AWS CodeBuild as the build provider.

Choose the region where your CodeBuild project is located.

Select the CodeBuild project you created earlier. Add any necessary environment variables required for the build process.

Skip Code Deploy: Since this guide focuses on Continuous Integration (CI), you may not need a deploy stage at this time. Review all the stages and configurations to ensure everything is set up correctly.

Scroll down and click on Create Pipelineto finalize and create your CodePipeline.

Trigger the Pipeline: Manually trigger the pipeline by clicking Release Change, or make code changes to your GitHub repository to automatically initiate the pipeline. Monitor the progress of the pipeline in the CodePipeline console. Each stage (Source, Build) will display its status.

Our CodePipeline has been successfully set up, eliminating the need for manual build triggers. Now, the entire process is orchestrated by CodePipeline, which automatically detects changes in our GitHub repository. Once changes are detected, CodePipeline triggers the CodeBuild service to compile and build the code. Additionally, if you check Docker Hub, you’ll find that the latest image has been automatically pushed to our repository.

Conclusion

In this project, we successfully set up a Continuous Integration (CI) pipeline using AWS CodePipeline and AWS CodeBuild, with secure Docker credential management handled through AWS Systems Manager. We leveraged AWS developer tools, including CodeBuild and CodePipeline, to streamline the process.

Thank you for reading! Stay tuned for more updates, and don’t forget to complete the Continuous Deployment (CD) part. You can find the CD setup here.

To explore more of my projects, click this button.