Our simple approach to AWS-powered CI/CD in monorepo

One big challenge in a monorepo is to determine which components need to be rebuilt and consequently – deployed. The market offers us a couple of solutions, with Bazel ahead. All of them force repository owner (and users) to make enormous adaptations to build a system. Long-term earnings from using unified system pays-off, but could we incorporate simpler approach that would do just as well  in a smaller environment?

Workflow can be approximated by the following steps:

1. Feature work is done on feature branch
2. PR to master branch is created
3. PR is approved and merged
4. Each component directory is checked for changes (with sub-directories filtering)
   1. Filtered sub-directories are ignored
5. Component for which the change is detected is marked as rebuildable
   1. For a rebuildable component, marker file is placed in S3 bucket
   2. CodePipeline listens to S3 events and starts rebuilt
6. If build succeeds, new artifact (Docker image) is created and the version tag is placed in another S3 bucket
   1. Component is marked for deployment
   2. CodePipeline listens to S3 events and starts deployment
7. CodePipeline uses version tag from S3 bucket to create new release
   1. Blue/green is done to replace old version

Let’s elaborate on the highlighted points.

To check if component is buildable, we compare two repository revisions:

• One that was just checked into the master
• One that was successfully built last time

git diff-index --name-only "${sha1}" -- "${component_dir}"

Command is issued on master branch checked and is compared to previously found revision of last successful build. The output of command is a filtered list of files that have changed. When diff is non-empty, list is zipped and uploaded to S3.

Build CodePipeline is started when change to S3 file is detected. If the build succeeds, Docker image is pushed and another S3 file with version tag is uploaded to a separate S3 bucket. File contains just a precise version of tagged Docker image:

0.1.4-7fb7da8

CodeBuild uses freshly uploaded S3 file to determine the version tag of Docker image that has to be released:

export TAG="$(cat "$CODEBUILD_SRC_DIR_VersionArtifact/$REF_NAME")"

make create_component_name

make release_component_name

Where create_* and release_* targets are used correspondingly to register new TaskDefinition and new TaskSet.

And that’s it!