CI Deployment

This guide outlines the essential steps for deploying the AWS Data Stack Template in a production environment.

1. Create Dedicated Users

A best practice when deploying with Terraform is to create dedicated credentials that Terraform will use during the deployment.

Terraform should only use these users and have the minimal rights required.

For AWS, we provide ready-to-use policies and scripts to create users with your admin account quickly.

You will notice that these users are environment-specific. Each Terraform can only deploy to one environment. What is an "environment" ?

cd init/
export AWS_PROFILE=<YOUR AWS ADMIN PROFILE>
make create-tf-user-aws env=<environemnt> aws_region=<aws_region>

This script will:

  • create a new user called <ENVIRONMENT>_AWS_ADMIN

  • assign him this policy

  • create files .env.<environment>.secrets and .env.<environment>.variables

2. CI/CD Pipeline Setup

The default version of the template does not contain a CICD.

To add it, run:

# Remove the boringdata's internal test workflow
rm .github/workflows/boringdata-test.yml

# Initialize GitHub workflows for CI/CD
# This will create a .github/workflows/ci.yml file with AWS deployment configuration
uvx boringdata github init --template-type aws

# Initialize Terragrunt configuration to use S3 remote state
# This will create/update the root.hcl file in the live/ directory
uvx boringdata terragrunt init --output-folder live

The GitHub Actions workflow requires AWS credentials to deploy the project.

You must, therefore, create the necessary variables and secrets in your GitHub repository.

If you have the GitHub CLI installed and are authorized for your repository, run the following commands from the project root:

cd init/
make github-ci-setup repo=<github account>/<repo> env=<your environemnt>

This command will automatically create the required variables in GitHub based on your AWS profile and the .env files you previously created.

Alternatively, you can manually set them up in the GitHub console.

That's it; you are now ready to deploy.

3. Set Up Terraform State S3 Bucket

You must use a dedicated S3 bucket to store the Terraform state for production deployment.

To create the bucket, run the following command:

cd init/
export AWS_PROFILE=<your-aws-profile>
make create-tf-bucket env=<environemnt> aws_region=<aws_region>

This command will:

  • Create a new S3 bucket named <environment>-<aws-region>-terraform-state-bucket.

  • Configure the appropriate bucket policies and enable encryption and versioning.

If you have deployed the template using the Quick Start guide (with a local state).

You can either:

  • Destroy and start fresh

terragrunt run-all destroy

  • Migrate the state:

export AWS_REGION=<bucket_region>

export ENVIRONMENT=<env>

terragrunt run-all init -migrate-state -input=true

4. Deployment

The CI pipeline runs on every merge to the main branch and deploys to the environment defined in the variables.

The CI pipeline will start automatically once you push your changes to the repository.

GitHub CI

The CI pipeline consists of two jobs:

  • Terragrunt-apply: Deploys the infrastructure using Terragrunt

  • Deploy-dockers: Builds and deploys Docker containers in pipelines/ingest and pipelines/transform.Only the folders with changes will be processed if the CI pipeline runs after a merge.

  • Perform schema migration: it runs the command make migrate in all pipelines/ingest/*-schema folders. (see Iceberg Landing Table Schema Evolution)

Verify the Deployment

After deployment is complete, verify the setup in your AWS console:

  1. Navigate to the AWS Step Functions service

  2. Locate your pipeline's step function (e.g., prod-chess-step-function)

  3. Execute the step function with an empty payload

  4. Monitor the execution to ensure the pipeline runs successfully

PreviousAdd a New PipelineNextFAQ

Last updated