CI Deployment
This guide outlines the essential steps for deploying the AWS Data Stack Template in a production environment.
1. Create Dedicated Users
A best practice when deploying with Terraform is to create dedicated credentials that Terraform will use during the deployment.
Terraform should only use these users and have the minimal rights required.
For AWS, we provide ready-to-use policies and scripts to create users with your admin account quickly.
cd init/
export AWS_PROFILE=<YOUR AWS ADMIN PROFILE>
make create-tf-user-aws env=<environemnt> aws_region=<aws_region>
This script will:
create a new user called
<ENVIRONMENT>_AWS_ADMIN
assign him this policy
create files
.env.<environment>.secrets
and.env.<environment>.variables
2. CI/CD Pipeline Setup
The default version of the template does not contain a CICD.
To add it, run:
# Remove the boringdata's internal test workflow
rm .github/workflows/boringdata-test.yml
# Initialize GitHub workflows for CI/CD
# This will create a .github/workflows/ci.yml file with AWS deployment configuration
uvx boringdata github init --template-type aws
# Initialize Terragrunt configuration to use S3 remote state
# This will create/update the root.hcl file in the live/ directory
uvx boringdata terragrunt init --output-folder live
The GitHub Actions workflow requires AWS credentials to deploy the project.
You must, therefore, create the necessary variables and secrets in your GitHub repository.
If you have the GitHub CLI installed and are authorized for your repository, run the following commands from the project root:
cd init/
make github-ci-setup repo=<github account>/<repo> env=<your environemnt>
This command will automatically create the required variables in GitHub based on your AWS profile and the .env files you previously created.
Alternatively, you can manually set them up in the GitHub console.
That's it; you are now ready to deploy.
3. Set Up Terraform State S3 Bucket
You must use a dedicated S3 bucket to store the Terraform state for production deployment.
To create the bucket, run the following command:
cd init/
export AWS_PROFILE=<your-aws-profile>
make create-tf-bucket env=<environemnt> aws_region=<aws_region>
This command will:
Create a new S3 bucket named
<environment>-<aws-region>-terraform-state-bucket
.Configure the appropriate bucket policies and enable encryption and versioning.
If you have deployed the template using the Quick Start guide (with a local state).
You can either:
Destroy and start fresh
terragrunt run-all destroy
Migrate the state:
export AWS_REGION=<bucket_region>
export ENVIRONMENT=<env>
terragrunt run-all init -migrate-state -input=true
4. Deployment
The CI pipeline runs on every merge to the main branch and deploys to the environment defined in the variables.
The CI pipeline will start automatically once you push your changes to the repository.

The CI pipeline consists of two jobs:
Terragrunt-apply: Deploys the infrastructure using Terragrunt
Deploy-dockers: Builds and deploys Docker containers in
pipelines/ingest
andpipelines/transform
.Only the folders with changes will be processed if the CI pipeline runs after a merge.Perform schema migration: it runs the command
make migrate
in allpipelines/ingest/*-schema
folders. (see Iceberg Landing Table Schema Evolution)
Verify the Deployment
After deployment is complete, verify the setup in your AWS console:
Navigate to the AWS Step Functions service
Locate your pipeline's step function (e.g.,
prod-chess-step-function
)Execute the step function with an empty payload
Monitor the execution to ensure the pipeline runs successfully

Last updated