Get Started

Prerequisites

Before you begin, ensure you have the following tools installed on your local machine:

• Git

• Python 3.12

• AWS CLI

• Terraform (v1.0+)

• Terragrunt: Wrapper for managing Terraform configurations

• Docker

• Make

• uv: Python package management tool

AWS Credentials

Set up your AWS credentials in the ~/.aws/credentials file:

[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Don't forget to include the region in your profile

Your AWS user should have permission listed in this example policy file (less privilege).

Quick Start

For a quick start with local deployment and local Terraform state:

# Set your AWS profile and environment name
export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>

# Deploy the infrastructure and Docker images
make deploy

This command will:

1. Deploy all Terraform modules in the correct order

2. Build and push Docker images for the ingestion and transformation pipelines

Verify Your Deployment

After deployment completes:

1. Navigate to the AWS Step Functions service

2. Find your pipeline's step function (e.g., dev-chess-step-function)

3. Execute the step function with an empty payload

4. Monitor the execution to verify the pipeline runs successfully

Next Steps

After your initial deployment, you might want to:

1. Add a New Pipeline - Create your own data pipeline

2. CI Deployment - Set up production deployment with CI/CD

3. FAQ - Find answers to frequently asked questions