Get Started

Prerequisites

Before you begin, ensure you have the following tools installed on your local machine:

  • Git

  • Python 3.12

  • AWS CLI

  • Terraform (v1.0+)

  • Terragrunt: Wrapper for managing Terraform configurations

  • Docker

  • Make

  • uv: Python package management tool

AWS Credentials

Set up your AWS credentials in the ~/.aws/credentials file:

[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Don't forget to include the region in your profile

Your AWS user should have permission listed in this example policy file (less privilege).

Quick Start

For a quick start with local deployment and local Terraform state:

# Set your AWS profile and environment name
export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>

# Deploy the infrastructure and Docker images
make deploy

This command will:

  1. Deploy all Terraform modules in the correct order

  2. Build and push Docker images for the ingestion and transformation pipelines

Verify Your Deployment

After deployment completes:

  1. Navigate to the AWS Step Functions service

  2. Find your pipeline's step function (e.g., dev-chess-step-function)

  3. Execute the step function with an empty payload

  4. Monitor the execution to verify the pipeline runs successfully

Next Steps

After your initial deployment, you might want to:

  1. Add a New Pipeline - Create your own data pipeline

  2. CI Deployment - Set up production deployment with CI/CD

  3. FAQ - Find answers to frequently asked questions

PreviousKey ConceptsNextpipelines/

Last updated