Get Started

Prerequisites

Before you begin, ensure you have the following tools installed on your local machine:

• Git

• Python 3.12

• AWS CLI

• Terraform (v1.0+)

• Terragrunt: Wrapper for managing Terraform configurations

• Docker

• Make

• uv: Python package management tool

AWS Credentials

Set up your AWS credentials in the ~/.aws/credentials file:

[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Don't forget to include the region in your profile

Your AWS user should have permission listed in this example policy file (less privilege).

Snowflake Credentials

Configure your Snowflake credentials in the ~/.snowflake/config file:

[YOUR_PROFILE]
organizationname=
accountname=
user=
password=
role=

Your Snowflake user should have the necessary permissions to manage databases, warehouses, and other Snowflake resources within your environment.

This SQL script provides the minimal grants required.

Quick Start

For a quick start with local deployment and local Terraform state:

# Set your AWS and Snowflake profiles and environment name
export AWS_PROFILE=<your_aws_profile>
export SNOWFLAKE_PROFILE=<your_snowflake_profile>
export ENVIRONMENT=<environment>

# Deploy the infrastructure and Docker images
make deploy

This command will:

1. Deploy all Terraform modules in the correct order

2. Build and push Docker images for the ingestion and transformation pipelines

3. Create and configure Snowflake resources (databases, schemas, warehouses)

Verify Your Deployment

After deployment completes:

1. Navigate to the AWS Step Functions service

2. Find your pipeline's step function (e.g., dev-chess-step-function)

3. Execute the step function with an empty payload

4. Monitor the execution to verify the pipeline runs successfully

5. Check your Snowflake database to verify the data has been loaded successfully

Next Steps

After your initial deployment, you might want to:

1. Add a New Pipeline - Create your own data pipeline

2. CI Deployment - Set up production deployment with CI/CD

3. FAQ - Find answers to frequently asked questions