githubEdit

circle-playGet Started

Prerequisites

Before you begin, ensure you have the following tools installed on your local machine:

  • Git

  • Python 3.12

  • AWS CLI

  • Terraform (v1.0+)

  • Terragrunt: Wrapper for managing Terraform configurations

  • Docker

  • Make

  • uv: Python package management tool

AWS Credentials

Set up your AWS credentials in the ~/.aws/credentials file:

[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
circle-exclamation
circle-info

Your AWS user should have permission listed in this example policyarrow-up-right file (less privilege).

Snowflake Credentials

Configure your Snowflake credentials in the ~/.snowflake/config file:

circle-info

Your Snowflake user should have the necessary permissions to manage databases, warehouses, and other Snowflake resources within your environment.

This SQL scriptarrow-up-right provides the minimal grants required.

Quick Start

For a quick start with local deployment and local Terraform state:

This command will:

  1. Deploy all Terraform modules in the correct order

  2. Build and push Docker images for the ingestion and transformation pipelines

  3. Create and configure Snowflake resources (databases, schemas, warehouses)

Verify Your Deployment

After deployment completes:

  1. Navigate to the AWS Step Functions service

  2. Find your pipeline's step function (e.g., dev-chess-step-function)

  3. Execute the step function with an empty payload

  4. Monitor the execution to verify the pipeline runs successfully

  5. Check your Snowflake database to verify the data has been loaded successfully

Next Steps

After your initial deployment, you might want to:

  1. Add a New Pipeline - Create your own data pipeline

  2. CI Deployment - Set up production deployment with CI/CD

  3. FAQ - Find answers to frequently asked questions

Last updated