# Boring Data
Hey It's [Julien](https://www.linkedin.com/in/julienhuraultanalytics/):clap:
I'm a data engineer specialized in building data platforms.
For the last 10 years implementing data stacks, I noticed I was repeatedly performing the same tasks: setting up dbt, configuring Snowflake, and, more recently, migrating to Iceberg data lakes.
For each project we spent precious time reinventing the wheel, building the stack from scratch.
That's why I built Boring Data: to help data teams reduce migration risks and easily adopt the latest data infrastructure innovation.
Don't waste time reinventing the wheel..
PS: I share insights with over 5000 readers every week on [Substack](https://juhache.substack.com/) and [LinkedIn](https://www.linkedin.com/in/julienhuraultanalytics/).
***
## What do you get with Boring Data?
### 1 - Templates
Each stack is composed of:
* a data stack built in Terraform, ready to be deployed on Github Action
* an example end-to-end pipeline that you can easily duplicate
* a doc explaining in detail the template structure and how to add new pipelines
{% content-ref url="/spaces/ryeUyIxiKpsTawnfUoTV" %}
[Template: AWS+Iceberg](https://docs.boringdata.io/template-aws-iceberg/)
{% endcontent-ref %}
{% content-ref url="/spaces/MV8jwUDrYLitfvOBJqeO" %}
[Template: AWS+Snowflake](https://docs.boringdata.io/template-aws-snowflake/)
{% endcontent-ref %}
## 2- boringdata CLI
This CLI helps you to build custom pipelines faster by generating boilerplate code in a template.
With it, you can integrate in one command:
* data ingestion tools
* data transformation frameworks
* orchestrators
* AWS resources
{% content-ref url="/pages/8s3u81CjEhZh9dm8vgIi" %}
[CLI](/reference/cli)
{% endcontent-ref %}
## How to get started?
Choose a template of your choice and follow the documentation provided.
You will get details on how the template is structured, how to deploy it, and how to add a new pipeline.
***
Support:
Reach out to me on [LinkedIn](https://www.linkedin.com/in/julienhuraultanalytics/) or .
# CLI
The BoringData CLI is a tool for generating boilerplate code and adding integration/pipelines to your stack.
This document provides a comprehensive overview of all available commands.
## Installation
{% tabs %}
{% tab title="SSH GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install git+ssh://git@github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% tab title="HTTPS GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install https://github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% endtabs %}
## Table of Contents
* [AWS Commands](#aws-commands)
* [DBT Commands](#dbt-commands)
* [DLT Commands](#dlt-commands)
* [Snowflake Commands](#snowflake-commands)
* [Project Commands](#project-commands)
* [Terragrunt Commands](#terragrunt-commands)
* [GitHub Commands](#github-commands)
## AWS Commands
Commands for managing AWS resources with BoringData.
### `aws bucket`
Create a new S3 bucket configuration with versioning and encryption.
```bash
boringdata aws bucket [--output-folder ]
```
**Arguments:**
* `bucket-name`: Name of the S3 bucket to create (required)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata aws bucket my-data-bucket
boringdata aws bucket my-data-bucket --output-folder pipelines/
```
**Files Created:**
```
.
└── _bucket.tf # Main bucket configuration
```
### `aws lambda`
Create a new AWS Lambda function with optional triggers.
```bash
boringdata aws lambda [--output-folder ] [--trigger ]
```
**Arguments:**
* `lambda-name`: Name of the Lambda function to create (required)
* `--output-folder`: Directory where files will be created (default: current directory)
* `--trigger`: Comma-separated list of triggers to enable. Options: 'sqs', 'cron'
**Example:**
```bash
boringdata aws lambda my-function
boringdata aws lambda my-function --trigger sqs,cron
boringdata aws lambda my-function --output-folder ./infrastructure
```
**Files Created:**
```
.
├── _lambda.tf # Lambda function configuration
└── -lambda/ # Lambda function code
├── .env.example # Environment variables template
├── .gitignore # Git ignore file
├── Dockerfile # Lambda container definition
├── requirements.txt # Python dependencies
└── lambda_handler.py # Lambda function code
```
### `aws step-function`
Create a new AWS Step Function configuration for orchestration.
```bash
boringdata aws step-function [--source-name ] [--dbt-command ] [--output-folder ]
```
**Arguments:**
* `type`: Type of step function to create (required). Options: lambda-dbt
* `--source-name`: Name of the source Lambda/ECS task (required for lambda-dbt and ecs-dbt types)
* `--dbt-command`: DBT command to execute (default: "run")
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata aws step-function lambda-dbt --source-name my-source --dbt-command "run --select tag:daily"
```
**Files Created:**
```
.
├── _step_function.tf # Step function configuration
└── orchestrate/ # Step function definitions
└── _step_function.json # Step function state machine
```
## DBT Commands
Commands for managing dbt projects with BoringData.
### `dbt init`
Initialize a new dbt project with configuration.
```bash
boringdata dbt init [--output-folder ] [--target ]
```
**Arguments:**
* `--output-folder`: Directory where files will be created (default: current directory)
* `--target`: Target of the project: snowflake or athena (default: snowflake)
**Example:**
```bash
boringdata dbt init
boringdata dbt init --output-folder ./transform --target athena
```
**Files Created:**
```
.
├── ecs_task_dbt.tf # ECS task definition for dbt
└── transform/ # dbt project directory
├── ...
├── Dockerfile # Container definition for dbt
├── Makefile # Common dbt commands
├── dbt_project.yml # dbt project configuration
├── macros/ # Custom macros
├── sources/ # Source models (bronze)
└── models/ # dbt models
├── marts/ # Business-layer models (gold)
└── staging/ # Staging-layer models (silver)
```
### `dbt import-source`
Import sources and generate corresponding dbt models.
```bash
boringdata dbt import-source --source [--output-folder ] [--schema-name ] [--target ]
```
**Arguments:**
* `--source`: Path to the source YAML file or folder \-schema (required)
* `--output-folder`: Directory where files will be created (default: current directory)
* `--schema-name`: Name of the DB schema where sources are stored (default: LANDING)
* `--target`: Target of the dbt project (snowflake, athena)
**Example:**
```bash
boringdata dbt import-source --source ./sources/my_source.yml
boringdata dbt import-source --source ./sources/my_source-schema --target athena
```
**Files Created:**
```
.
├── sources/ # Source definitions
│ └── .yml # Source configuration
└── models/ # Generated models
└── staging/ # Staging models
└── / # Source-specific models
├── stg__.sql # Generated staging models
└── ... # Additional models as needed
```
## DLT Commands
Commands for managing DLT pipelines with BoringData.
### `dlt add-source`
Add a new DLT source using Lambda function.
```bash
boringdata dlt add-source [--source-name ] [--destination ] [--output-folder ]
```
**Arguments:**
* `connector-name`: Name of the DLT connector to use (required)
* `--source-name`: Name of your source, defaults to connector\_name
* `--destination`: Destination to use for the lambda (s3 or iceberg) (default: s3)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata dlt add-source chess
boringdata dlt add-source chess --source-name my_chess --destination iceberg
boringdata dlt add-source chess --output-folder ./pipelines
```
**Files Created:**
```
.
├── _lambda.tf # Lambda function configuration
└── ingest/ # Lambda function code
└── -ingestion/ # Source-specific Lambda
├── .dlt/ # DLT configuration
├── .env.example # Environment variables template
├── .env.local # Local environment variables
├── Dockerfile # Lambda container definition
├── Makefile # Common commands
├── lambda_handler.py # Lambda function code
├── requirements.txt # Python dependencies
└── requirements-dev.txt # Development dependencies
```
### `dlt get-schema`
Get the destination table schema of the DLT pipeline.
```bash
boringdata dlt get-schema [--engine ] [--target ] [--output-folder ]
```
**Arguments:**
* `source-name`: Name of your source (required)
* `--engine`: Target typing format: arrow or snowflake (default: arrow)
* `--target`: Target output format: yaml or pyiceberg (default: yaml)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata dlt get-schema my-source
boringdata dlt get-schema my-source --engine snowflake --target yaml
boringdata dlt get-schema my-source --engine arrow --target pyiceberg
boringdata dlt get-schema my-source --output-folder ./schemas
```
**Files Created (for yaml target):**
```
.
└── _source_schema.yml # Data contract for the DLT pipeline
```
**Files Created (for pyiceberg target):**
```
.
└── -schema/ # Schema migration directory
├── Makefile # Makefile to run schema migration
└── _.py # Migration script for each table
```
## Snowflake Commands
Commands for managing Snowflake resources with BoringData.
### `snowflake tf-module`
Generate a Terraform Snowflake module for database and warehouse management.
```bash
boringdata snowflake tf-module [--output-folder ]
```
**Arguments:**
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata snowflake tf-module
boringdata snowflake tf-module --output-folder ./modules/snowflake
```
**Files Created:**
```
.
├── data.tf # Data sources
├── db.tf # Database configuration
├── locals.tf # Local variables
├── schema.tf # Schema configuration
├── tech_user.tf # Technical user configuration
├── versions.tf # Provider and terraform versions
└── warehouse.tf # Warehouse configuration
```
## Project Commands
Commands for managing project-level resources with BoringData.
### `project init`
Initialize a new BoringData project with infrastructure as code.
```bash
boringdata project init [--local-state] [--output-folder ]
```
**Arguments:**
* `project-type`: Type of project to create (required). Options: 'aws', 'aws-snowflake'
* `--local-state`: Use local state instead of remote state in S3 (default: remote S3 state)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata project init aws
boringdata project init aws-snowflake --local-state
boringdata project init aws --output-folder ./my-project
```
**Files Created:**
```
.
├── README.md # Project documentation
├── Makefile # Common commands
├── .gitignore # Git ignore file
├── base/ # Base infrastructure
│ └── aws/ # AWS-specific base
│ └── main.tf # Main configuration
│ └── snowflake/ # Snowflake-specific base
│ └── main.tf # Main configuration
├── pipelines/ # Data pipeline infrastructure
│ └── infra/ # Infrastructure configuration
│ └── aws/ # AWS pipeline resources
│ └── main.tf # Main configuration
└── live/ # Terragrunt configurations
├── root.hcl # Root Terragrunt configuration
├── base/ # Base infrastructure live configs
│ ├── aws/ # AWS base live config
│ │ └── terragrunt.hcl # Terragrunt configuration
│ └── snowflake/ # Snowflake base live config
│ └── terragrunt.hcl # Terragrunt configuration
└── pipelines/ # Pipeline live configs
└── infra/ # Infrastructure live configs
└── aws/ # AWS pipeline live config
└── terragrunt.hcl # Terragrunt configuration
```
## Terragrunt Commands
Commands for managing Terragrunt configurations with BoringData.
### `terragrunt init`
Initialize a new Terragrunt directory by generating the root.hcl file.
```bash
boringdata terragrunt init [--local-state] [--output-folder ]
```
**Arguments:**
* `--local-state`: Use local state instead of remote state in S3 (default: False)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata terragrunt init
boringdata terragrunt init --local-state
boringdata terragrunt init --output-folder live/
```
**Files Created:**
```
.
└── root.hcl # Root Terragrunt configuration
```
### `terragrunt add-module`
Add a new Terragrunt module configuration.
```bash
boringdata terragrunt add-module --module-path [--output-folder ]
```
**Arguments:**
* `--module-path`: Relative path to the module from --output-folder (required)
* `--output-folder`: Directory where terragrunt.hcl file will be created (default: current directory)
**Example:**
```bash
boringdata terragrunt add-module --module-path ../../../base/aws --output-folder live/base/aws
```
**Files Created:**
```
.
└── terragrunt.hcl # Module Terragrunt configuration
```
## GitHub Commands
Commands for managing GitHub workflows with BoringData.
### `github init`
Initialize GitHub workflows for CI/CD.
```bash
boringdata github init [--template-type ] [--output-folder ]
```
**Arguments:**
* `--template-type`: Type of template to use (aws or aws-snowflake) (default: aws)
* `--output-folder`: Directory where files will be created (default: current directory)
**Example:**
```bash
boringdata github init
boringdata github init --template-type aws-snowflake
boringdata github init --output-folder ./my-project
```
**Files Created:**
```
.
└── .github/
└── workflows/
└── ci.yml # CI workflow
```
# Roadmap
We’re constantly working to expand and improve our templates.
Here’s what you can expect soon
## 🆕 New Templates
• “Sovereign Stack”: MinIO + DuckDB + Iceberg
• GCP + BigQuery Template
• Azure + Snowflake Template
## 🔗 New Integrations
• Orchestrators: Airflow, Dagster
• ELT Tools: CloudQuery, Airbyte
• SQLMesh
• BI Tools: Rill, Metabase
## 🛠️ DevOps Features
• Alerting Model for proactive monitoring
• Git Branching Model
• FinOps Model to optimize costs and resource management
💬 We’d love your feedback!
Let us know which integrations and tools would benefit you most.
👉 Reach out on [LinkedIn](https://www.linkedin.com/) or email me at .
# Overview
Welcome to the AWS + Snowflake Data Stack Template!
This template provides everything you need to build a modern data platform on AWS and Snowflake.
The infrastructure is fully automated with Terraform and includes:
* An example ingestion pipeline: dlt + lambda feeding Snowflake
* A dbt project for transformations in Snowflake
* AWS Step Functions to orchestrate ingestion + transformation
* A GitHub CI workflow
The template is designed to be fully modular and customizable.
You can easily swap out any component with your preferred tools:
* Replace dlt with any other ingestion framework
* Use an alternative to dbt for transformations
* Switch to a different orchestration service instead of Step Functions
***
## Next Steps
1. [Key Concepts](/template-aws-snowflake/introduction/key-concepts) - Understand the core architecture and components
2. [Get Started](/template-aws-snowflake/introduction/get-started) - Set up your environment and run your first deployment
# Key Concepts
Understand template's structure
This section explains the core concepts and architecture of this template.
## Code Structure
The template's code is organized into three main components:
```
📁
├── 📁 pipelines/ # Data pipelines:
│ ├── 📁 ingest/ # Data ingestion layer
│ ├── 📁 transform/ # Data transformation layer
│ └── 📁 orchestrate/ # Workflow orchestration layer
│
├── 📁 base/ # Cloud infrastructure
│ ├── 📁 aws/ # Cloud provider resources (VPC, IAM, etc.)
│ └── 📁 snowflake/ # Data warehouse resources
│
└── 📁 live/ # Environment-specific deployment configuration
```
Each component is documented separately here:
{% content-ref url="/pages/EJXrCe6oH8IPD55h0XcB" %}
[pipelines/](/template-aws-snowflake/project-structure/pipelines)
{% endcontent-ref %}
{% content-ref url="/pages/37HSeF65Ojjhqeds5ut8" %}
[base/aws/](/template-aws-snowflake/project-structure/aws)
{% endcontent-ref %}
{% content-ref url="/pages/jDxLt8aeBweeadmqK3AA" %}
[base/snowflake/](/template-aws-snowflake/project-structure/snowflake)
{% endcontent-ref %}
{% content-ref url="/pages/81qgGGSPoCwraBvvXfNe" %}
[live/](/template-aws-snowflake/project-structure/live)
{% endcontent-ref %}
## Data Flow
1. Serverless function ingest data to S3
2. Snowpipes copy data from S3 into tables in Snowflake (landing tables)
3. Data transformations are applied to create staging and mart tables using SQL transformations in [dbt](https://docs.getdbt.com/)
## Data Pipeline Architecture
Our data platform follows a layered architecture:
### 1. Data Ingestion Layer
For each source, the ingestion layer is structured as follows:
```
📁 pipelines/
├── 📁 ingest/
│ ├── 📁 -ingestion/ # Core ingestion logic
│ │
│ └── _source_schema.yml # Table schema definitions (YAML)
│
└── _*.tf # Infrastructure definition (serverless functions, containers, etc.)
```
Each source has:
* A folder `pipelines/ingest/-ingestion/` containing the core ingestion logic packaged in a container
* Infrastructure as Code files in `pipelines/*tf` for deploying this ingestion container (as serverless functions (AWS Lambda) or container tasks ([Amazon ECS](https://aws.amazon.com/ecs/)))
* A YAML file `pipelines/_source_schema.yml` for the management of the data warehouse tables
{% hint style="info" %}
Schema management is handled through YAML files, making it easy to define and evolve table structures. More info in [FAQ](/template-aws-snowflake/help/faq#snowflake-schema-management)
{% endhint %}
The template comes with an example data ingestion pipeline deployed as a serverless function using [dlt](https://dlthub.com/docs/intro); more details here:
{% content-ref url="/pages/RqedU5uiYGSIFxVjYlEk" %}
[Ingestion: dlt + lambda](/template-aws-snowflake/project-structure/pipelines/chess-ingestion)
{% endcontent-ref %}
### 2. Data Transformation Layer
The transformation layer is a SQL-based project that transforms the data into analytics-ready tables using [dbt](https://docs.getdbt.com/):
This project is located in the `pipelines/transform` folder and uses [dbt](https://docs.getdbt.com/) as the transformation framework:
```
📁 pipelines/
├── 📁 transform/ # SQL transformation project
│ ├── 📁 models/
│ │ ├── 📁 staging/ # Raw table connections
│ │ └── 📁 marts/ # Transformations
│ │
│ ├── dbt_project.yml
│ └── Dockerfile # For container deployment
│
└── ecs_task_dbt.tf # Infrastructure for transformation tasks
```
This transformation project runs on container infrastructure ([Amazon ECS](https://aws.amazon.com/ecs/) Fargate) and connects directly to [Snowflake](https://www.snowflake.com/en/).
More details on how this transformation project is structured here:
{% content-ref url="/pages/BqrIPVlcMp0Rf72zx0eS" %}
[Transformation: dbt](/template-aws-snowflake/project-structure/pipelines/transform)
{% endcontent-ref %}
### 3. Workflow Orchestration Layer
The orchestration layer coordinates the execution of the ingestion and transformation layers using workflow automation.
This template proposes an example orchestration using [AWS Step Functions](https://aws.amazon.com/step-functions/):
```
📁 pipelines/
├── 📁 orchestrate/
│ └── _step_function.json # Workflow definition
│
└── _step_function.tf # Creates an orchestration workflow in [AWS Step Functions](https://aws.amazon.com/step-functions/)
```
## Deployment
This template is ready to be deployed.
The stack deployment is structured in 2 steps:
* First, the infrastructure modules (base/ and pipelines/) are deployed using [Terragrunt](https://terragrunt.gruntwork.io/) for infrastructure management
* Then, the containers for the ingestion and transformation layers are built and pushed to the container registry ([Amazon ECR](https://aws.amazon.com/ecr/))
If you want to get started quickly and deploy the template from your machine, follow this guide:
{% content-ref url="/pages/r6bH5amxRDrqdxb3yZXB" %}
[Get Started](/template-aws-snowflake/introduction/get-started)
{% endcontent-ref %}
To get started deploying from [GitHub Actions](https://github.com/features/actions) CI/CD, head there:
{% content-ref url="/pages/zlrSG5bGls3JF20DZqOS" %}
[CI Deployment](/template-aws-snowflake/guides/production-deployment)
{% endcontent-ref %}
## Makefile
The template is composed of many Makefiles providing utilities.
Here are some examples:
* `make deploy` in the root folder will deploy the template from your machine
* `make build` in a folder with a Dockerfile will build the container
* `make local-run` will run the code locally
* etc.
Everywhere you see a Makefile, run `make` and the list of possible actions will be listed.
{% content-ref url="/pages/r6bH5amxRDrqdxb3yZXB" %}
[Get Started](/template-aws-snowflake/introduction/get-started)
{% endcontent-ref %}
# Get Started
## Prerequisites
Before you begin, ensure you have the following tools installed on your local machine:
* **Git**
* **Python 3.12**
* **AWS CLI**
* **Terraform (v1.0+)**
* **Terragrunt**: Wrapper for managing Terraform configurations
* **Docker**
* **Make**
* **uv**: Python package management tool
## AWS Credentials
Set up your AWS credentials in the `~/.aws/credentials` file:
```bash
[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
```
{% hint style="warning" %}
Don't forget to include the region in your profile
{% endhint %}
{% hint style="info" %}
Your AWS user should have permission listed in this [example policy](https://github.com/boringdata/boringdata-template-aws-snowflake/blob/main/init/init_aws_tf_user_policy.json) file (less privilege).
{% endhint %}
## Snowflake Credentials
Configure your Snowflake credentials in the `~/.snowflake/config` file:
```bash
[YOUR_PROFILE]
organizationname=
accountname=
user=
password=
role=
```
{% hint style="info" %}
Your Snowflake user should have the necessary permissions to manage databases, warehouses, and other Snowflake resources within your environment.
This [SQL script](https://github.com/boringdata/boringdata-template-aws-snowflake/blob/main/init/init_snowflake_tf_user.sql) provides the minimal grants required.
{% endhint %}
## Quick Start
For a quick start with local deployment and local Terraform state:
```bash
# Set your AWS and Snowflake profiles and environment name
export AWS_PROFILE=
export SNOWFLAKE_PROFILE=
export ENVIRONMENT=
# Deploy the infrastructure and Docker images
make deploy
```
This command will:
1. Deploy all Terraform modules in the correct order
2. Build and push Docker images for the ingestion and transformation pipelines
3. Create and configure Snowflake resources (databases, schemas, warehouses)
## Verify Your Deployment
After deployment completes:
1. Navigate to the AWS Step Functions service
2. Find your pipeline's step function (e.g., `dev-chess-step-function`)
3. Execute the step function with an empty payload
4. Monitor the execution to verify the pipeline runs successfully
5. Check your Snowflake database to verify the data has been loaded successfully
## Next Steps
After your initial deployment, you might want to:
1. [Add a New Pipeline](/template-aws-snowflake/guides/add-a-pipeline) - Create your own data pipeline
2. [CI Deployment](/template-aws-snowflake/guides/production-deployment) - Set up production deployment with CI/CD
3. [FAQ](/template-aws-snowflake/help/faq) - Find answers to frequently asked questions
# pipelines/
Contents of the pipelines/ folder
Data Pipelines are built using a 2-layer architecture:
* `ingest/` for data ingestion from source to S3
* `transform/` for data transformation in Snowflake via dbt
For each layer, the application code is in a separate folder while the underlying infrastructure is defined in terraform files in pipelines/.
```
pipelines/
├── ingest/
├── transform/
├── *.tf
```
A typical data flow looks like this:
{% @mermaid/diagram content="%%{init: {'theme':'dark'}}%%
graph LR
subgraph Ingest Layer
Source --> Lambda --> S3
end
```
subgraph Transform Layer
S3 --> Snowpipes --> Snowflake+dbt
end
%% Styling
classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
classDef layer fill:#e1f7d5,stroke:#333,stroke-width:2px;
class Ingest,Transform layer;" %}
```
## Ingestion Layer
The ingest layer is composed of three artifacts:
* the ingestion code in `pipelines/ingest/{SOURCE_NAME}-ingestion/`
* the data source schema in `ingest/{SOURCE_NAME}_source_schema.yml`
* the infrastructure code (terraform) in `pipelines/*.tf`
Let's take the example of the chess.com pipeline example provided in this repo:
```
pipelines/
├── ingest/
│ ├── chess-lambda/
│ │ ├── lambda_handler.py # Lambda code embedding DLT for Chess.com ingestion
│ │ └── ...
│ └── chess_source_schema.yml. # YAML file defining the Chess.com data schema
├── chess_lambda.tf # Terraform creating the lambda function
├── ingestion_bucket.tf # Terraform creating target S3 bucket
...
```
The ingestion is done in a lambda function embedding dlt with:
* Source code in `pipelines/ingest/chess-ingestion`
* Terraform in`pipelines/chess_lambda.tf`
This lambda writes to a bucket defined in ingestion\_bucket.tf.
{% hint style="info" %}
Get more info on how to run/test this lambda [here](https://github.com/boringdata/boringdata-template-aws-snowflake/blob/main/pipelines/ingest/chess-lambda/README.md)
{% endhint %}
We maintain a YAML file for each data source `ingest/{source_name}_source_schema.yml` to track the source schema and automatically create landing tables in the Snowflake Warehouse.
{% hint style="info" %}
[Why do you need to generate a yaml for each source ?](https://docs.boringdata.io/template-aws-snowflake/project-structure/pages/mvzXlWASNIL69qnoS81s#what-are-less-than-source-greater-than-source_schema.yml-files)
{% endhint %}
## Transform Layer
The transform layer is composed of two artifacts:
* The transformation code in `transform/` (typically a dbt project)
* The infrastructure code (terraform) in `pipelines/*.tf`
### S3 -> Snowflake
`ingestion_snowpipe.tf` automatically reads all the yml files in the `ingest/` folder and creates:
* all landing tables in Snowflake
* all Snowflake's pipes to copy automatically the data from S3 to these tables
### dbt
`transform/` is a standard dbt project with models split into two folders (schemas in Snowflake):
* STAGING: for the transformed data
* MART: for the data ready to be used by the business
The dbt project is run in an ECS task (`ecs_task_dbt.tf`) .
{% hint style="info" %}
You can get more info on this project and how to run dbt locally and remotely [here](/template-aws-snowflake/project-structure/pipelines/transform)
{% endhint %}
Let's take the example of the chess pipeline provided in this repo:
```
├── pipelines/
│ ├── models/
│ │ └── staging/
│ │ └── chess/ # Chess staging models
│ │ ├── stg_chess_games.sql
│ │ ├── stg_chess_players.sql
│ │ ├── stg_chess_players_games.sql
│ │ ├── stg_chess_...
│ │
│ └── sources/
│ └── chess.yml
├── ingestion_snowpipe.tf # Terraform for Snowflake landing table + snowpipe creation
├── ecs_task_dbt.tf # Terraform for creating the ECS task running dbt in AWS
...
```
## Terraform Module
### Example Usage
```hcl
module "chess_lambda" {
source = "git::https://github.com/boringdata/boringdata-template-aws-snowflake.git//modules/chess_lambda"
environment = "prod"
vpc_name = "vpc-12345678"
ecs_cluster_name = "ecs-cluster-12345678"
}
```
### Diagram
{% @mermaid/diagram content="%%{init: {'theme':'dark'}}%%
graph TB
%% Variables
env\[environment variable] --> all
vpc\_name\[vpc\_name variable] --> vpc\_data
ecs\_name\[ecs\_cluster\_name variable] --> ecs\_data
```
%% Data Sources
vpc_data[aws_vpc data source] --> subnets
subnets[aws_subnets data source] --> ecs_task
ecs_data[aws_ecs_cluster data source] --> ecs_task
region[aws_region data source] --> all
caller[aws_caller_identity data source] --> all
%% ECR Repositories
chess_ecr[Chess ECR Repository] --> chess_lambda
dbt_ecr[DBT ECR Repository] --> ecs_task
%% Lambda Resources
chess_secrets[Chess Secrets Manager] --> chess_lambda
chess_lambda[Chess Lambda Function] --> |writes to| s3_bucket
%% S3 and Snowflake Resources
s3_bucket[Ingestion S3 Bucket] --> snowflake_stage
snowflake_stage[Snowflake External Stage] --> snowpipes
snowpipes[Snowflake Pipes] --> landing_tables[Snowflake Landing Tables]
storage_int[Storage Integration] --> snowflake_stage
%% ECS Resources
ecs_task[DBT ECS Task Definition]
%% Styling
classDef variable fill:#e1f7d5
classDef data fill:#c6e2ff
classDef resource fill:#f9d6ff
classDef storage fill:#ffebcc
class env,vpc_name,ecs_name variable
class vpc_data,subnets,ecs_data,region,caller data
class chess_secrets,chess_lambda,ecs_task resource
class chess_ecr,dbt_ecr,s3_bucket,snowflake_stage,snowpipes,landing_tables,storage_int storage" %}
```
## Requirements
| Name | Version |
| ----------------------------------- | -------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [aws](#requirement_aws) | >=5.63.1 |
| [snowflake](#requirement_snowflake) | >=1.0.0 |
## Providers
| Name | Version |
| -------------------------------- | ------- |
| [aws](#provider_aws) | 5.92.0 |
| [null](#provider_null) | 3.2.3 |
| [snowflake](#provider_snowflake) | 1.0.4 |
| [time](#provider_time) | 0.13.0 |
## Modules
| Name | Source | Version |
| --------------------------------------------------------------------------- | --------------------------------------------------------- | ------- |
| [bucket\_ingestion\_read\_policies](#module_bucket_ingestion_read_policies) | terraform-aws-modules/iam/aws//modules/iam-policy | 5.39.1 |
| [chess\_ecr](#module_chess_ecr) | terraform-aws-modules/ecr/aws | n/a |
| [chess\_lambda\_function](#module_chess_lambda_function) | terraform-aws-modules/lambda/aws | 7.2.1 |
| [chess\_pipeline](#module_chess_pipeline) | terraform-aws-modules/step-functions/aws | 4.2.1 |
| [chess\_secrets](#module_chess_secrets) | terraform-aws-modules/secrets-manager/aws | 1.1.2 |
| [dbt\_ecr](#module_dbt_ecr) | terraform-aws-modules/ecr/aws | n/a |
| [dbt\_task\_definition](#module_dbt_task_definition) | terraform-aws-modules/ssm-parameter/aws | 1.1.1 |
| [ecs\_task\_definition\_dbt](#module_ecs_task_definition_dbt) | terraform-aws-modules/ecs/aws///modules/service | 5.11.2 |
| [iam\_role\_assumable\_snowflake](#module_iam_role_assumable_snowflake) | terraform-aws-modules/iam/aws//modules/iam-assumable-role | 5.39.1 |
| [ingestion\_bucket](#module_ingestion_bucket) | terraform-aws-modules/s3-bucket/aws | 4.1.0 |
| [s3\_notifications](#module_s3_notifications) | terraform-aws-modules/s3-bucket/aws//modules/notification | n/a |
## Resources
| Name | Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [null\_resource.chess\_empty\_image](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
| [snowflake\_pipe.ingestion\_pipes](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/pipe) | resource |
| [snowflake\_stage.snowflake\_landing\_stage](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/stage) | resource |
| [snowflake\_storage\_integration.storage\_integration](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/storage_integration) | resource |
| [snowflake\_table.snowflake\_landing\_tables](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/table) | resource |
| [time\_sleep.wait\_for\_iam\_role](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
| [aws\_caller\_identity.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws\_ecs\_cluster.ecs-cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecs_cluster) | data source |
| [aws\_iam\_policy\_document.bucket\_ingestion\_read\_write\_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws\_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
| [aws\_subnets.subnets](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
| [aws\_vpc.vpc](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| --------------------------------------------- | -------------------------------------------------------------------- | -------- | ------- | :------: |
| [ecs\_cluster\_name](#input_ecs_cluster_name) | The name of the ECS cluster | `string` | `null` | no |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
| [vpc\_name](#input_vpc_name) | The name of the VPC to deploy the ECS cluster in | `string` | `null` | no |
## Outputs
No outputs.
# Ingestion: dlt + lambda
## Overview
This example demonstrates a serverless data ingestion pipeline that:
1. Fetches chess data from an external source and processes it using [dlt](https://dlthub.com/)
2. Writes the data to S3
The pipeline runs as an AWS Lambda function packaged in a Docker container.
## How It Works
### Infrastructure Components
* **AWS Lambda**: Executes the ingestion code on demand
* **Amazon ECR**: Stores the Docker container image
* **Amazon S3**: Temporary storage for data files before loading to Snowflake
* **AWS Secrets Manager**: Stores credentials and configuration
* **Terraform**: Provisions and manages all infrastructure
### Code Structure
```
pipelines/
├── chess_lambda.tf # Terraform creating the lambda function and ECR repository
└── ingest/
├── chess_source_schema.yml # Snowflake table schema definitions in YAML format
└── chess-ingestion/ # Lambda function code
├── Dockerfile
├── lambda_handler.py # Lambda code with DLT pipeline
└── ...
```
### Data Flow Process
The pipeline follows these steps:
1. **Extraction**: DLT extracts data from the source
2. **Transformation**: DLT performs basic transformations (typing, normalization)
3. **Loading**: DLT loads the data directly to S3
4. **Schema Management**: Table schemas are defined in YAML files and managed by the pipeline
## Development Guide
### 1. Local Development with DuckDB
For rapid iteration without AWS resources, use DuckDB as the destination:
1. Create a `.env.local` file with:
```
DESTINATION=duckdb
# Add any source-specific credentials here
```
2. Run the pipeline locally:
```bash
make run-local
```
3. Examine results in the local .duckdb database
### 2. Local Development with S3
To run the lambda with S3 as a temporary destination:
1. Configure `.env.local` with:
```
DESTINATION=filesystem
AWS_REGION=
S3_BUCKET_NAME=
AWS_PROFILE=
# Add any source-specific credentials here
```
2. Run with the same command:
```bash
make run-local
```
### 3. Testing on AWS
Once your code is deployed to AWS you can run the lambda with:
```bash
export AWS_PROFILE=
make run-lambda env=
```
### 4. VSCode Debugging
For interactive debugging, add this to `.vscode/launch.json`:
```json
{
"name": "Debug chess lambda",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/pipelines/ingest/chess-ingestion/lambda_handler.py",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}/pipelines/ingest/chess-ingestion",
"justMyCode": false
}
```
## Schema Management
Snowflake landing table schemas are defined in YAML files in the `pipelines/ingest/_source_schema.yml` file.
After running the pipeline locally, generate a source schema definition:
```bash
cd pipelines/
uvx boringdata dlt get-schema chess
```
This will generate a schema file `chess_source_schema.yml` in the pipelines folder to define the Snowflake tables.
## Manual Deployment
For manual deployment:
```bash
# Set required environment variables
export AWS_PROFILE=
# Build and deploy
make deploy env=
```
This process:
1. Builds the Docker image locally
2. Pushes it to ECR
3. Updates the Lambda to use the new image
## Common Commands
```bash
# Development
make run-local # Run locally with settings from .env.local
make run-lambda env= # Execute on AWS Lambda
# Deployment
make build env= # Build Docker image
make deploy env= # Build and deploy to ECR
# Utilities
make help # Show all available commands
```
## Resources
* [DLT Documentation](https://dlthub.com/docs/)
* [Snowflake SQL API Documentation](https://docs.snowflake.com/en/developer-guide/sql-api/index)
# Transformation: dbt
## Overview
This directory contains a data transformation pipeline that:
1. Takes data from Snowflake landing tables
2. Transforms it using [dbt](https://www.getdbt.com/) (data build tool)
3. Creates analytics-ready tables in staging and mart schemas
The pipeline runs as an AWS ECS Fargate task using a Docker container.
## How It Works
### Infrastructure Components
* **Snowflake**: Data warehouse for both source and transformed data
* **Amazon ECS**: Orchestrates the dbt container execution
* **Amazon ECR**: Stores the dbt docker container image
* **Terraform**: Provisions and manages all infrastructure
### Project Structure
```
pipelines/
├── transform/ # dbt project root
│ ├── Dockerfile
│ ├── dbt_project.yml # dbt project configuration
│ ├── sources/
│ │ ├──.yml # List all landing tables for a source
│ ├── models/
│ │ ├── staging/ # Staging models (first transformation layer)
│ │ └── mart/ # Final business-ready models
│ └── ...
└── ecs_task_dbt.tf # Terraform creating the ECS task
```
### Data Transformation Flow
The pipeline follows these transformation layers:
1. **Sources**: Raw data from landing tables created by ingestion pipelines
2. **Staging**: Initial cleaning, type conversion, deduplication and renaming
3. **Mart**: Final models organized by business domain, ready for analytics and reporting
## Sources
Sources are defined in the `models/sources/` folder and reference the landing tables created by the ingestion pipelines:
{% code title="models/sources/\.yml" %}
```yaml
sources:
- name:
schema:
tables:
- name: __dlt_version
- name: __dlt_loads
...
```
{% endcode %}
You can generate this file automatically using the BoringData CLI:
```bash
cd pipelines/transform
uvx boringdata dbt import-sources --source ../ingest/_source_schema.yml
```
## Models Structure
The dbt models follow a layered architecture pattern:
* Each folder in the `models` directory corresponds to a distinct schema in Snowflake
* `models/staging/` ➡️ `STAGING` schema in Snowflake
* `models/mart/` ➡️ `MART` schema in Snowflake
This behavior is configured in the `macros/schema_name.yml` and `dbt_project.yml` files.
## Development Guide
### Option 1: Execute dbt Locally
For rapid development with local dbt execution:
1. **Setup your environment**:
```bash
uv venv --python=python3.12
uv pip install -r requirements.txt
uv run dbt deps
```
2. **Configure dbt profile**:\
Create or update `~/.dbt/profiles.yml` with:
```yaml
local:
target:
outputs:
:
type: snowflake
account: - # stored in AWS Parameter Store //SNOWFLAKE/HOST
user:
password:
role: _TECH_USER_ROLE # Upper case!
database: _DB # Upper case!
schema: LANDING
warehouse: _WH # Upper case!
```
3. **Run dbt commands**:
```bash
export DBT_PROFILE=local
# Run a specific model
uv run dbt run --select model_name
# Run with Makefile shortcut
make run-local cmd="run --select model_name"
```
### Option 2: Execute in AWS ECS Fargate
Once your template is deployed to AWS you can run dbt in the cloud environment:
```bash
export AWS_PROFILE=
export ENVIRONMENT=
make run cmd="run"
```
This will trigger an ECS Fargate task to execute the specified dbt command and store results in Snowflake.
## Deployment
For manual deployment:
```bash
# Set required environment variables
export AWS_PROFILE=
export ENVIRONMENT=
cd pipelines/transform
# Build and deploy
make deploy
```
This process:
1. Builds the Docker image locally
2. Pushes it to ECR
The next time you trigger an ECS task, it will use the latest image.
## Common Commands
```bash
# Development
make run-local cmd="run" # Run dbt locally with specified command
make run-local cmd="test" # Run dbt tests locally
make run-local cmd="docs generate" # Generate dbt documentation
# Cloud Execution
make run cmd="run" # Run dbt in ECS Fargate
make run cmd="test" # Run tests in ECS Fargate
# Deployment
make build # Build Docker image
make deploy # Build and deploy to ECR
```
## Resources
* [dbt Documentation](https://docs.getdbt.com/)
* [Snowflake User Guide](https://docs.snowflake.com/)
* [BoringData CLI Guide](https://docs.boringdata.io/)
# base/aws/
## Overview
This Terraform module provisions the core AWS infrastructure needed for a data platform, including:
* VPC with subnets
* ECS cluster for containerized workloads
* Secrets Manager for sensitive values
* SSM Parameters for configuration
## Quick Start
```hcl
module "aws" {
source = "git::https://github.com/boringdata/boringdata-template-aws-iceberg.git//base/aws"
environment = "dev"
secrets = {
"api_key" = "your-secret-value"
}
}
```
## Key Features
* **Environment-based naming**: All resources are prefixed with your environment name
* **Secure networking**: Properly configured VPC with public and private subnets
* **Containerization**: Ready-to-use ECS cluster for your workloads
* **Configuration management**: Built-in secrets and parameters management
## Module Structure
```
aws/
├── data.tf # AWS region and availability zones
├── ecs_cluster.tf # ECS cluster configuration
├── vpc.tf # VPC and networking resources
├── secrets.tf # AWS Secrets Manager resources
├── ssm_parameters.tf # SSM Parameter Store resources
├── variables.tf # Input variables
├── outputs.tf # Output values
├── locals.tf # Local variables
└── versions.tf # Version constraints
```
## Architecture
{% @mermaid/diagram content="%%{init: {'theme':'neutral'}}%%
graph TD
env(\[Environment])
vpc\[VPC]
ecs\[ECS Cluster]
secrets\[Secrets Manager]
ssm\[SSM Parameters]
```
env --> vpc & ecs & secrets & ssm
vpc --> subnets[Public & Private Subnets]" %}
```
## Requirements
| Name | Version |
| ----------------------------------- | -------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [aws](#requirement_aws) | >=5.63.1 |
## Providers
| Name | Version |
| -------------------- | ------- |
| [aws](#provider_aws) | 5.92.0 |
## Modules
| Name | Source | Version |
| ----------------------------------- | ---------------------------------------------- | ------- |
| [ecs\_cluster](#module_ecs_cluster) | terraform-aws-modules/ecs/aws//modules/cluster | 5.11.2 |
| [parameters](#module_parameters) | terraform-aws-modules/ssm-parameter/aws | 1.1.1 |
| [secrets](#module_secrets) | terraform-aws-modules/secrets-manager/aws | 1.1.2 |
| [vpc](#module_vpc) | terraform-aws-modules/vpc/aws | \~> 5.0 |
## Resources
| Name | Type |
| --------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [aws\_availability\_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws\_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| ---------------------------------------- | -------------------------------------------------------------------- | ------------- | ------- | :------: |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
| [secrets](#input_secrets) | A map of secrets to create | `map(string)` | `{}` | no |
| [ssm\_parameters](#input_ssm_parameters) | A map of SSM parameters to create | `map(string)` | `{}` | no |
## Outputs
No outputs.
# base/snowflake/
## snowflake/
This module creates the necessary Snowflake resources for the data platform:
* Database and schemas
* Warehouse
* Technical user with appropriate permissions (used typically by dbt)
### Example Usage
```hcl
module "snowflake" {
source = "git::https://github.com/boringdata/boringdata-template-aws-snowflake.git//base/snowflake"
environment = "dev"
}
```
### Filetree
```
base/
└── snowflake/
├── data.tf # Snowflake account data sources
├── db.tf # Database definition
├── locals.tf
├── outputs.tf
├── schema.tf # Schema definitions
├── tech_user.tf # Technical user and permissions
├── variables.tf
├── versions.tf # Snowflake provider versions
└── warehouse.tf # Warehouse configuration
```
### Diagram
{% @mermaid/diagram content="%%{init: {'theme':'dark'}}%%
graph TD
%% Variables
env\[input: environment]
```
%% Data Sources
accounts[data: snowflake_accounts]
%% Resources
db[snowflake_database]
warehouse[snowflake_warehouse]
schemas[snowflake_schema]
%% Tech User Resources
tech_role[snowflake_account_role tech_user_role]
tech_user[snowflake_user tech_user]
private_key[tls_private_key tech_user_private_key]
%% Grants
grant_role_admin[snowflake_grant_account_role tech_user_role_grant_accountadmin]
grant_role_user[snowflake_grant_account_role tech_user_grant_tech_user_role]
grant_db[snowflake_grant_privileges_to_account_role tech_user_db_access]
grant_schema[snowflake_grant_privileges_to_account_role tech_user_schema_access]
grant_warehouse[snowflake_grant_privileges_to_account_role tech_user_warehouse_access]
grant_objects[snowflake_grant_privileges_to_account_role grant_tech_user_objects_all]
grant_future[snowflake_grant_privileges_to_account_role grant_tech_user_objects_all_future]
%% Outputs
out_secrets[output: secrets]
out_ssm[output: ssm_parameters]
%% Relationships
env --> db
env --> warehouse
env --> schemas
env --> tech_role
env --> tech_user
accounts --> db
tech_role --> grant_role_admin
tech_role --> grant_role_user
tech_role --> grant_db
tech_role --> grant_schema
tech_role --> grant_warehouse
tech_role --> grant_objects
tech_role --> grant_future
private_key --> tech_user
tech_user --> out_secrets
tech_user --> out_ssm
db --> grant_db
schemas --> grant_schema
warehouse --> grant_warehouse
classDef variable fill:#e1f5fe,stroke:#01579b
classDef resource fill:#e8f5e9,stroke:#2e7d32
classDef output fill:#fce4ec,stroke:#880e4f
classDef datasource fill:#fff3e0,stroke:#e65100
class env variable
class out_secrets,out_ssm output
class accounts datasource
class db,warehouse,schemas,tech_role,tech_user,private_key,grant_role_admin,grant_role_user,grant_db,grant_schema,grant_warehouse,grant_objects,grant_future resource" %}
```
## Requirements
| Name | Version |
| ----------------------------------- | ------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [snowflake](#requirement_snowflake) | >=1.0.0 |
## Providers
| Name | Version |
| -------------------------------- | ------- |
| [snowflake](#provider_snowflake) | 1.0.4 |
| [tls](#provider_tls) | 4.0.6 |
## Modules
No modules.
## Resources
| Name | Type |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [snowflake\_account\_role.tech\_user\_role](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/account_role) | resource |
| [snowflake\_database.snowflake\_database](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/database) | resource |
| [snowflake\_database\_role.snowflake\_database\_role](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/database_role) | resource |
| [snowflake\_grant\_account\_role.tech\_user\_grant\_tech\_user\_role](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_account_role) | resource |
| [snowflake\_grant\_account\_role.tech\_user\_role\_grant\_accountadmin](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_account_role) | resource |
| [snowflake\_grant\_database\_role.grant\_db\_role\_to\_accountadmin](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_database_role) | resource |
| [snowflake\_grant\_database\_role.grant\_tech\_user\_role\_to\_db\_role](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_database_role) | resource |
| [snowflake\_grant\_privileges\_to\_account\_role.tech\_user\_warehouse\_access](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_privileges_to_account_role) | resource |
| [snowflake\_grant\_privileges\_to\_database\_role.grant\_tech\_user\_objects\_all](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_privileges_to_database_role) | resource |
| [snowflake\_grant\_privileges\_to\_database\_role.grant\_tech\_user\_objects\_all\_future](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_privileges_to_database_role) | resource |
| [snowflake\_grant\_privileges\_to\_database\_role.tech\_user\_db\_access](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_privileges_to_database_role) | resource |
| [snowflake\_grant\_privileges\_to\_database\_role.tech\_user\_schema\_access](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/grant_privileges_to_database_role) | resource |
| [snowflake\_schema.snowflake\_schemas](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/schema) | resource |
| [snowflake\_user.tech\_user](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/user) | resource |
| [snowflake\_warehouse.snowflake\_warehouse](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/warehouse) | resource |
| [tls\_private\_key.tech\_user\_private\_key](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource |
| [snowflake\_accounts.account](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/data-sources/accounts) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| --------------------------------- | -------------------------------------------------------------------- | -------- | ------- | :------: |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
## Outputs
| Name | Description |
| ----------------------------------------- | ----------- |
| [secrets](#output_secrets) | n/a |
| [ssm\_parameters](#output_ssm_parameters) | n/a |
# live/
What does the live/ folder contain?
This template uses Terragrunt for streamlined deployment and is designed to be deployed either locally or via GitHub CI.
In the `live/` directory, you will find Terragrunt configurations for each environment.
Terragrunt is a powerful tool for managing Terraform configurations across multiple modules and environments from a single source of truth. It helps keep infrastructure code DRY (Don't Repeat Yourself) through modular configuration and dependency management.
The template consists of three core Terraform modules:
* `base/aws`
* `base/snowflake`
* `pipelines/`
This structure is replicated in the `live/` directory, where each module has its own `terragrunt.hcl` file containing:
* Input values
* Terraform provider configuration
* Terraform backend configuration
* Dependencies on other modules
```
live/
├── prod/
│ ├── base/
│ │ ├── aws/
│ │ │ └── terragrunt.hcl
│ │ └── snowflake/
│ │ └── terragrunt.hcl
│ ├── pipelines/
│ │ └── terragrunt.hcl
│ └── root.hcl
```
Note that all `terragrunt.hcl` files reference the `root.hcl` file, where common configurations are defined.
There are three ways to deploy the project:
1. Local deployment with local state
2. Local deployment with remote state
3. GitHub deployment with remote state
Option 1 is ideal for quickly getting started but is not recommended for production deployments.
To deploy using local state, follow the instructions in the [Get Started](/template-aws-snowflake/introduction/get-started) file and run:
```bash
export AWS_PROFILE=
export SNOWFLAKE_PROFILE=
export ENVIRONMENT=
make deploy
```
The deploy command will:
* Run `terragrunt run-all apply` to deploy all Terraform modules in the correct order
* Build and push all Docker images found in the `pipelines/ingest` and `pipelines/transform` folders
#### Why Separate Terraform and Docker Processes?
The infrastructure deployment and container build processes must be separated due to dependencies between resources.
For example, when Terraform creates an AWS Lambda function with an associated ECR repository, the ECR repository must exist before the container image can be built and pushed to it. However, the container image must be built and pushed to ECR before the Lambda function can be fully functional.
This circular dependency is resolved by:
1. First, deploying the infrastructure (ECR, Lambda, etc.) via Terraform/Terragrunt
2. Then, building and pushing container images in a separate process
This two-step approach ensures that all required infrastructure exists before building and deploying containers.
## Local Deployment with Remote State
Do not store the Terraform state on your local machine. Instead, store the state in a remote backend such as AWS S3.
Follow the instructions in the [production-deployment](/template-aws-snowflake/guides/production-deployment) file to create the S3 bucket and run:
```bash
export AWS_PROFILE=
export SNOWFLAKE_PROFILE=
export ENVIRONMENT=
make deploy
```
## GitHub Deployment with Remote State
Follow the instructions in the [production-deployment](/template-aws-snowflake/guides/production-deployment) file. Push your changes to the repository, and the CI will automatically initiate the deployment.
# Add a New Pipeline
This guide explains how to add a new data pipeline to the template.
The pipeline architecture includes:
1. Data ingestion using serverless functions (AWS Lambda) and an ELT tool ([dlt](https://dlthub.com/docs/intro))
2. Staging in cloud object storage ([Amazon S3](https://aws.amazon.com/s3/))
3. Automated data loading into [Snowflake](https://www.snowflake.com/en/) landing tables
4. Data transformation using SQL analytics with [dbt](https://docs.getdbt.com/)
The boringdata CLI automates many steps along the way.
Before you start, make sure you have installed the boringdata CLI:
{% tabs %}
{% tab title="SSH GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install git+ssh://git@github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% tab title="HTTPS GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install https://github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% endtabs %}
You can then use the boringdata CLI from any directory:
uvx boringdata --help
## Step 1: Add a New Data Source
Let's start by adding a new data source for ingestion.
The template uses [dlt](https://dlthub.com/docs/intro) as the ingestion framework. Check the [dlt ecosystem](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) to find the connector you want.
You can then generate a full ingestion pipeline for this connector by running:
```bash
cd pipelines && uvx boringdata dlt add-source
```
This command will create the following files:
`pipelines/_lambda.tf` = serverless function (AWS lambda) infrastructure
`pipelines/ingest/-ingestion/*` = Lambda's dockerized code
Boringdata will also run some helpful operations:
* Set up a Python virtual environment and install the necessary dependencies
* Copy `.env.example` to `.env.local`
* Initialize the [dlt](https://dlthub.com/docs/intro) data connector
* Parse required secrets from configuration files and update both environment variables and infrastructure configurations
Example using the [Notion API](https://developers.notion.com/) as a source:
```
cd pipelines && uvx boringdata dlt add-source notion
```
{% hint style="info" %}
You can assign a different name to your source than the connector name.
To do so, add the CLI option: --source-name \
{% endhint %}
## Step 2: Configure Secrets
If your source requires secrets (for example, an API key), update the .env.example.
After deployment, update these secrets manually in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) if needed.
Example for Notion integration:
The following lines should be present in the .env file:
```bash
SOURCES__NOTION__API_KEY="your_api_key_here"
```
## Step 3: Customize the Ingestion Logic
Edit `pipelines/ingest/-ingestion/lambda_handler.py`
{% code title="pipelines/ingest/\-ingestion/lambda\_handler.py" %}
```python
#Add missing imports
from import
...
#Update the scope of data to be loaded
load_data =
```
{% endcode %}
Example for Notion integration:
```python
from notion import notion_databases
...
#Update the scope of data to be loaded
load_data = notion_databases(database_ids=["your_database_id"])
```
{% hint style="info" %}
Use the \\_pipeline.py generated by [dlt](https://dlthub.com/docs/intro) as an inspiration
{% endhint %}
## Step 4: Test the Ingestion Function Locally
To verify your changes, run the function locally (using [DuckDB](https://duckdb.org/) as a local target):
```bash
cd pipelines/ingest/-ingestion/ && make run-local
```
This step allows you to test the function and inspect the output data format.
## Step 5: Generate the Source Schema
{% hint style="info" %}
[Why do you need to generate a yaml for each source ?](https://docs.boringdata.io/template-aws-snowflake/guides/pages/mvzXlWASNIL69qnoS81s#what-are-less-than-source-greater-than-source_schema.yml-files)
{% endhint %}
Generate a YAML file that defines your source's data structure (used to create data warehouse tables in [Snowflake](https://www.snowflake.com/en/)):
```bash
uvx boringdata dlt get-schema \
--engine snowflake \
--output-folder pipelines/ingest/
```
## Step 6: Create Transformation Models
Based on the YAML file generated in step 5, boringdata can automatically generate corresponding SQL transformation models for each of the tables:
```bash
uvx boringdata dbt import-source \
--source-yml pipelines/ingest/_source_schema.yml \
--output-folder pipelines/transform
```
## Step 7: (Optional) Add Workflow Automation
To coordinate the ingestion and transformation steps, add workflow automation using [AWS Step Functions](https://aws.amazon.com/step-functions/):
```bash
uvx boringdata aws step-function lambda-dbt \
--output-folder pipelines \
--source-name
```
## Step 8: Deploy the Infrastructure
Finally, deploy the project:
```bash
export AWS_PROFILE=your_aws_profile
export SNOWFLAKE_PROFILE=your_snowflake_profile
export ENVIRONMENT=dev
make deploy
```
# CI Deployment
This guide outlines the essential steps for deploying the AWS + Snowflake Data Stack Template in a production environment.
## 1. Create Dedicated Users
A best practice when deploying with Terraform is to create dedicated credentials that Terraform will use during the deployment.
Terraform should only use these users and have the minimal rights required.
For both AWS and Snowflake, we provide ready-to-use policies and scripts to create these users with your admin account quickly.
{% hint style="info" %}
You will notice that these users are environment-specific. Each Terraform can only deploy to one environment. [FAQ](/template-aws-snowflake/help/faq#what-is-an-environment)
{% endhint %}
### For AWS:
```bash
cd init/
export AWS_PROFILE=
make create-tf-user-aws env= aws_region=
```
This script will:
* create a new user called \\_AWS\_SF\_ADMIN
* assign him this [policy](https://github.com/boringdata/boringdata-template-aws-snowflake/blob/main/init/init_aws_tf_user_policy.json)
* create files `.env..secrets` and .env.\.variables
### For Snowflake:
```bash
cd init/
export SNOWFLAKE_PROFILE=
make create-tf-user-snowflake env= aws_region=
```
This script will:
* Add credentials to `.env..secrets` and `.env..variables` . The account name is directly parsed from your SNOWFLAKE\_PROFILE.
* Create a new SQL script `init_snowflake_tf_user_.sql`. This script creates the user `_AWS_SF_ADMIN`
* The script will be executed if SnowSQL is installed and set up (with a SnowSQL connection having the same name as your profile). If not, you can run it directly in your Snowflake console.
## 2. CI/CD Pipeline Setup
The default version of the template does not contain a CICD.
To add it, run:
```bash
# Remove the boringdata's internal test workflow
rm .github/workflows/boringdata-test.yml
# Initialize GitHub workflows for CI/CD
# This will create a .github/workflows/ci.yml file with AWS deployment configuration
uvx boringdata github init --template-type aws-snowflake
# Initialize Terragrunt configuration to use S3 remote state
# This will create/update the root.hcl file in the live/ directory
uvx boringdata terragrunt init --output-folder live
```
This will:
* add a ready-to-use GitHub Actions workflow in the `.github/workflows` folder
* Update the Terragrunt configuration to point to the S3 bucket
The GitHub Actions workflow requires AWS and Snowflake credentials to deploy the project.
You must, therefore, create the necessary variables and secrets in your GitHub repository.
If you have the GitHub CLI installed and are authorized for your repository, run the following commands from the project root:
```bash
cd init/
make github-ci-setup repo=/ env=
```
This command will automatically create the required variables in GitHub based on your AWS and Snowflake profiles based on the .env files created previously.
Alternatively, you can manually set them up in the GitHub console.
That's it; you are now ready to deploy.
## 3. Set Up Terraform State S3 Bucket
You must use a dedicated S3 bucket to store the Terraform state for production deployment.
To create the bucket, run the following command:
```bash
cd init/
export AWS_PROFILE=
make create-tf-bucket env= aws_region=
```
This command will:
* Create a new S3 bucket named `--terraform-state-bucket`.
* Configure the appropriate bucket policies and enable encryption and versioning.
{% hint style="warning" %}
If you have deployed the template using the Quick Start guide (with a local state).
You can either:
* Destroy and start fresh
terragrunt run-all destroy
* Migrate the state:
export AWS\_REGION=\
export ENVIRONMENT=\
terragrunt run-all init -migrate-state -input=true
{% endhint %}
## 4. Deployment
The CI pipeline runs on every merge to the main branch and deploys to the environment defined in the variables.
The CI pipeline will start automatically once you push your changes to the repository.
GitHub CI
The CI pipeline consists of two jobs:
* **Terragrunt-apply**: Deploys the infrastructure using Terragrunt
* **Deploy-dockers**: Builds and deploys Docker containers
The deploy-dockers job executes `make deploy` in all `pipelines/ingest` and `pipelines/transform` folders that contain a Dockerfile.
Only the folders with changes will be processed if the CI pipeline runs after a merge.
All Docker images will be built and deployed when the workflow is run from scratch.
CI Docker Build Process
### Verify the Deployment
After deployment is complete, verify the setup in your AWS console:
1. Navigate to the AWS Step Functions service
2. Locate your pipeline's step function (e.g., `prod-chess-step-function`)
3. Execute the step function with an empty payload
4. Monitor the execution to ensure the pipeline runs successfully
# FAQ
How do I integrate the template into my existing Terraform stack?
Our templates are organized into two types of modules:
• Base modules (base/aws) – Infrastructure components.
• Pipeline modules (pipeline/) – Pipeline-specific components.
Typically, a data team manages the `pipelines/` module.
The company's infra team usually manages the resources defined in the base/aws module.
Having this split already done in this template makes it easy to use the `base/aws` as "spec" for your infra team.
There are too many files—I don’t know where to start!
For codebase discovery, LLMs are our best allies.
Get Cursor or Copilot and start asking questions in the chat interface.
The documentation is included in the repo as Markdown files, and LLMs usually find the necessary information independently.
What is an "environment" ?
Throughout this documentation, you will see references to the **ENVIRONMENT**.
In our template, the environment represents a specific version or instance of your project, such as `prod`, `dev`, or `ctlq`.
This value is used as a prefix for all resources created in both AWS and Snowflake, ensuring that each deployment is isolated and clearly identified.
#### How Environments are Used
* **Resource Naming:**\
Every resource (e.g., S3 buckets, Lambda functions, Snowflake databases) is prefixed with the environment name. This makes it easy to distinguish between resources belonging to different environments.
* **Deployment Isolation:**\
With Terragrunt, you can deploy the project to multiple environments concurrently. Each environment can have its own set of custom input values and configuration settings. For example, you can deploy the same project in different AWS regions or accounts.
* **Configuration Customization:**\
Different environments allow you to adjust resource configurations according to your needs. You might choose a larger warehouse size or different Lambda settings in production compared to development.
#### Choosing a Name for Your Environment
When selecting a name for your environment, follow these guidelines:
* **Keep it Short and Lowercase:**\
Use concise, lowercase names such as `dev`, `prod`, or `qa`.
* **Avoid Special Characters or Spaces:**\
Stick to alphanumeric characters and simple words to ensure compatibility across all resource naming conventions.
Using clear and consistent environment names helps maintain organization, prevents resource conflicts, and simplifies management across your AWS and Snowflake deployments.
What are <source>source_schema.yml files ?
We maintain a **YAML file** for every data source that defines the structure of the landing tables in Snowflake.
This YAML schema ensures each source's data model is clearly documented and versioned.
The format of this YAML is inspired by the [data contract cli project](https://github.com/datacontract/datacontract-cli).
***
#### Why We Use a YAML Schema File
1. **Documentation of the Data Model** Having a human-readable YAML file allows any team member to quickly see the tables, columns, and data types for a particular source, making it easier to understand how the data flows through the pipeline.
2. **Automated Table Creation** Our Terraform Snowpipe configuration can automatically read the YAML file to create landing tables for each source. You don't have to manually create or update your Snowflake tables whenever you change your schema.
3. **Data Contract Enforcement** The YAML schema acts as a contract between the ingestion layer and Snowflake. If the data in your pipeline doesn't match the declared schema, it can trigger validation rules or highlight mismatches, preventing corrupt or malformed data from being loaded.
4. **Version Control** By checking the YAML schema into Git, you can track when and why schema changes occur (adding, removing, or modifying columns). This history helps you audit and review changes before they reach production.
5. **Data Quality Visibility** Discrepancies between the defined schema and the real data can indicate potential data issues. Because schema mismatches are surfaced early, issues can be caught quickly—before they cause bigger downstream problems.
***
#### Example YAML Schema
Below is an abbreviated example of a YAML schema file for a **Chess** data source.
It illustrates how multiple tables (or "models") are defined within a single file, listing each column's data type and whether it's required, unique, or part of a primary key.
```yaml
dataContractSpecification: 1.1.0
id: chess
info:
title: chess
version: 1.1.0
models:
players_profiles:
description: ''
fields:
username:
type: VARCHAR
required: false
primaryKey: false
unique: false
description: ''
last_online:
type: TIMESTAMP_TZ
required: false
primaryKey: false
unique: false
description: ''
joined:
type: TIMESTAMP_TZ
required: false
primaryKey: false
unique: false
description: ''
file_format: PARQUET
players_games:
description: ''
fields:
end_time:
type: TIMESTAMP_TZ
required: false
primaryKey: false
unique: false
description: ''
white__username:
type: VARCHAR
required: false
primaryKey: false
unique: false
description: ''
black__username:
type: VARCHAR
required: false
primaryKey: false
unique: false
description: ''
file_format: PARQUET
_dlt_loads:
description: Created by DLT. Tracks completed loads
fields:
load_id:
type: VARCHAR
required: true
primaryKey: false
unique: false
description: ''
status:
type: NUMBER(19,0)
required: true
primaryKey: false
unique: false
description: ''
file_format: JSON
```
# Overview
Welcome to the AWS + Apache Iceberg Data Stack Template!
This template provides everything you need to build a modern data lakehouse on AWS using Apache Iceberg.
The infrastructure is fully automated with Terraform and includes:
* An example ingestion pipeline: dlt + lambda
* A dbt project transformation Iceberg data with Athena
* AWS Step Functions to orchestrate ingestion + transformation
* A Github CI workflow
The template is designed to be fully modular and customizable.\
You can easily swap out any component with your preferred tools:
* Replace dlt with any other ingestion framework
* Use an alternative to dbt for transformations
* Switch to a different orchestration service instead of Step Functions
***
## Next Steps
1. [Key Concepts](/template-aws-iceberg/introduction/key-concepts) - Understand the core architecture and components
2. [Get Started](/template-aws-iceberg/introduction/get-started) - Set up your environment and run your first deployment
# Key Concepts
Understand template's structure
This section explains the core concepts and architecture of this template.
## Code Structure
The template's code is organized into three main components:
```
📁
├── 📁 pipelines/ # Data pipelines:
│ ├── 📁 ingest/ # Data ingestion layer
│ ├── 📁 transform/ # Data transformation layer
│ └── 📁 orchestrate/ # Workflow orchestration layer
│
├── 📁 base/ # Cloud infrastructure (VPC, roles, users, compute cluster, etc.)
│
└── 📁 live/ # Environment-specific deployment configuration
```
Each component is documented separately here:
{% content-ref url="/pages/G2khJSaacMix8oYa7iVH" %}
[pipelines/](/template-aws-iceberg/project-structure/pipelines)
{% endcontent-ref %}
{% content-ref url="/pages/qo9vW9N1VXDI9z43Fdr5" %}
[base/aws/](/template-aws-iceberg/project-structure/aws)
{% endcontent-ref %}
{% content-ref url="/pages/txxF4gycrydceZI6n15j" %}
[live/](/template-aws-iceberg/project-structure/live)
{% endcontent-ref %}
## Data Flow
1. Source data is ingested into [Apache Iceberg](https://iceberg.apache.org/) landing tables: code in `pipelines/ingest/-*/`
2. Data transformations are applied to create staging tables using SQL engine ([Amazon Athena](https://aws.amazon.com/athena/)): code in `pipelines/transform/`
## Data Pipeline Architecture
Our data platform follows a layered architecture:
### 1. Data Ingestion Layer
For each source, the ingestion layer is structured as follows:
```
📁 pipelines/
├── 📁 ingest/
│ ├── 📁 -ingestion/ # Core ingestion logic
│ │
│ └── 📁 -schema/ # Iceberg Table schema definitions
│ └── .py
│ └── ...
│
└── _*.tf # Infrastructure definition (serverless functions, containers, etc.)
```
Each source has:
* A folder `pipelines/ingest/-ingestion/` containing the core ingestion logic packaged in a container
* Infrastructure as Code files in `pipelines/*tf` for deploying this ingestion container (as serverless functions (AWS lambda) or container tasks ([Amazon ECS](https://aws.amazon.com/ecs/)))
* A folder for the management of the landing tables (`-schema/`)
{% hint style="info" %}
More info about landing table schema evolution in [FAQ](/template-aws-iceberg/help/faq#iceberg-landing-table-schema-evolution)
{% endhint %}
The template comes with an example data ingestion pipeline deployed as a serverless function (lambda) using [dlt](https://dlthub.com/docs/intro); more details here:
{% content-ref url="/pages/QmSXv61cHRK1XAaEgwKO" %}
[Ingestion: dlt + lambda](/template-aws-iceberg/project-structure/pipelines/chess-ingestion)
{% endcontent-ref %}
### 2. Data Transformation Layer
The transformation layer is a [dbt](https://docs.getdbt.com/) project that transforms the data into Iceberg staging tables using the SQL query engine [Amazon Athena](https://aws.amazon.com/athena/).
This project is located in the `pipelines/transform` folder:
```
📁 pipelines/
├── 📁 transform/ # SQL transformation project
│ ├── 📁 models/
│ │ ├── 📁 staging/ # Raw table connections
│ │ └── 📁 marts/ # Transformations
│ │
│ ├── dbt_project.yml
│ └── Dockerfile # For container deployment
│
└── ecs_task_dbt.tf # Infrastructure definition for running dbt container
```
This transformation project runs on container infrastructure ([Amazon ECS](https://aws.amazon.com/ecs/) Fargate).
More details on how this transformation project is structured here:
{% content-ref url="/pages/vC7jupqVP6pcjq6OCJ4f" %}
[Transformation: dbt](/template-aws-iceberg/project-structure/pipelines/transform)
{% endcontent-ref %}
### 3. Workflow Orchestration Layer
The orchestration layer coordinates the execution of the ingestion and transformation layers using workflow automation.
This template proposes an example orchestration using [AWS Step Functions](https://aws.amazon.com/step-functions/):
```
📁 pipelines/
├── 📁 orchestrate/
│ └── _step_function.json # Workflow definition
│
└── _step_function.tf # Creates an orchestration workflow in AWS Step Functions
```
## Deployment
This template is ready to be deployed.
The stack deployment is structured in 3 steps:
* First, the infrastructure modules (base/ and pipelines/) are deployed using [Terragrunt](https://terragrunt.gruntwork.io/) for infrastructure management
* Then, the containers for the ingestion and transformation layers are built and pushed to the container registry
* Finally, the schema evolution scripts of the Iceberg landing tables are run
If you want to get started quickly and deploy the template from your machine, follow this guide:
{% content-ref url="/pages/Sp2fdZp3rxZePnTg8vMP" %}
[Get Started](/template-aws-iceberg/introduction/get-started)
{% endcontent-ref %}
To get started deploying from [GitHub Actions](https://github.com/features/actions) CI/CD, head there:
{% content-ref url="/pages/MMbEuLtK7JyjMXkDAboW" %}
[CI Deployment](/template-aws-iceberg/guides/production-deployment)
{% endcontent-ref %}
## Makefile
The template is composed of many Makefiles providing utilities.
Here are some examples:
* `make deploy` in the root folder will deploy the template from your machine
* `make build` in a folder with a Dockerfile will build the container
* `make local-run` in a serverless function folder will test the function locally
* etc
Everywhere you see a Makefile, run `make` and the list of possible actions will be listed
{% content-ref url="/pages/Sp2fdZp3rxZePnTg8vMP" %}
[Get Started](/template-aws-iceberg/introduction/get-started)
{% endcontent-ref %}
# Get Started
## Prerequisites
Before you begin, ensure you have the following tools installed on your local machine:
* **Git**
* **Python 3.12**
* **AWS CLI**
* **Terraform (v1.0+)**
* **Terragrunt**: Wrapper for managing Terraform configurations
* **Docker**
* **Make**
* **uv**: Python package management tool
## AWS Credentials
Set up your AWS credentials in the `~/.aws/credentials` file:
```bash
[YOUR_PROFILE]
region=your-region
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
```
{% hint style="warning" %}
Don't forget to include the region in your profile
{% endhint %}
{% hint style="info" %}
Your AWS user should have permission listed in this [example policy](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/init/init_aws_tf_user_policy.json) file (less privilege).
{% endhint %}
## Quick Start
For a quick start with local deployment and local Terraform state:
```bash
# Set your AWS profile and environment name
export AWS_PROFILE=
export ENVIRONMENT=
# Deploy the infrastructure and Docker images
make deploy
```
This command will:
1. Deploy all Terraform modules in the correct order
2. Build and push Docker images for the ingestion and transformation pipelines
## Verify Your Deployment
After deployment completes:
1. Navigate to the AWS Step Functions service
2. Find your pipeline's step function (e.g., `dev-chess-step-function`)
3. Execute the step function with an empty payload
4. Monitor the execution to verify the pipeline runs successfully
## Next Steps
After your initial deployment, you might want to:
1. [Add a New Pipeline](/template-aws-iceberg/guides/add-a-pipeline) - Create your own data pipeline
2. [CI Deployment](/template-aws-iceberg/guides/production-deployment) - Set up production deployment with CI/CD
3. [FAQ](/template-aws-iceberg/help/faq) - Find answers to frequently asked questions
# pipelines/
Contents of the pipelines/ folder
This template implements a modern data architecture using AWS services and Apache Iceberg, featuring a clean separation between ingestion and transformation layers.
## Architecture Overview
Data pipelines are built using a 2-layer architecture:
* **Ingestion Layer** (`ingest/`): Extracts data from external sources and loads it into Apache Iceberg tables
* **Transformation Layer** (`transform/`): Transforms raw data using dbt and AWS Athena into analytics-ready tables
### Project Structure
```
pipelines/
├── ingest/ # Ingestion layer code
│ ├── {source}-ingestion/ # Source-specific ingestion code (Lambda)
│ └── {source}-schema/ # Schema definitions for landing tables
├── transform/ # Transformation layer code (dbt project)
│ ├── models/ # dbt models
│ ├── sources/ # dbt source definitions
│ └── ...
└── *.tf # Terraform infrastructure definitions
```
### Data Flow Diagram
The following diagram illustrates how data flows through the system:
```mermaid
%%{init: {'theme':'dark', 'themeVariables': {'primaryColor': '#2a9d8f', 'primaryTextColor': '#fff', 'primaryBorderColor': '#219287', 'lineColor': '#f4a261', 'secondaryColor': '#e76f51', 'tertiaryColor': '#264653'}}}%%
graph LR
subgraph "Ingest Layer"
api[External Data APIs & Sources] -->|Extract Data| lambda[AWS Lambda with DLT]
lambda -->|Write Files| s3raw[(S3 Raw Parquet Files)]
s3raw -->|Add to Tables| icelandZone[(Iceberg Landing Zone)]
end
subgraph "Transform Layer"
icelandZone -->|Source Tables| dbt[dbt Models via ECS Task]
dbt -->|Transform Data| athena[AWS Athena Query Engine]
athena -->|Write Results| icebergAnalytics[(Iceberg Analytics Zone)]
end
%% Connectors
lambda -.->|Scheduled or Event-triggered| stepFunction[AWS Step Function]
stepFunction -.->|Can Trigger| dbt
%% Styling
classDef default fill:#2a3d45,color:#fff,stroke:#333,stroke-width:1px;
classDef source fill:#264653,color:#fff,stroke:#333,stroke-width:1px;
classDef compute fill:#2a9d8f,color:#fff,stroke:#219287,stroke-width:1px;
classDef storage fill:#e9c46a,color:#333,stroke:#e9b949,stroke-width:1px;
classDef orchestration fill:#f4a261,color:#333,stroke:#f39c52,stroke-width:1px;
class api source;
class lambda,dbt,athena compute;
class s3raw,icelandZone,icebergAnalytics storage;
class stepFunction orchestration;
```
## Ingestion Layer
The ingestion layer extracts data from external sources and loads it into Apache Iceberg landing tables. It consists of three main components:
1. **Source-specific ingestion code** in `pipelines/ingest/{SOURCE_NAME}-ingestion/`
2. **Schema definitions** in `pipelines/ingest/{SOURCE_NAME}-schema/`
3. **Infrastructure as code** in Terraform files (`pipelines/*.tf`)
### Example: Chess.com Pipeline
This repository includes an example pipeline that ingests data from Chess.com:
```
pipelines/
├── ingest/
│ ├── chess-ingestion/
│ │ ├── lambda_handler.py # Lambda code using DLT for Chess.com ingestion
│ │ ├── Dockerfile # Container image definition
│ │ └── ...
│ ├── chess-schema/
│ │ ├── chess_players_games.py # Schema definition for players_games table
│ │ ├── chess_players.py # Schema definition for players table
│ │ └── ...
├── chess_lambda.tf # Terraform creating the Lambda function
├── ingestion_bucket.tf # S3 bucket for landing zone
├── staging_bucket.tf # S3 bucket for staging/analytics zone
└── ...
```
### Ingestion Process
The data ingestion process follows these steps:
1. **Extraction & Load**: A Lambda function uses [Data Load Tool (DLT)](https://dlthub.com/) to extract data from external sources and store it as Parquet files in S3
2. **Table Management**: The same Lambda then uses PyIceberg to add these files to Iceberg tables
{% hint style="info" %}
For detailed instructions on running and testing the Chess.com Lambda function, see the [chess-ingestion README](/template-aws-iceberg/project-structure/pipelines/chess-ingestion).
{% endhint %}
### Landing Table Management
The landing tables are defined and managed through schema scripts in `pipelines/ingest/{SOURCE_NAME}-schema/`. These scripts are automatically executed during deployment to:
* Create new tables if they don't exist
* Update existing table schemas when needed
* Maintain table properties and metadata
When schema changes are required, you modify and redeploy these definition files.
{% hint style="info" %}
More details about schema evolution here: [FAQ](/template-aws-iceberg/help/faq#iceberg-landing-table-schema-evolution)
{% endhint %}
## Transformation Layer
The transformation layer processes data from landing tables into analytics-ready formats using dbt. It consists of:
1. **dbt project** in `transform/`
2. **Infrastructure code** in `pipelines/*.tf` (especially `ecs_task_dbt.tf`)
{% hint style="info" %}
For details on developing and running dbt models, see the [transform README](/template-aws-iceberg/project-structure/pipelines/transform).
{% endhint %}
## Infrastructure Overview
The following diagram shows the AWS infrastructure components and their relationships:
```mermaid
%%{init: {'theme':'dark', 'themeVariables': {'primaryColor': '#06d6a0', 'primaryTextColor': '#fff', 'primaryBorderColor': '#05c491', 'lineColor': '#ef476f', 'secondaryColor': '#118ab2', 'tertiaryColor': '#073b4c'}}}%%
graph TB
%% INFRASTRUCTURE COMPONENTS
subgraph "AWS Infrastructure"
%% Data Storage
subgraph "Data Storage"
ingestion_bucket[(Ingestion S3 Bucket)]
staging_bucket[(Staging S3 Bucket)]
glue_catalog[AWS Glue Catalog]
end
%% Compute Resources
subgraph "Compute Resources"
chess_lambda[Chess Lambda Function]
athena[AWS Athena]
ecs_task[dbt ECS Task]
end
%% Container Registry
subgraph "Container Registry"
chess_ecr[Chess ECR Repository]
dbt_ecr[dbt ECR Repository]
end
%% Supporting Services
subgraph "Supporting Services"
chess_secrets[Chess Secrets Manager]
chess_step_function[Chess Step Function]
end
end
%% CONFIGURATION
subgraph "Configuration"
env[Environment Variable]
vpc_name[VPC Name Variable]
ecs_name[ECS Cluster Variable]
end
%% DATA SOURCES
subgraph "AWS Data Sources"
vpc_data[VPC Data Source]
subnets[Subnets Data Source]
ecs_data[ECS Cluster Data Source]
region[AWS Region]
caller[AWS Caller Identity]
end
%% RELATIONSHIPS
%% Configuration to Data Sources
env --> |References| all
vpc_name --> vpc_data
ecs_name --> ecs_data
%% Data Sources to Resources
vpc_data --> subnets
subnets --> ecs_task
ecs_data --> ecs_task
region --> all
caller --> all
%% Registry to Compute
chess_ecr --> |Image Source| chess_lambda
dbt_ecr --> |Image Source| ecs_task
%% Orchestration Flow
chess_secrets --> |Credentials| chess_lambda
chess_lambda --> |Writes Data| ingestion_bucket
ingestion_bucket --> |Glue Table Ref| glue_catalog
staging_bucket --> |Glue Table Ref| glue_catalog
glue_catalog --> |Table Metadata| athena
athena --> |Query Execution| ecs_task
chess_step_function --> |Triggers| chess_lambda
chess_step_function --> |Can Trigger| ecs_task
ecs_task --> |Writes Data| staging_bucket
%% Styling
classDef variables fill:#06d6a0,color:#fff,stroke:#05c491,stroke-width:1px;
classDef datasources fill:#118ab2,color:#fff,stroke:#1179a1,stroke-width:1px;
classDef compute fill:#ef476f,color:#fff,stroke:#de3660,stroke-width:1px;
classDef storage fill:#ffd166,color:#073b4c,stroke:#ffcc57,stroke-width:1px;
classDef services fill:#073b4c,color:#fff,stroke:#062a37,stroke-width:1px;
classDef registry fill:#f78c6b,color:#073b4c,stroke:#f67b5c,stroke-width:1px;
classDef grouping fill:none,stroke:#aaa,stroke-width:1px,color:#fff;
class env,vpc_name,ecs_name variables;
class vpc_data,subnets,ecs_data,region,caller datasources;
class chess_lambda,athena,ecs_task compute;
class ingestion_bucket,staging_bucket,glue_catalog storage;
class chess_secrets,chess_step_function services;
class chess_ecr,dbt_ecr registry;
class "AWS Infrastructure","Configuration","AWS Data Sources","Data Storage","Compute Resources","Container Registry","Supporting Services" grouping;
```
### Module documentation
## Requirements
| Name | Version |
| ----------------------------------- | -------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [aws](#requirement_aws) | >=5.63.1 |
## Providers
| Name | Version |
| ---------------------- | ------- |
| [aws](#provider_aws) | 5.92.0 |
| [null](#provider_null) | 3.2.3 |
## Modules
| Name | Source | Version |
| --------------------------------------------------------------------------- | ------------------------------------------------- | ------- |
| [bucket\_ingestion\_read\_policies](#module_bucket_ingestion_read_policies) | terraform-aws-modules/iam/aws//modules/iam-policy | 5.39.1 |
| [bucket\_staging\_read\_policies](#module_bucket_staging_read_policies) | terraform-aws-modules/iam/aws//modules/iam-policy | 5.39.1 |
| [chess\_ecr](#module_chess_ecr) | terraform-aws-modules/ecr/aws | n/a |
| [chess\_lambda\_function](#module_chess_lambda_function) | terraform-aws-modules/lambda/aws | 7.2.1 |
| [chess\_pipeline](#module_chess_pipeline) | terraform-aws-modules/step-functions/aws | 4.2.1 |
| [chess\_secrets](#module_chess_secrets) | terraform-aws-modules/secrets-manager/aws | 1.1.2 |
| [dbt\_ecr](#module_dbt_ecr) | terraform-aws-modules/ecr/aws | n/a |
| [dbt\_task\_definition](#module_dbt_task_definition) | terraform-aws-modules/ssm-parameter/aws | 1.1.1 |
| [ecs\_task\_definition\_dbt](#module_ecs_task_definition_dbt) | terraform-aws-modules/ecs/aws///modules/service | 5.11.2 |
| [ingestion\_bucket](#module_ingestion_bucket) | terraform-aws-modules/s3-bucket/aws | 4.1.0 |
| [staging\_bucket](#module_staging_bucket) | terraform-aws-modules/s3-bucket/aws | 4.1.0 |
## Resources
| Name | Type |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [null\_resource.chess\_empty\_image](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
| [aws\_caller\_identity.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws\_ecs\_cluster.ecs-cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecs_cluster) | data source |
| [aws\_iam\_policy\_document.bucket\_ingestion\_read\_write\_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws\_iam\_policy\_document.bucket\_staging\_read\_write\_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws\_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
| [aws\_subnets.subnets](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
| [aws\_vpc.vpc](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| --------------------------------------------- | -------------------------------------------------------------------- | -------- | ------- | :------: |
| [ecs\_cluster\_name](#input_ecs_cluster_name) | The name of the ECS cluster | `string` | `null` | no |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
| [vpc\_name](#input_vpc_name) | The name of the VPC to deploy the ECS cluster in | `string` | `null` | no |
## Outputs
No outputs.
# Ingestion: dlt + lambda
## Overview
This example demonstrates a serverless data ingestion pipeline that:
1. Fetches chess data from an external source and processes it using [dlt](https://dlthub.com/)
2. Writes the data to Apache Iceberg tables on S3
The pipeline runs as an AWS Lambda function packaged in a Docker container.
## How It Works
### Infrastructure Components
* **AWS Lambda**: Executes the ingestion code on demand
* **Amazon ECR**: Stores the Docker container image
* **Amazon Glue**: Hosts the Iceberg Catalog
* **Amazon S3**: Hosts the Iceberg tables
* **AWS Secrets Manager**: Stores credentials and configuration
* **Terraform**: Provisions and manages all infrastructure
### Code Structure
```
pipelines/
├── chess_lambda.tf # Terraform creating the lambda function and ECR repository
├── ingestion_bucket.tf # Terraform creating S3 bucket
└── ingest/
└── chess-ingestion/ # Lambda function code
├── Dockerfile
├── lambda_handler.py # Lambda code with DLT pipeline
└── ...
└── chess-schema/ # Iceberg schema definition in Glue Catalog
├── table_schema.py
└── ...
```
### Data Flow Process
The pipeline follows these steps:
1. **Extraction and Loading to S3**: DLT loads data as Parquet files to an ingestion S3 bucket following this path pattern:
```
{source_name}/raw/{table_name}/{load_id}.{file_id}.{ext}
```
2. **Iceberg Integration**: PyIceberg adds these files to Iceberg tables.\
Iceberg tables are located in the same ingestion S3 bucket under:
```
{source_name}/landing/{table_name}/
```
Files are inserted in append mode.
## Development Guide
### 1. Local Development with DuckDB
For rapid iteration without AWS resources, use DuckDB as the destination:
1. Create a `.env.local` file with:
```
DESTINATION=duckdb
# Add any source-specific credentials here
```
2. Run the pipeline locally:
```bash
make run-local
```
3. Examine results in the local .duckdb database
### 2. Local Development with S3
To run the lambda with Iceberg destination:
1. Configure `.env.local` with:
```
DESTINATION=filesystem
AWS_REGION=
S3_BUCKET_NAME=
AWS_PROFILE=
# Add any source-specific credentials here
```
2. Run with the same command:
```bash
make run-local
```
### 3. Testing on AWS
Once your code is deployed to AWS you can run the lambda with:
```bash
export AWS_PROFILE=
make run-lambda env=
```
### 4. VSCode Debugging
For interactive debugging, add this to `.vscode/launch.json`:
```json
{
"name": "Debug chess lambda",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/pipelines/ingest/chess-ingestion/lambda_handler.py",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}/pipelines/ingest/chess-ingestion",
"justMyCode": false
}
```
## Schema Management
dlt needs to run a first time to provide the target schema definition.
After running the pipeline locally (see above), generate a source schema definition:
```bash
cd pipelines/
uvx boringdata dlt get-schema chess
```
This will generate one schema file per table in the `ingest/chess-schema` folder to create the iceberg table in the AWS Glue Catalog.
These scripts will be run automatically by the CI/CD pipeline.
More details about schema management can be found [here](https://dlthub.com/docs/guides/schema-management).
## Manual Deployment
For manual deployment:
```bash
# Set required environment variables
export AWS_PROFILE=
# Build and deploy
make deploy env=
```
This process:
1. Builds the Docker image locally
2. Pushes it to ECR
3. Updates the Lambda to use the new image
## Common Commands
```bash
# Development
make run-local # Run locally with settings from .env.local
make run-lambda env= # Execute on AWS Lambda
# Deployment
make build env= # Build Docker image
make deploy env= # Build and deploy to ECR
# Utilities
make help # Show all available commands
```
## Resources
* [DLT Documentation](https://dlthub.com/docs/)
* [PyIceberg Documentation](https://py.iceberg.apache.org/)
# Transformation: dbt
## Overview
This directory contains a data transformation pipeline that:
1. Takes data from Iceberg tables in the landing zone
2. Transforms it using [dbt](https://www.getdbt.com/) (data build tool)
3. Creates analytics-ready tables in staging and mart schemas
The pipeline runs as an AWS ECS Fargate task using a Docker container.
## How It Works
### Infrastructure Components
* **AWS Athena**: SQL query engine for data transformation
* **Amazon S3**: Hosts the Iceberg tables for both source and transformed data
* **AWS Glue**: Provides the catalog for Iceberg tables
* **Amazon ECS**: Orchestrates the dbt container execution
* **Amazon ECR**: Stores the dbt docker container image
* **Terraform**: Provisions and manages all infrastructure
### Project Structure
```
pipelines/
├── transform/ # dbt project root
│ ├── Dockerfile
│ ├── dbt_project.yml # dbt project configuration
│ ├── sources/
│ │ ├──.yml # List all landing tables for a source
│ ├── models/
│ │ ├── staging/ # Staging models (first transformation layer)
│ │ └── mart/ # Final business-ready models
│ └── ...
└── ecs_task_dbt.tf # Terraform creating the ECS task
```
### Data Transformation Flow
The pipeline follows these transformation layers:
1. **Sources**: Raw data from landing tables created by ingestion pipelines
2. **Staging**: Initial cleaning, type conversion, deduplication and renaming
3. **Mart**: Final models organized by business domain, ready for analytics and reporting
## Sources
Sources are defined in the `sources/` folder and reference the landing tables created by the ingestion pipelines:
{% code title="sources/\.yml" %}
```yaml
sources:
- name:
schema:
tables:
- name: __dlt_version
- name: __dlt_loads
...
```
{% endcode %}
You can generate this file automatically using the BoringData CLI:
```bash
cd pipelines/transform
uvx boringdata dbt import-source --source ../ingest/-schema/
```
## Models Structure
The dbt models follow a layered architecture pattern:
* Each folder in the `models` directory corresponds to a distinct schema in Athena
* `models/staging/` ➡️ `_staging` schema in Athena
* `models/mart/` ➡️ `_mart` schema in Athena
## Development Guide
### Option 1: Execute dbt Locally
For rapid development with local dbt execution:
1. **Setup your environment**:
```bash
uv venv --python=python3.12
uv pip install -r requirements.txt
uv run dbt deps
```
2. **Configure dbt profile**:\
Create or update `~/.dbt/profiles.yml` with:
```yaml
local:
target:
outputs:
:
type: athena
database: awsdatacatalog
region_name: "{{ env_var('AWS_REGION') }}"
schema: "_staging"
s3_staging_dir: "s3://--staging-bucket/athena"
s3_data_dir: "s3://--staging-bucket/data"
s3_tmp_table_dir: "s3://--staging-bucket/tmp"
```
3. **Run dbt commands**:
```bash
export DBT_PROFILE=local
export AWS_PROFILE=
export AWS_REGION=
# Run a specific model
uv run dbt run --select model_name
# Run with Makefile shortcut
make run-local cmd="run --select model_name"
```
### Option 2: Execute in AWS ECS Fargate
Once your template is deployed to AWS you can run dbt in the cloud environment:
```bash
export AWS_PROFILE=
export ENVIRONMENT=
make run cmd="run"
```
This will trigger an ECS Fargate task to execute the specified dbt command and store results in Iceberg.
## Deployment
For manual deployment:
```bash
# Set required environment variables
export AWS_PROFILE=
export ENVIRONMENT=
cd pipelines/transform
# Build and deploy
make deploy
```
This process:
1. Builds the Docker image locally
2. Pushes it to ECR
The next time you trigger an ECS task, it will use the latest image.
## Common Commands
```bash
# Development
make run-local cmd="run" # Run dbt locally with specified command
make run-local cmd="test" # Run dbt tests locally
make run-local cmd="docs generate" # Generate dbt documentation
# Cloud Execution
make run cmd="run" # Run dbt in ECS Fargate
make run cmd="test" # Run tests in ECS Fargate
# Deployment
make build # Build Docker image
make deploy # Build and deploy to ECR
```
## Resources
* [dbt Documentation](https://docs.getdbt.com/)
* [AWS Athena User Guide](https://aws.amazon.com/athena/)
* [Apache Iceberg Documentation](https://iceberg.apache.org/)
* [BoringData CLI Guide](https://docs.boringdata.io/)
# base/aws/
## Overview
This Terraform module provisions the core AWS infrastructure needed for a data platform, including:
* VPC with subnets
* ECS cluster for containerized workloads
* Secrets Manager for sensitive values
* SSM Parameters for configuration
## Quick Start
```hcl
module "aws" {
source = "git::https://github.com/boringdata/boringdata-template-aws-iceberg.git//base/aws"
environment = "dev"
secrets = {
"api_key" = "your-secret-value"
}
}
```
## Key Features
* **Environment-based naming**: All resources are prefixed with your environment name
* **Secure networking**: Properly configured VPC with public and private subnets
* **Containerization**: Ready-to-use ECS cluster for your workloads
* **Configuration management**: Built-in secrets and parameters management
## Module Structure
```
aws/
├── data.tf # AWS region and availability zones
├── ecs_cluster.tf # ECS cluster configuration
├── vpc.tf # VPC and networking resources
├── secrets.tf # AWS Secrets Manager resources
├── ssm_parameters.tf # SSM Parameter Store resources
├── variables.tf # Input variables
├── outputs.tf # Output values
├── locals.tf # Local variables
└── versions.tf # Version constraints
```
## Architecture
```mermaid
%%{init: {'theme':'neutral'}}%%
graph TD
env([Environment])
vpc[VPC]
ecs[ECS Cluster]
secrets[Secrets Manager]
ssm[SSM Parameters]
env --> vpc & ecs & secrets & ssm
vpc --> subnets[Public & Private Subnets]
```
## Requirements
| Name | Version |
| ----------------------------------- | -------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [aws](#requirement_aws) | >=5.63.1 |
## Providers
| Name | Version |
| -------------------- | ------- |
| [aws](#provider_aws) | 5.91.0 |
## Modules
| Name | Source | Version |
| ----------------------------------- | ---------------------------------------------- | ------- |
| [ecs\_cluster](#module_ecs_cluster) | terraform-aws-modules/ecs/aws//modules/cluster | 5.11.2 |
| [parameters](#module_parameters) | terraform-aws-modules/ssm-parameter/aws | 1.1.1 |
| [secrets](#module_secrets) | terraform-aws-modules/secrets-manager/aws | 1.1.2 |
| [vpc](#module_vpc) | terraform-aws-modules/vpc/aws | \~> 5.0 |
## Resources
| Name | Type |
| --------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [aws\_availability\_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws\_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| ---------------------------------------- | -------------------------------------------------------------------- | ------------- | ------- | :------: |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
| [secrets](#input_secrets) | A map of secrets to create | `map(string)` | `{}` | no |
| [ssm\_parameters](#input_ssm_parameters) | A map of SSM parameters to create | `map(string)` | `{}` | no |
## Outputs
No outputs.
## Requirements
| Name | Version |
| ----------------------------------- | -------- |
| [terraform](#requirement_terraform) | >=1.5.7 |
| [aws](#requirement_aws) | >=5.63.1 |
## Providers
| Name | Version |
| -------------------- | ------- |
| [aws](#provider_aws) | 5.92.0 |
## Modules
| Name | Source | Version |
| ----------------------------------- | ---------------------------------------------- | ------- |
| [ecs\_cluster](#module_ecs_cluster) | terraform-aws-modules/ecs/aws//modules/cluster | 5.11.2 |
| [parameters](#module_parameters) | terraform-aws-modules/ssm-parameter/aws | 1.1.1 |
| [secrets](#module_secrets) | terraform-aws-modules/secrets-manager/aws | 1.1.2 |
| [vpc](#module_vpc) | terraform-aws-modules/vpc/aws | \~> 5.0 |
## Resources
| Name | Type |
| --------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [aws\_availability\_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws\_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
| ---------------------------------------- | -------------------------------------------------------------------- | ------------- | ------- | :------: |
| [environment](#input_environment) | The environment to deploy to - will prefix the name of all resources | `string` | n/a | yes |
| [secrets](#input_secrets) | A map of secrets to create | `map(string)` | `{}` | no |
| [ssm\_parameters](#input_ssm_parameters) | A map of SSM parameters to create | `map(string)` | `{}` | no |
## Outputs
No outputs.
# live/
Terragrunt Deployment Configuration Directory
## Overview
The `live/` directory contains Terragrunt configurations for each environment in this AWS Iceberg template.
Terragrunt serves as a thin wrapper around Terraform that provides:
* Consistent configuration management across environments
* DRY (Don't Repeat Yourself) infrastructure code
* Dependency handling between modules
* Simplified remote state management
## Folder Structure
The template is built around two primary Terraform modules:
* `base/aws` - Core AWS infrastructure components
* `pipelines/` - Data processing pipeline components
These modules are organized in the `live/` directory with environment-specific configurations:
```
live/
│ ├── base/
│ │ └── aws/
│ │ └── terragrunt.hcl
│ ├── pipelines/
│ │ └── terragrunt.hcl
│ └── root.hcl # Common configuration shared across all modules
```
Each `terragrunt.hcl` file contains:
* Environment-specific input values
* Terraform provider configuration
* Backend configuration for state management
* Module dependencies
## Deployment Options
There are three ways to deploy this infrastructure:
| Deployment Method | State Storage | Recommended Use |
| --------------------------- | ---------------- | ------------------------ |
| Local with local state | Local filesystem | Development/testing only |
| Local with remote state | AWS S3 | Development and staging |
| GitHub CI with remote state | AWS S3 | Production deployments |
### 1. Local Deployment with Local State
Best for quick testing and initial development. It is not recommended for shared or production environments.
```bash
export AWS_PROFILE=
export ENVIRONMENT=
make deploy
```
### 2. Local Deployment with Remote State
Recommended for development work requiring state persistence:
1. First create the S3 remote state bucket by following the instructions in the [production deployment guide](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/production-deployment.md)
2. Then run:
```bash
export AWS_PROFILE=
export ENVIRONMENT=
make deploy
```
### 3. GitHub CI Deployment with Remote State
Recommended for production environments:
1. Configure the remote state following the [production deployment guide](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/production-deployment.md)
2. Push your changes to the configured branch
3. GitHub Actions will execute the deployment process automatically
## Deployment Process
The `make deploy` command performs two sequential operations:
1. **Infrastructure Deployment**: Runs `terragrunt run-all apply` to create all infrastructure resources
2. **Container Deployment**: Builds and pushes Docker images for components in `pipelines/ingest` and `pipelines/transform`
3. **Schema Migration**: Runs `make migrate-schemas` to migrate the schemas for the ingestion and transformation layers
### Why a separate step for container deployment?
This approach resolves circular dependencies between infrastructure and container resources:
1. First, Terraform creates the infrastructure (ECR repositories, Lambda functions, etc.)
2. Then, container images are built and pushed to the newly created repositories
This sequence ensures all necessary infrastructure exists before container deployment occurs, resolving the "chicken and egg" problem where:
* ECR repositories must exist before container images can be pushed
* Lambda functions need a reference to container images that don't exist yet
By separating these processes, we ensure proper resource creation while maintaining infrastructure as code principles.
## Resources
* [Terragrunt](https://terragrunt.gruntwork.io/)
# Add a New Pipeline
This guide explains how to add a new data pipeline to the template.
The pipeline architecture includes:
1. Data ingestion using serverless functions (AWS Lambda) and an ELT tool (dlt)
2. Data lake storage in cloud object storage (AWS S3)
3. Data transformation using an SQL transformation engine ([Amazon Athena](https://aws.amazon.com/athena/)) and dbt.
The boringdata CLI automates many steps along the way.
Before you start, make sure you have installed the boringdata CLI:
{% tabs %}
{% tab title="SSH GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install git+ssh://git@github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% tab title="HTTPS GitHub auth" %}
{% code overflow="wrap" %}
```bash
uv tool install https://github.com/boringdata/boringdata-cli.git --python 3.12
```
{% endcode %}
{% endtab %}
{% endtabs %}
You can then use the boringdata CLI from any directory:
uvx boringdata --help
## Step 1: Add a New Data Source
Let's start by adding a new data source for ingestion.
The template uses [dlt](https://dlthub.com/docs/intro) as the ingestion framework. Check the [dlt ecosystem](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) to find the connector you want.
You can then generate a full ingestion pipeline for this connector by running:
```bash
cd pipelines && uvx boringdata dlt add-source --destination iceberg
```
This command will create the following files:
`pipelines/-lambda.tf` = serverless function infrastructure
`pipelines/ingest/-ingestion/*` = ingestion code embedded in a serverless function
Boringdata will also run some helpful operations:
* Set up a Python virtual environment and install necessary dependencies
* Copy `.env.example` to `.env.local`
* Initialize the data connector
* Parse required secrets from configuration files and update both environment variables and infrastructure configurations
Example using the [Notion API](https://developers.notion.com/) as a source:
```
cd pipelines && uvx boringdata dlt add-source notion --destination iceberg
```
{% hint style="info" %}
You can assign a different name to your source than the connector name.
To do so, add the CLI option: --source-name \
{% endhint %}
## Step 2: Configure Secrets
If your source requires secrets (for example, an API key), update the .env.example.
After deployment, update these secrets manually in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) if needed.
Example for Notion integration:
The following lines should be present in the .env file:
```bash
SOURCES__NOTION__API_KEY="your_api_key_here"
```
## Step 3: Customize the Ingestion Logic
Edit `pipelines/ingest/-ingestion/lambda_handler.py`
{% code title="pipelines/ingest/\-lambda/lambda\_handler.py" %}
```python
#Add missing imports
from import
...
#Update the scope of data to be loaded
load_data =
```
{% endcode %}
Example for Notion integration:
```python
from notion import notion_databases
...
#Update the scope of data to be loaded
load_data = notion_databases(database_ids=["your_database_id"])
```
{% hint style="info" %}
Use the \\_pipeline.py generated by the framework as inspiration
{% endhint %}
## Step 4: Test the Ingestion Function Locally
To verify your changes, run the function locally (using [DuckDB](https://duckdb.org/) as a local target):
```bash
cd pipelines/ingest/-ingestion/ && make run-local
```
This step allows you to test the function and inspect the output data format.
## Step 5: Generate the Source Schema
After running the pipeline locally (see above), generate a source schema definition:
```bash
cd pipelines/
uvx boringdata dlt get-schema \
--engine iceberg \
--output-folder ingest
```
## Step 6: Create Transformation Models
Based on the schema files generated in step 5, boringdata can automatically generate corresponding SQL transformation models for each of the tables using [Amazon Athena](https://aws.amazon.com/athena/):
```bash
cd pipelines/transform
uvx boringdata dbt import-source \
--source-yml ../ingest/-schema/
```
## Step 7: (Optional) Add Workflow Automation
To coordinate the ingestion and transformation steps, add workflow automation using [AWS Step Functions](https://aws.amazon.com/step-functions/):
```bash
cd pipelines
uvx boringdata aws step-function lambda-dbt \
--source-name
```
## Step 8: Deploy the Infrastructure
Finally, deploy the project from the root directory:
```bash
export AWS_PROFILE=your_aws_profile
export ENVIRONMENT=dev
make deploy
```
# CI Deployment
This guide outlines the essential steps for deploying the AWS Data Stack Template in a production environment.
## 1. Create Dedicated Users
A best practice when deploying with Terraform is to create dedicated credentials that Terraform will use during the deployment.
Terraform should only use these users and have the minimal rights required.
For AWS, we provide ready-to-use policies and scripts to create users with your admin account quickly.
{% hint style="info" %}
You will notice that these users are environment-specific. Each Terraform can only deploy to one environment. [FAQ](/template-aws-iceberg/help/faq#what-is-an-environment)
{% endhint %}
```bash
cd init/
export AWS_PROFILE=
make create-tf-user-aws env= aws_region=
```
This script will:
* create a new user called `_AWS_ADMIN`
* assign him this [policy](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/init/init_aws_tf_user_policy.json)
* create files `.env..secrets` and `.env..variables`
## 2. CI/CD Pipeline Setup
The default version of the template does not contain a CICD.
To add it, run:
```bash
# Remove the boringdata's internal test workflow
rm .github/workflows/boringdata-test.yml
# Initialize GitHub workflows for CI/CD
# This will create a .github/workflows/ci.yml file with AWS deployment configuration
uvx boringdata github init --template-type aws
# Initialize Terragrunt configuration to use S3 remote state
# This will create/update the root.hcl file in the live/ directory
uvx boringdata terragrunt init --output-folder live
```
The GitHub Actions workflow requires AWS credentials to deploy the project.
You must, therefore, create the necessary variables and secrets in your GitHub repository.
If you have the GitHub CLI installed and are authorized for your repository, run the following commands from the project root:
```bash
cd init/
make github-ci-setup repo=/ env=
```
This command will automatically create the required variables in GitHub based on your AWS profile and the .env files you previously created.
Alternatively, you can manually set them up in the GitHub console.
That's it; you are now ready to deploy.
## 3. Set Up Terraform State S3 Bucket
You must use a dedicated S3 bucket to store the Terraform state for production deployment.
To create the bucket, run the following command:
```bash
cd init/
export AWS_PROFILE=
make create-tf-bucket env= aws_region=
```
This command will:
* Create a new S3 bucket named `--terraform-state-bucket`.
* Configure the appropriate bucket policies and enable encryption and versioning.
{% hint style="warning" %}
If you have deployed the template using the Quick Start guide (with a local state).
You can either:
* Destroy and start fresh
terragrunt run-all destroy
* Migrate the state:
export AWS\_REGION=\
export ENVIRONMENT=\
terragrunt run-all init -migrate-state -input=true
{% endhint %}
## 4. Deployment
The CI pipeline runs on every merge to the main branch and deploys to the environment defined in the variables.
The CI pipeline will start automatically once you push your changes to the repository.
The CI pipeline consists of two jobs:
* **Terragrunt-apply**: Deploys the infrastructure using Terragrunt
* **Deploy-dockers**: Builds and deploys Docker containers in `pipelines/ingest` and `pipelines/transform`.Only the folders with changes will be processed if the CI pipeline runs after a merge.
* **Perform schema migration**: it runs the command `make migrate` in all `pipelines/ingest/*-schema` folders. (see [FAQ](/template-aws-iceberg/help/faq#iceberg-landing-table-schema-evolution))
### Verify the Deployment
After deployment is complete, verify the setup in your AWS console:
1. Navigate to the AWS Step Functions service
2. Locate your pipeline's step function (e.g., `prod-chess-step-function`)
3. Execute the step function with an empty payload
4. Monitor the execution to ensure the pipeline runs successfully
# FAQ
How do I integrate it into my existing Terraform stack?
Our templates are organized into two types of modules:
• Base modules (base/aws) – Infrastructure components.
• Pipeline modules (pipeline/) – Pipeline-specific components.
Typically, a data team manages the `pipelines/` module.
The company's infra team usually manages the resources defined in the base/aws module.
Having this split already done in this template makes it easy to use the `base/aws` as "spec" for your infra team.
There are too many files—I don't know where to start!
For codebase discovery, LLMs are our best allies.
Get Cursor or Copilot and start asking questions in the chat interface.
The documentation is included in the repo as Markdown files, and LLMs usually find the necessary information independently.
What is an "environment" ?
Throughout this documentation, you will see references to the **ENVIRONMENT**. In our template, the environment represents a specific version or instance of your project, such as `prod`, `dev`, or `ctlq`.
This value is used as a prefix for all resources created in AWS, ensuring that each deployment is isolated and clearly identified.
**How Environments are Used**
* **Resource Naming:**\
Every resource (e.g., S3 buckets, Lambda functions) is prefixed with the environment name. This makes it easy to distinguish between resources belonging to different environments.
* **Deployment Isolation:**\
With Terragrunt, you can deploy the project to multiple environments concurrently. Each environment can have its own set of custom input values and configuration settings. For example, you can deploy the same project in different AWS regions or accounts.
* **Configuration Customization:**\
Different environments allow you to adjust resource configurations according to your needs. You might choose different Lambda settings in production compared to development.
**Choosing a Name for Your Environment**
When selecting a name for your environment, follow these guidelines:
* **Keep it Short and Lowercase:**\
Use concise, lowercase names such as `dev`, `prod`, or `qa`.
* **Avoid Special Characters or Spaces:**\
Stick to alphanumeric characters and simple words to ensure compatibility across all resource naming conventions.
Using clear and consistent environment names helps maintain organization, prevents resource conflicts, and simplifies management across your AWS deployments.
Iceberg Landing Table Schema Evolution
### Overview
Schema evolution in our Iceberg landing tables is managed through Python files in the `pipelines/ingest/-schema/` directory.
### How It Works
1. **Schema Definition Files**
* Each table has a dedicated Python file (e.g., `chess__dlt_version.py`)
* Schemas are defined using PyArrow and managed by PyIceberg
* Files are designed to be idempotent (safe to run multiple times)
2. **Making Schema Changes**
* Add new PyIceberg operations at the bottom of the schema file
* Never modify existing operations to maintain backward compatibility
* Only use idempotent operations
Example schema file:
```python
...
catalog.create_table_if_not_exists(
(NAMESPACE, "table_name"),
pa.schema([
pa.field("column1", "string", nullable=False),
pa.field("column2", "int64", nullable=True),
]),
location=f"s3://{os.environ.get('S3_BUCKET_NAME')}/path/to/table",
)
# New schema evolution operations go here
# Example: Adding a new column
catalog.update_schema(
(NAMESPACE, "table_name"),
[("add", "new_column", pa.string(), True)]
)
```
You can also add partitioning to your table.
```python
# Update partitioning on existing table
with table.update_spec() as update:
update.add_field("id", BucketTransform(16), "bucketed_id")
update.add_field("event_ts", DayTransform(), "day_ts")
```
3. **Applying Schema Changes**
* Use the Makefile in the schema directory:
```bash
# Migrate all tables
cd pipelines/ingest/-schema/
make migrate
# Migrate specific table
make migrate table_name=
```
4. **CI/CD Integration**
* Schema migrations run automatically in CI/CD pipelines.\
The typical CI workflow is the following:\
1: deploy terraform\
2: deploy docker images\
3: run schema migration
### Best Practices
* Ensure changes don't break existing data pipelines. Ideally, you should always add a new column and never delete or modify an existing column.
* Add new columns as nullable to avoid breaking existing writes
* Consider the impact on downstream consumers