live/

What does the live/ folder contain?

This template uses Terragrunt for streamlined deployment and is designed to be deployed either locally or via GitHub CI.

In the live/ directory, you will find Terragrunt configurations for each environment.

Terragrunt is a powerful tool for managing Terraform configurations across multiple modules and environments from a single source of truth. It helps keep infrastructure code DRY (Don't Repeat Yourself) through modular configuration and dependency management.

The template consists of three core Terraform modules:

• base/aws

• base/snowflake

• pipelines/

This structure is replicated in the live/ directory, where each module has its own terragrunt.hcl file containing:

• Input values

• Terraform provider configuration

• Terraform backend configuration

• Dependencies on other modules

live/
├── prod/
│   ├── base/
│   │   ├── aws/
│   │   │   └── terragrunt.hcl
│   │   └── snowflake/
│   │       └── terragrunt.hcl
│   ├── pipelines/
│   │   └── terragrunt.hcl
│   └── root.hcl

Note that all terragrunt.hcl files reference the root.hcl file, where common configurations are defined.

There are three ways to deploy the project:

1. Local deployment with local state

2. Local deployment with remote state

3. GitHub deployment with remote state

Option 1 is ideal for quickly getting started but is not recommended for production deployments.

To deploy using local state, follow the instructions in the Get Started file and run:

export AWS_PROFILE=<your_profile>
export SNOWFLAKE_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy

The deploy command will:

• Run terragrunt run-all apply to deploy all Terraform modules in the correct order

• Build and push all Docker images found in the pipelines/ingest and pipelines/transform folders

Why Separate Terraform and Docker Processes?

The infrastructure deployment and container build processes must be separated due to dependencies between resources.

For example, when Terraform creates an AWS Lambda function with an associated ECR repository, the ECR repository must exist before the container image can be built and pushed to it. However, the container image must be built and pushed to ECR before the Lambda function can be fully functional.

This circular dependency is resolved by:

1. First, deploying the infrastructure (ECR, Lambda, etc.) via Terraform/Terragrunt

2. Then, building and pushing container images in a separate process

This two-step approach ensures that all required infrastructure exists before building and deploying containers.

Local Deployment with Remote State

Do not store the Terraform state on your local machine. Instead, store the state in a remote backend such as AWS S3.

Follow the instructions in the production-deployment file to create the S3 bucket and run:

export AWS_PROFILE=<your_profile>
export SNOWFLAKE_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy

GitHub Deployment with Remote State

Follow the instructions in the production-deployment file. Push your changes to the repository, and the CI will automatically initiate the deployment.