live/

Terragrunt Deployment Configuration Directory

Overview

The live/ directory contains Terragrunt configurations for each environment in this AWS Iceberg template.

Terragrunt serves as a thin wrapper around Terraform that provides:

• Consistent configuration management across environments

• DRY (Don't Repeat Yourself) infrastructure code

• Dependency handling between modules

• Simplified remote state management

Folder Structure

The template is built around two primary Terraform modules:

• base/aws - Core AWS infrastructure components

• pipelines/ - Data processing pipeline components

These modules are organized in the live/ directory with environment-specific configurations:

live/
│   ├── base/
│   │   └── aws/
│   │       └── terragrunt.hcl
│   ├── pipelines/
│   │   └── terragrunt.hcl
│   └── root.hcl      # Common configuration shared across all modules

Each terragrunt.hcl file contains:

• Environment-specific input values

• Terraform provider configuration

• Backend configuration for state management

• Module dependencies

Deployment Options

There are three ways to deploy this infrastructure:

Deployment Method	State Storage	Recommended Use
Local with local state	Local filesystem	Development/testing only
Local with remote state	AWS S3	Development and staging
GitHub CI with remote state	AWS S3	Production deployments

1. Local Deployment with Local State

Best for quick testing and initial development. It is not recommended for shared or production environments.

export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy

2. Local Deployment with Remote State

Recommended for development work requiring state persistence:

1. First create the S3 remote state bucket by following the instructions in the production deployment guide

2. Then run:

export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy

3. GitHub CI Deployment with Remote State

Recommended for production environments:

1. Configure the remote state following the production deployment guide

2. Push your changes to the configured branch

3. GitHub Actions will execute the deployment process automatically

Deployment Process

The make deploy command performs two sequential operations:

1. Infrastructure Deployment: Runs terragrunt run-all apply to create all infrastructure resources

2. Container Deployment: Builds and pushes Docker images for components in pipelines/ingest and pipelines/transform

3. Schema Migration: Runs make migrate-schemas to migrate the schemas for the ingestion and transformation layers

Why a separate step for container deployment?

This approach resolves circular dependencies between infrastructure and container resources:

1. First, Terraform creates the infrastructure (ECR repositories, Lambda functions, etc.)

2. Then, container images are built and pushed to the newly created repositories

This sequence ensures all necessary infrastructure exists before container deployment occurs, resolving the "chicken and egg" problem where:

• ECR repositories must exist before container images can be pushed

• Lambda functions need a reference to container images that don't exist yet

By separating these processes, we ensure proper resource creation while maintaining infrastructure as code principles.

Resources

• Terragrunt