# live/

## Overview

The `live/` directory contains Terragrunt configurations for each environment in this AWS Iceberg template. 

Terragrunt serves as a thin wrapper around Terraform that provides:

* Consistent configuration management across environments
* DRY (Don't Repeat Yourself) infrastructure code
* Dependency handling between modules
* Simplified remote state management

## Folder Structure

The template is built around two primary Terraform modules:

* `base/aws` - Core AWS infrastructure components
* `pipelines/` - Data processing pipeline components

These modules are organized in the `live/` directory with environment-specific configurations:

```
live/
│   ├── base/
│   │   └── aws/
│   │       └── terragrunt.hcl
│   ├── pipelines/
│   │   └── terragrunt.hcl
│   └── root.hcl      # Common configuration shared across all modules
```

Each `terragrunt.hcl` file contains:

* Environment-specific input values
* Terraform provider configuration
* Backend configuration for state management
* Module dependencies

## Deployment Options

There are three ways to deploy this infrastructure:

| Deployment Method           | State Storage    | Recommended Use          |
| --------------------------- | ---------------- | ------------------------ |
| Local with local state      | Local filesystem | Development/testing only |
| Local with remote state     | AWS S3           | Development and staging  |
| GitHub CI with remote state | AWS S3           | Production deployments   |

### 1. Local Deployment with Local State

Best for quick testing and initial development. It is not recommended for shared or production environments.

```bash
export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy
```

### 2. Local Deployment with Remote State

Recommended for development work requiring state persistence:

1. First create the S3 remote state bucket by following the instructions in the [production deployment guide](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/production-deployment.md)
2. Then run:

```bash
export AWS_PROFILE=<your_profile>
export ENVIRONMENT=<environment>
make deploy
```

### 3. GitHub CI Deployment with Remote State

Recommended for production environments:

1. Configure the remote state following the [production deployment guide](https://github.com/boringdata/boringdata-template-aws-iceberg/blob/main/production-deployment.md)
2. Push your changes to the configured branch
3. GitHub Actions will execute the deployment process automatically

## Deployment Process

The `make deploy` command performs two sequential operations:

1. **Infrastructure Deployment**: Runs `terragrunt run-all apply` to create all infrastructure resources
2. **Container Deployment**: Builds and pushes Docker images for components in `pipelines/ingest` and `pipelines/transform`
3. **Schema Migration**: Runs `make migrate-schemas` to migrate the schemas for the ingestion and transformation layers

### Why a separate step for container deployment?

This approach resolves circular dependencies between infrastructure and container resources:

1. First, Terraform creates the infrastructure (ECR repositories, Lambda functions, etc.)
2. Then, container images are built and pushed to the newly created repositories

This sequence ensures all necessary infrastructure exists before container deployment occurs, resolving the "chicken and egg" problem where:

* ECR repositories must exist before container images can be pushed
* Lambda functions need a reference to container images that don't exist yet

By separating these processes, we ensure proper resource creation while maintaining infrastructure as code principles.

## Resources

* [Terragrunt](https://terragrunt.gruntwork.io/)