> For the complete documentation index, see [llms.txt](https://docs.boringdata.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.boringdata.io/template-aws-iceberg/introduction/key-concepts.md).

# Key Concepts

This section explains the core concepts and architecture of this template.

## Code Structure

The template's code is organized into three main components:

```
📁
├── 📁 pipelines/             # Data pipelines:
│   ├── 📁 ingest/                      # Data ingestion layer
│   ├── 📁 transform/                   # Data transformation layer
│   └── 📁 orchestrate/                 # Workflow orchestration layer
│
├── 📁 base/                  # Cloud infrastructure (VPC, roles, users, compute cluster, etc.)
│
└── 📁 live/                  # Environment-specific deployment configuration
```

Each component is documented separately here:

{% content-ref url="/pages/G2khJSaacMix8oYa7iVH" %}
[pipelines/](/template-aws-iceberg/project-structure/pipelines.md)
{% endcontent-ref %}

{% content-ref url="/pages/qo9vW9N1VXDI9z43Fdr5" %}
[base/aws/](/template-aws-iceberg/project-structure/aws.md)
{% endcontent-ref %}

{% content-ref url="/pages/txxF4gycrydceZI6n15j" %}
[live/](/template-aws-iceberg/project-structure/live.md)
{% endcontent-ref %}

## Data Flow

1. Source data is ingested into [Apache Iceberg](https://iceberg.apache.org/) landing tables: code in `pipelines/ingest/<source_name>-*/`
2. Data transformations are applied to create staging tables using SQL engine ([Amazon Athena](https://aws.amazon.com/athena/)): code in `pipelines/transform/`

## Data Pipeline Architecture

Our data platform follows a layered architecture:

### 1. Data Ingestion Layer

For each source, the ingestion layer is structured as follows:

```
📁 pipelines/
├── 📁 ingest/
│   ├── 📁 <source>-ingestion/      # Core ingestion logic
│   │
│   └── 📁 <source>-schema/         # Iceberg Table schema definitions
│       └── <table_name>.py
│       └── ...
│
└── <source>_*.tf                   # Infrastructure definition (serverless functions, containers, etc.)
```

Each source has:

* A folder `pipelines/ingest/<source>-ingestion/` containing the core ingestion logic packaged in a container
* Infrastructure as Code files in `pipelines/*tf` for deploying this ingestion container (as serverless functions (AWS lambda) or container tasks ([Amazon ECS](https://aws.amazon.com/ecs/)))
* A folder for the management of the landing tables (`<source>-schema/`)

{% hint style="info" %}
More info about landing table schema evolution in [FAQ](/template-aws-iceberg/help/faq.md#iceberg-landing-table-schema-evolution)
{% endhint %}

The template comes with an example data ingestion pipeline deployed as a serverless function (lambda) using [dlt](https://dlthub.com/docs/intro); more details here:

{% content-ref url="/pages/QmSXv61cHRK1XAaEgwKO" %}
[Ingestion: dlt + lambda](/template-aws-iceberg/project-structure/pipelines/chess-ingestion.md)
{% endcontent-ref %}

### 2. Data Transformation Layer

The transformation layer is a [dbt](https://docs.getdbt.com/) project that transforms the data into Iceberg staging tables using the SQL query engine [Amazon Athena](https://aws.amazon.com/athena/).

This project is located in the `pipelines/transform` folder:

```
📁 pipelines/
├── 📁 transform/                   # SQL transformation project
│   ├── 📁 models/
│   │   ├── 📁 staging/            # Raw table connections
│   │   └── 📁 marts/              # Transformations
│   │
│   ├── dbt_project.yml
│   └── Dockerfile                  # For container deployment
│
└── ecs_task_dbt.tf                 # Infrastructure definition for running dbt container
```

This transformation project runs on container infrastructure ([Amazon ECS](https://aws.amazon.com/ecs/) Fargate).

More details on how this transformation project is structured here:

{% content-ref url="/pages/vC7jupqVP6pcjq6OCJ4f" %}
[Transformation: dbt](/template-aws-iceberg/project-structure/pipelines/transform.md)
{% endcontent-ref %}

### 3. Workflow Orchestration Layer

The orchestration layer coordinates the execution of the ingestion and transformation layers using workflow automation.

This template proposes an example orchestration using [AWS Step Functions](https://aws.amazon.com/step-functions/):

```
📁 pipelines/
├── 📁 orchestrate/
│   └── <source>_step_function.json  # Workflow definition
│
└── <source>_step_function.tf        # Creates an orchestration workflow in AWS Step Functions
```

<div align="center"><img src="/files/gWaz2PlyVokT92tbGOuQ" alt="Chess Pipeline Workflow" width="375"></div>

## Deployment

This template is ready to be deployed.

The stack deployment is structured in 3 steps:

* First, the infrastructure modules (base/ and pipelines/) are deployed using [Terragrunt](https://terragrunt.gruntwork.io/) for infrastructure management
* Then, the containers for the ingestion and transformation layers are built and pushed to the container registry
* Finally, the schema evolution scripts of the Iceberg landing tables are run

<figure><img src="/files/W8wRIXYoBiBa1pLwWWpi" alt=""><figcaption></figcaption></figure>

If you want to get started quickly and deploy the template from your machine, follow this guide:

{% content-ref url="/pages/Sp2fdZp3rxZePnTg8vMP" %}
[Get Started](/template-aws-iceberg/introduction/get-started.md)
{% endcontent-ref %}

To get started deploying from [GitHub Actions](https://github.com/features/actions) CI/CD, head there:

{% content-ref url="/pages/MMbEuLtK7JyjMXkDAboW" %}
[CI Deployment](/template-aws-iceberg/guides/production-deployment.md)
{% endcontent-ref %}

## Makefile

The template is composed of many Makefiles providing utilities.

Here are some examples:

* `make deploy` in the root folder will deploy the template from your machine
* `make build` in a folder with a Dockerfile will build the container
* `make local-run` in a serverless function folder will test the function locally
* etc

Everywhere you see a Makefile, run `make` and the list of possible actions will be listed

{% content-ref url="/pages/Sp2fdZp3rxZePnTg8vMP" %}
[Get Started](/template-aws-iceberg/introduction/get-started.md)
{% endcontent-ref %}