# Ingestion: dlt + lambda

## Overview

This example demonstrates a serverless data ingestion pipeline that:

1. Fetches chess data from an external source and processes it using [dlt](https://dlthub.com/)
2. Writes the data to Apache Iceberg tables on S3

The pipeline runs as an AWS Lambda function packaged in a Docker container.

<figure><img src="/files/c7kWK1ohyu3vVfRlIRFb" alt=""><figcaption></figcaption></figure>

## How It Works

### Infrastructure Components

* **AWS Lambda**: Executes the ingestion code on demand
* **Amazon ECR**: Stores the Docker container image
* **Amazon Glue**: Hosts the Iceberg Catalog
* **Amazon S3**: Hosts the Iceberg tables
* **AWS Secrets Manager**: Stores credentials and configuration
* **Terraform**: Provisions and manages all infrastructure

### Code Structure

```
pipelines/
├── chess_lambda.tf           # Terraform creating the lambda function and ECR repository
├── ingestion_bucket.tf       # Terraform creating S3 bucket
└── ingest/
    └── chess-ingestion/      # Lambda function code
        ├── Dockerfile
        ├── lambda_handler.py # Lambda code with DLT pipeline
        └── ...
    └── chess-schema/         # Iceberg schema definition in Glue Catalog
        ├── table_schema.py
        └── ...
```

### Data Flow Process

The pipeline follows these steps:

1. **Extraction and Loading to S3**: DLT loads data as Parquet files to an ingestion S3 bucket following this path pattern:

   ```
   {source_name}/raw/{table_name}/{load_id}.{file_id}.{ext}
   ```
2. **Iceberg Integration**: PyIceberg adds these files to Iceberg tables.\
   Iceberg tables are located in the same ingestion S3 bucket under:

   ```
   {source_name}/landing/{table_name}/
   ```

Files are inserted in append mode.

## Development Guide

### 1. Local Development with DuckDB

For rapid iteration without AWS resources, use DuckDB as the destination:

1. Create a `.env.local` file with:

   ```
   DESTINATION=duckdb
   # Add any source-specific credentials here
   ```
2. Run the pipeline locally:

   ```bash
   make run-local
   ```
3. Examine results in the local .duckdb database

### 2. Local Development with S3

To run the lambda with Iceberg destination:

1. Configure `.env.local` with:

   ```
   DESTINATION=filesystem
   AWS_REGION=<your-aws-region>
   S3_BUCKET_NAME=<your-s3-bucket-name>
   AWS_PROFILE=<your-aws-profile>
   # Add any source-specific credentials here
   ```
2. Run with the same command:

   ```bash
   make run-local
   ```

### 3. Testing on AWS

Once your code is deployed to AWS you can run the lambda with:

```bash
export AWS_PROFILE=<your_profile>
make run-lambda env=<your_environment>
```

### 4. VSCode Debugging

For interactive debugging, add this to `.vscode/launch.json`:

```json
{
    "name": "Debug chess lambda",
    "type": "debugpy",
    "request": "launch",
    "program": "${workspaceFolder}/pipelines/ingest/chess-ingestion/lambda_handler.py",
    "console": "integratedTerminal",
    "cwd": "${workspaceFolder}/pipelines/ingest/chess-ingestion",
    "justMyCode": false
}
```

## Schema Management

dlt needs to run a first time to provide the target schema definition.

After running the pipeline locally (see above), generate a source schema definition:

```bash
cd pipelines/
uvx boringdata dlt get-schema chess
```

This will generate one schema file per table in the `ingest/chess-schema` folder to create the iceberg table in the AWS Glue Catalog.

These scripts will be run automatically by the CI/CD pipeline.

More details about schema management can be found [here](https://dlthub.com/docs/guides/schema-management).

## Manual Deployment

For manual deployment:

```bash
# Set required environment variables
export AWS_PROFILE=<your_profile>

# Build and deploy
make deploy env=<your_environment>
```

This process:

1. Builds the Docker image locally
2. Pushes it to ECR
3. Updates the Lambda to use the new image

## Common Commands

```bash
# Development
make run-local                        # Run locally with settings from .env.local
make run-lambda env=<environment>     # Execute on AWS Lambda

# Deployment
make build env=<environment>          # Build Docker image
make deploy env=<environment>         # Build and deploy to ECR

# Utilities
make help                             # Show all available commands
```

## Resources

* [DLT Documentation](https://dlthub.com/docs/)
* [PyIceberg Documentation](https://py.iceberg.apache.org/)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.boringdata.io/template-aws-iceberg/project-structure/pipelines/chess-ingestion.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.