CLI

The BoringData CLI is a tool for generating boilerplate code and adding integration/pipelines to your stack.

This document provides a comprehensive overview of all available commands.

Installation

SSH GitHub auth

uv tool install git+ssh://[email protected]/boringdata/boringdata-cli.git --python 3.12

HTTPS GitHub auth

Table of Contents

• AWS Commands

• DBT Commands

• DLT Commands

• Snowflake Commands

• Project Commands

• Terragrunt Commands

• GitHub Commands

AWS Commands

Commands for managing AWS resources with BoringData.

aws bucket

Create a new S3 bucket configuration with versioning and encryption.

boringdata aws bucket <bucket-name> [--output-folder <path>]

Arguments:

• bucket-name: Name of the S3 bucket to create (required)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata aws bucket my-data-bucket
boringdata aws bucket my-data-bucket --output-folder pipelines/

Files Created:

.
└── <bucket_name>_bucket.tf      # Main bucket configuration

aws lambda

Create a new AWS Lambda function with optional triggers.

boringdata aws lambda <lambda-name> [--output-folder <path>] [--trigger <triggers>]

Arguments:

• lambda-name: Name of the Lambda function to create (required)

• --output-folder: Directory where files will be created (default: current directory)

• --trigger: Comma-separated list of triggers to enable. Options: 'sqs', 'cron'

Example:

boringdata aws lambda my-function
boringdata aws lambda my-function --trigger sqs,cron
boringdata aws lambda my-function --output-folder ./infrastructure

Files Created:

.
├── <lambda_name>_lambda.tf       # Lambda function configuration
└── <lambda_name>-lambda/         # Lambda function code
    ├── .env.example             # Environment variables template
    ├── .gitignore              # Git ignore file
    ├── Dockerfile              # Lambda container definition
    ├── requirements.txt        # Python dependencies
    └── lambda_handler.py       # Lambda function code

aws step-function

Create a new AWS Step Function configuration for orchestration.

boringdata aws step-function <type> [--source-name <name>] [--dbt-command <command>] [--output-folder <path>]

Arguments:

• type: Type of step function to create (required). Options: lambda-dbt

• --source-name: Name of the source Lambda/ECS task (required for lambda-dbt and ecs-dbt types)

• --dbt-command: DBT command to execute (default: "run")

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata aws step-function lambda-dbt --source-name my-source --dbt-command "run --select tag:daily"

Files Created:

.
├── <source_name>_step_function.tf    # Step function configuration
└── orchestrate/                      # Step function definitions
    └── <source_name>_step_function.json  # Step function state machine

DBT Commands

Commands for managing dbt projects with BoringData.

dbt init

Initialize a new dbt project with configuration.

boringdata dbt init [--output-folder <path>] [--target <type>]

Arguments:

• --output-folder: Directory where files will be created (default: current directory)

• --target: Target of the project: snowflake or athena (default: snowflake)

Example:

boringdata dbt init
boringdata dbt init --output-folder ./transform --target athena

Files Created:

.
├── ecs_task_dbt.tf             # ECS task definition for dbt
└── transform/                  # dbt project directory
    ├── ...
    ├── Dockerfile              # Container definition for dbt
    ├── Makefile                # Common dbt commands
    ├── dbt_project.yml         # dbt project configuration
    ├── macros/                 # Custom macros
    ├── sources/                # Source models (bronze)
    └── models/                 # dbt models
        ├── marts/              # Business-layer models (gold)
        └── staging/            # Staging-layer models (silver)

dbt import-source

Import sources and generate corresponding dbt models.

boringdata dbt import-source --source <path> [--output-folder <path>] [--schema-name <name>] [--target <type>]

Arguments:

• --source: Path to the source YAML file or folder <source_name>-schema (required)

• --output-folder: Directory where files will be created (default: current directory)

• --schema-name: Name of the DB schema where sources are stored (default: LANDING)

• --target: Target of the dbt project (snowflake, athena)

Example:

boringdata dbt import-source --source ./sources/my_source.yml
boringdata dbt import-source --source ./sources/my_source-schema --target athena

Files Created:

.
├── sources/                               # Source definitions
│   └── <source_name>.yml                  # Source configuration
└── models/                                # Generated models
    └── staging/                           # Staging models
        └── <source_name>/                 # Source-specific models
            ├── stg_<source_name>_<model_name>.sql  # Generated staging models
            └── ...                        # Additional models as needed

DLT Commands

Commands for managing DLT pipelines with BoringData.

dlt add-source

Add a new DLT source using Lambda function.

boringdata dlt add-source <connector-name> [--source-name <name>] [--destination <type>] [--output-folder <path>]

Arguments:

• connector-name: Name of the DLT connector to use (required)

• --source-name: Name of your source, defaults to connector_name

• --destination: Destination to use for the lambda (s3 or iceberg) (default: s3)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata dlt add-source chess
boringdata dlt add-source chess --source-name my_chess --destination iceberg
boringdata dlt add-source chess --output-folder ./pipelines

Files Created:

.
├── <source_name>_lambda.tf          # Lambda function configuration
└── ingest/                          # Lambda function code
    └── <source_name>-ingestion/     # Source-specific Lambda
        ├── .dlt/                    # DLT configuration
        ├── .env.example             # Environment variables template
        ├── .env.local               # Local environment variables
        ├── Dockerfile               # Lambda container definition
        ├── Makefile                 # Common commands
        ├── lambda_handler.py        # Lambda function code
        ├── requirements.txt         # Python dependencies
        └── requirements-dev.txt     # Development dependencies

dlt get-schema

Get the destination table schema of the DLT pipeline.

boringdata dlt get-schema <source-name> [--engine <type>] [--target <format>] [--output-folder <path>]

Arguments:

• source-name: Name of your source (required)

• --engine: Target typing format: arrow or snowflake (default: arrow)

• --target: Target output format: yaml or pyiceberg (default: yaml)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata dlt get-schema my-source
boringdata dlt get-schema my-source --engine snowflake --target yaml
boringdata dlt get-schema my-source --engine arrow --target pyiceberg
boringdata dlt get-schema my-source --output-folder ./schemas

Files Created (for yaml target):

.
└── <source_name>_source_schema.yml    # Data contract for the DLT pipeline

Files Created (for pyiceberg target):

.
└── <source_name>-schema/              # Schema migration directory
    ├── Makefile                       # Makefile to run schema migration
    └── <source_name>_<table_name>.py  # Migration script for each table

Snowflake Commands

Commands for managing Snowflake resources with BoringData.

snowflake tf-module

Generate a Terraform Snowflake module for database and warehouse management.

boringdata snowflake tf-module [--output-folder <path>]

Arguments:

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata snowflake tf-module
boringdata snowflake tf-module --output-folder ./modules/snowflake

Files Created:

.
├── data.tf                        # Data sources
├── db.tf                          # Database configuration
├── locals.tf                      # Local variables
├── schema.tf                      # Schema configuration
├── tech_user.tf                   # Technical user configuration
├── versions.tf                    # Provider and terraform versions
└── warehouse.tf                   # Warehouse configuration

Project Commands

Commands for managing project-level resources with BoringData.

project init

Initialize a new BoringData project with infrastructure as code.

boringdata project init <project-type> [--local-state] [--output-folder <path>]

Arguments:

• project-type: Type of project to create (required). Options: 'aws', 'aws-snowflake'

• --local-state: Use local state instead of remote state in S3 (default: remote S3 state)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata project init aws
boringdata project init aws-snowflake --local-state
boringdata project init aws --output-folder ./my-project

Files Created:

.
├── README.md                       # Project documentation
├── Makefile                        # Common commands
├── .gitignore                      # Git ignore file
├── base/                           # Base infrastructure
│   └── aws/                        # AWS-specific base
│       └── main.tf                # Main configuration
│   └── snowflake/                 # Snowflake-specific base
│       └── main.tf                # Main configuration
├── pipelines/                      # Data pipeline infrastructure
│   └── infra/                     # Infrastructure configuration
│       └── aws/                   # AWS pipeline resources
│           └── main.tf            # Main configuration
└── live/                           # Terragrunt configurations
    ├── root.hcl                   # Root Terragrunt configuration
    ├── base/                      # Base infrastructure live configs
    │   ├── aws/                  # AWS base live config
    │   │   └── terragrunt.hcl   # Terragrunt configuration
    │   └── snowflake/           # Snowflake base live config
    │       └── terragrunt.hcl   # Terragrunt configuration
    └── pipelines/                # Pipeline live configs
        └── infra/               # Infrastructure live configs
            └── aws/            # AWS pipeline live config
                └── terragrunt.hcl  # Terragrunt configuration

Terragrunt Commands

Commands for managing Terragrunt configurations with BoringData.

terragrunt init

Initialize a new Terragrunt directory by generating the root.hcl file.

boringdata terragrunt init [--local-state] [--output-folder <path>]

Arguments:

• --local-state: Use local state instead of remote state in S3 (default: False)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata terragrunt init
boringdata terragrunt init --local-state
boringdata terragrunt init --output-folder live/

Files Created:

.
└── root.hcl                    # Root Terragrunt configuration

terragrunt add-module

Add a new Terragrunt module configuration.

boringdata terragrunt add-module --module-path <path> [--output-folder <path>]

Arguments:

• --module-path: Relative path to the module from --output-folder (required)

• --output-folder: Directory where terragrunt.hcl file will be created (default: current directory)

Example:

boringdata terragrunt add-module --module-path ../../../base/aws --output-folder live/base/aws

Files Created:

.
└── terragrunt.hcl        # Module Terragrunt configuration

GitHub Commands

Commands for managing GitHub workflows with BoringData.

github init

Initialize GitHub workflows for CI/CD.

boringdata github init [--template-type <type>] [--output-folder <path>]

Arguments:

• --template-type: Type of template to use (aws or aws-snowflake) (default: aws)

• --output-folder: Directory where files will be created (default: current directory)

Example:

boringdata github init
boringdata github init --template-type aws-snowflake
boringdata github init --output-folder ./my-project

Files Created:

.
└── .github/
    └── workflows/
        └── ci.yml                # CI workflow