Singer Target for Iceberg Tables

This is a Singer target that loads data from Singer streams into Iceberg tables. This reposotory provides mulitple singer targets, one for each iceberg catalog. These are:

• SQL catalog target

Features

• Creates Iceberg tables automatically if they don't exist
• Incrementally loads versions into Iceberg tables
• Converts Singer stream schemas into Iceberg table schemas
• Validates records against the Singer stream schema
• Generates metadata about syncs for data governance

Usage

The target ingests singer messages into the Icberg tables and stores the state in the singer-bookmark property.

Sync mode

To run:

target-iceberg-sql --config config.json

Configuration

Example:

{
    "streams": {
      "inventory-orders": { 
        "identifier": "bronze.inventory.orders",
        "replicationMethod": "LOG_BASED"
      },
      "inventory-customers": { 
        "identifier": "bronze.inventory.customers",
        "replicationMethod": "LOG_BASED"
      },
      "inventory-products": { 
        "identifier": "bronze.inventory.products",
        "replicationMethod": "LOG_BASED"
      }
    },
    "bucket": "s3://example-postgres",
    "catalogName": "bronze",
    "catalogUrl": "postgres://postgres:postgres@postgres:5432",
    "awsRegion": "us-east-1",
    "awsAccessKeyId": "AKIAIOSFODNN7EXAMPLE",
    "awsSecretAccessKey": "$AWS_SECRET_ACCESS_KEY",
    "awsEndpoint": "http://localstack:4566",
    "awsAllowHttp": "true"
}

The configuration consists of 3 parts:

• General parameters
• Catalog parameters
• Object store parameters

General parameters

The general parameters apply to each catalog and object store.

Catalog parameters

Only one set of catalog parameters should be used in the configuration. Choose which catalog you want to use.

SQL catalog

Object store parameters

Only one set of object store parameters should be used in the configuration. Choose which object store you want to use.

AWS S3

Docker containers

• dashbook/target-iceberg:sql

Contributing

Feel free to open issues for any feedback or ideas! PRs are welcome.