#redpanda #kafka #data-streaming

redpanda-transform-sdk

A Rust library for authoring Redpanda Data Transforms

6 releases (stable)

23.3.4 Jan 26, 2024
23.3.4-rc1 Jan 25, 2024
23.3.1-rc4 Dec 18, 2023
1.1.0 Jul 26, 2024
0.0.1-dev+edeac7a3ef Nov 30, 2023

#183 in WebAssembly

Apache-2.0

40KB
648 lines

Redpanda Data Transforms Rust SDK

This crate contains the SDK for Redpanda's inline Data Transforms, based on WebAssembly.

Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro.

Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with rpk transform commands.

See examples/ for sample usages.


lib.rs:

Redpanda Data Transforms Rust library.

Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro.

Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. Thus, transforms must compile to WebAssembly via --target=wasm32-wasi. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with rpk transform commands.

This crate provides a framework for writing transforms.

on_record_written: Transforms individual records after they have been written to an input topic. Written records are output to the destination topic.

Dependencies