#kafka #serde #derive-deserialize #database-interfaces #io-write

kafka-serde

serialization and deserialization for the Kafka protocol

1 unstable release

0.1.0 Feb 12, 2021

#1532 in Encoding

Download history 50/week @ 2024-11-13 68/week @ 2024-11-20 32/week @ 2024-11-27 53/week @ 2024-12-04 54/week @ 2024-12-11 13/week @ 2024-12-18 7/week @ 2025-01-01 22/week @ 2025-01-08 30/week @ 2025-01-15 8/week @ 2025-01-22 6/week @ 2025-02-05 28/week @ 2025-02-12 55/week @ 2025-02-19 9/week @ 2025-02-26

98 downloads per month

Apache-2.0 OR MIT

35KB
973 lines

kafka-serde

Rust's serde implementation for the Kafka protocol.

This allows you to serialize and deserialize kafka payloads. It can be used as a building block for a native-rust kafka client.

Usage

Serializing the kafka request header:

use serde::Serialize;
use std::io::{Write, Cursor};

#[derive(Serialize, Debug)]
struct RequestHeader {
    api_key: i16,
    api_version: i16,
    correlation_id: i32,
    client_id: &'static str,
}

let req = RequestHeader {
    api_key: 0,
    api_version: 0,
    correlation_id: 1,
    client_id: ""
};

let mut x = Cursor::new(Vec::<u8>::new());
kafka_serde::to_writer(x, &req).unwrap();

Deserializing the kafka response header:

use serde::Serialize;

#[derive(Deserialize, Default, Debug, Clone)]
struct ResponseHeader {
    pub correlation: i32,
}

let data : Vec<u8> = [0x0, 0x0, 0x0, 0x1];
let resp: ResponseHeader = kafka_serde::from_bytes(&data).unwrap();

Support

All Kafka protocol types are listed here

  • The fixed-size types are supported and map to their native rust types (int8 -> i8, etc).
  • Strings can be deserialized to both String and &str, and similarly for bytes
  • UUID is not supported (yet)
  • variable-size types like VARLONG and COMPACT_STRING are not supported (yet)

Dependencies

~0.3–0.9MB
~20K SLoC