#parquet #thrift #async #byte

polars-parquet-format

Safe Parquet and Thrift reader and writer (sync and async)

1 unstable release

0.1.0 Oct 27, 2024

#838 in Database interfaces

Download history 9197/week @ 2024-12-08 8880/week @ 2024-12-15 7747/week @ 2024-12-22 7889/week @ 2024-12-29 12543/week @ 2025-01-05 13179/week @ 2025-01-12 10885/week @ 2025-01-19 11574/week @ 2025-01-26 15321/week @ 2025-02-02 16240/week @ 2025-02-09 15215/week @ 2025-02-16 17497/week @ 2025-02-23 17462/week @ 2025-03-02 17863/week @ 2025-03-09 18063/week @ 2025-03-16 18379/week @ 2025-03-23

72,817 downloads per month
Used in 135 crates (via polars-parquet)

MIT AND Apache-2.0

550KB
13K SLoC

parquet-format-safe

This crate contains an implementation of Thrift and generated Rust code associated to Parquet's thrift definition.

  • supports sync and async read API
  • supports sync and async write API
  • the write API returns the number of written bytes
  • the read API is panic free
  • the read API has a bound on the maximum number of possible bytes read, to avoid OOM.

The Rust generated code is done by a fork of thrift's compiler, available at https://github.com/coastalwhite/thrift/tree/safe.

Usage

To regenerate the thrift format implementation in Rust.

nix-shell
generate_parquet_format
mv parquet.rs src/parquet_format.rs

Dependencies

~225KB