1 unstable release
new 0.1.0 | Mar 30, 2025 |
---|
#20 in #brain
3.5MB
5K
SLoC
tpchgen-rs
Blazing fast TPCH benchmark data generator, in pure Rust with zero dependencies.
Features
- Blazing Speed 🚀
- Obsessively Tested 📋
- Fully parallel, streaming, reasonable memory usage 🧠
Try now!
First install Rust and this tool:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install tpchgen-cli
# create Scale Factor 10 (3.6GB, 8 files, 60M rows in lineitem) in 5 seconds on a modern laptop
tpchgen-cli -s 10 --format=parquet
Performance
tpchgen-cli
is more than 10x faster than any other
TPCH generator we know of. On a 2023 Mac M3 Max laptop, it easily generates data
faster than can be written to SSD. See BENCHMARKS.md
for more details on performance and benchmarking.
Testing
This crate has extensive tests to ensure correctness. We compare the output of
this crate with the original dbgen
implementation as part of every checkin.
See TESTING.md for more details.
Crates
tpchgen
is the library that implements the data generation logic for TPCH and it can be used to embed data generation logic natively in Rust.
tpchgen-arrow
is a library for generating in memory Apache Arrow
record batches for each of the TPCH tables
tpchgen-cli
is adbgen
compatible CLI tool that generates tables from the TPCH benchmark dataset.
Contributing
Pull requests are welcome. For major changes, please open an issue first for discussion. See our contributors guide for more details.
Architecture
Please see architecture guide for details on how the code is structured.
License
The project is licensed under the APACHE 2.0 license.
References
Dependencies
~42MB
~804K SLoC