#jsonl #length #input #format #json2bin

app json2bin

A fast jsonl to RWKV binidx converter in Rust

7 releases

0.2.1 Mar 13, 2025
0.2.0 Aug 10, 2024
0.1.4 Aug 4, 2024
0.1.3 Jul 14, 2024

#97 in Machine learning

Download history 1/week @ 2025-02-10 4/week @ 2025-02-17 4/week @ 2025-02-24 4/week @ 2025-03-03 125/week @ 2025-03-10 30/week @ 2025-03-17 2/week @ 2025-03-24

157 downloads per month

Apache-2.0

165KB
254 lines

Json2bin

Crates.io Version Crates.io Downloads License: Apache 2.0

A fast multithreading Jsonl converter to RWKV binidx files written in Rust.

performance-multithreading

Installation

$ cargo install json2bin

Usage

$ json2bin -h
Json converter to RWKV binidx file format
Usage: json2bin [OPTIONS] --input <INPUT>

Options:
  -i, --input <INPUT>            Jsonlines file to read
  -o, --output-dir <OUTPUT_DIR>  Output directory for binidx files [default: -]
  -t, --thread <THREAD>          Number of threads [default: 8]
  -v, --verbose                  Verbosity
  -h, --help                     Print help
  -V, --version                  Print version

Following command will convert the jsonl file src/sample.jsonl into src/sample.bin and src/sample.idx files.

$ json2bin -i src/sample.jsonl

The output directory can be set with the argument "--output-dir <OUTPUT_DIR>" or "-o <OUTPUT_DIR>"

$ json2bin -i src/sample.jsonl -o output

The default threads number is 8, it can be changed with the argument "--thread" or "-t"

$ json2bin -i src/sample.jsonl -t 4

Performance comparison

We converted a 19GB English Wikipedia (20231101.en) in jsonl format to binidx format in M2 Apple machine. The Rust json2bin run with 7 threads, and it was 70 times faster than the Python json2binidx:

  • The Python json2binidx: 1:01:45 or 5.13MB/s
  • This Rust json2bin: 52.64s or 360.86MB/s

Dependencies

~8–21MB
~209K SLoC