#tokenizer #version #python

bytepiece

Rust version of bytepiece tokenizer

2 unstable releases

0.2.0 Sep 26, 2023
0.1.0 Sep 21, 2023

#648 in Machine learning

Apache-2.0

12KB
230 lines

bytepiece

Dependencies

~4.5–6.5MB
~121K SLoC