1 unstable release
| new 0.1.0 | Oct 16, 2024 |
|---|
#9 in #bpe
138 downloads per month
21KB
496 lines
tokin
Experimental Rust Tokenizer
Dependencies
~6–18MB
~275K SLoC
| new 0.1.0 | Oct 16, 2024 |
|---|
#9 in #bpe
138 downloads per month
21KB
496 lines
Experimental Rust Tokenizer
~6–18MB
~275K SLoC