Experimental fast tokenizer
Owned by Mik.
#14 in #bpe
21KB 496 lines
Experimental Rust Tokenizer
~6–19MB ~275K SLoC