Lib.rs

› Text processing
#tokenizer #tokenize #text-tokenizer

text-tokenizer

Custom text tokenizer

by merl-twin

12 releases

0.5.10 Nov 22, 2024
0.5.9 Nov 20, 2024
0.5.6 Sep 6, 2024
0.5.3 Aug 7, 2024
0.4.1 Dec 20, 2023

#460 in Text processing

Download history 460/week @ 2024-11-16 95/week @ 2024-11-23 32/week @ 2024-11-30 11/week @ 2024-12-07 2/week @ 2024-12-14

893 downloads per month

MIT license

490KB
10K SLoC

Custom text tokenizer

Dependencies

~1MB
~18K SLoC