1 unstable release
0.1.0 | Jan 2, 2025 |
---|
#1192 in Machine learning
530 downloads per month
16KB
438 lines
Tocken
Tokenizer implemented in Rust.
This tokenizer is based on Lucene's EnglishAnalyzer.
Usage
Dependencies
~9.5MB
~168K SLoC
0.1.0 | Jan 2, 2025 |
---|
#1192 in Machine learning
530 downloads per month
16KB
438 lines
Tokenizer implemented in Rust.
This tokenizer is based on Lucene's EnglishAnalyzer.
~9.5MB
~168K SLoC