7 releases
Uses new Rust 2024
| 0.3.4 | Mar 11, 2025 |
|---|---|
| 0.3.3 | Mar 9, 2025 |
| 0.3.1 | Feb 26, 2025 |
| 0.2.1 | Feb 1, 2025 |
| 0.2.0 | Dec 30, 2024 |
#1042 in Text processing
785 downloads per month
Used in 2 crates
(via ragit)
66KB
1K
SLoC
ragit-korean
Ragit-korean is a very simple korean tokenizer.
Ragit used to use charabia to tokenize cjk documents, but it has too many issues.
- Charabia bundles cjk dictionaries in the binary, which makes the file 70MiB bigger.
- It silently converts 완성형 korean to 조합형 korean. That silently messes up tfidf searches.
Dependencies
~10KB