3 releases (breaking)
0.3.0 | Apr 18, 2024 |
---|---|
0.2.0 | Oct 14, 2023 |
0.1.0 | Sep 8, 2023 |
#242 in Database implementations
Used in 3 crates
(via izihawa-tantivy)
7KB
117 lines
#Tokenizer-API
An API to interface a tokenizer with tantivy.
The API will be kept stable in order to not break support for existing tokenizers.
lib.rs
:
Tokenizer are in charge of chopping text into a stream of tokens ready for indexing. This is an seperate crate from tantivy, so implementors don't need to update for each new tantivy version.
To add support for a tokenizer, implement the Tokenizer
trait.
Checkout the tantivy repo for some examples.
Dependencies
~0.4–1MB
~23K SLoC