3 releases (breaking)

0.3.0 Apr 18, 2024
0.2.0 Oct 14, 2023
0.1.0 Sep 8, 2023

#242 in Database implementations


Used in 3 crates (via izihawa-tantivy)

MIT license

7KB
117 lines

#Tokenizer-API

An API to interface a tokenizer with tantivy.

The API will be kept stable in order to not break support for existing tokenizers.


lib.rs:

Tokenizer are in charge of chopping text into a stream of tokens ready for indexing. This is an seperate crate from tantivy, so implementors don't need to update for each new tantivy version.

To add support for a tokenizer, implement the Tokenizer trait. Checkout the tantivy repo for some examples.

Dependencies

~0.4–1MB
~23K SLoC