#tantivy #bridge #adapter #tokenizer #jieba #jieba-rs

tantivy-jieba

A library that bridges between tantivy and jieba-rs

12 releases (breaking)

0.11.0 Apr 28, 2024
0.10.0 Oct 13, 2023
0.9.0 Jul 3, 2023
0.7.0 Jan 5, 2023
0.1.1 Feb 13, 2019

#184 in Text processing

Download history 2681/week @ 2024-07-21 2467/week @ 2024-07-28 2631/week @ 2024-08-04 2263/week @ 2024-08-11 2095/week @ 2024-08-18 1467/week @ 2024-08-25 3293/week @ 2024-09-01 3283/week @ 2024-09-08 1539/week @ 2024-09-15 2091/week @ 2024-09-22 765/week @ 2024-09-29 1656/week @ 2024-10-06 2280/week @ 2024-10-13 2730/week @ 2024-10-20 3329/week @ 2024-10-27 3454/week @ 2024-11-03

12,109 downloads per month

MIT license

8KB
104 lines

tantivy-jieba

Crates.io version docs.rs Changelog FOSSA Status

An adapter that bridges between tantivy and jieba-rs.

Usage

Add dependency tantivy-jieba to your Cargo.toml.

Example

use tantivy::tokenizer::*;
let mut tokenizer = tantivy_jieba::JiebaTokenizer {};
let mut token_stream = tokenizer.token_stream("测试");
assert_eq!(token_stream.next().unwrap().text, "测试");
assert!(token_stream.next().is_none());

Register tantivy tokenizer

use tantivy::schema::Schema;
use tantivy::tokenizer::*;
use tantivy::Index;
let tokenizer = tantivy_jieba::JiebaTokenizer {};
let index = Index::create_in_ram(schema);
index.tokenizers()
     .register("jieba", tokenizer);

License

FOSSA Status

Dependencies

~9.5MB
~97K SLoC