19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#2685 in Parser implementations

Download history 432/week @ 2024-12-25 891/week @ 2025-01-01 1356/week @ 2025-01-08 1473/week @ 2025-01-15 1602/week @ 2025-01-22 2853/week @ 2025-01-29 1493/week @ 2025-02-05 1002/week @ 2025-02-12 1249/week @ 2025-02-19 1272/week @ 2025-02-26 1080/week @ 2025-03-05 2657/week @ 2025-03-12 2552/week @ 2025-03-19 1499/week @ 2025-03-26 1965/week @ 2025-04-02 3182/week @ 2025-04-09

9,611 downloads per month
Used in 13 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

sentencepiece

This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.

libsentencepiece dependency

This crate depends on the sentencepiece C++ library. By default, this dependency is treated as follows:

  • If sentencepiece could be found with pkg-config, the crate will link against the library found through pkg-config. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions.
  • Otherwise, the crate's build script will do a static build of the sentencepiece library. This requires that cmake is available.

If you wish to override this behavior, the sentencepiece-sys crate offers two features:

  • system: always attempt to link to the sentencepiece library found with pkg-config.
  • static: always do a static build of the sentencepiece library and link against that.

No runtime deps