19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#2619 in Parser implementations

Download history 1024/week @ 2024-12-08 779/week @ 2024-12-15 436/week @ 2024-12-22 708/week @ 2024-12-29 1396/week @ 2025-01-05 1463/week @ 2025-01-12 1456/week @ 2025-01-19 2703/week @ 2025-01-26 1602/week @ 2025-02-02 1283/week @ 2025-02-09 813/week @ 2025-02-16 1646/week @ 2025-02-23 1056/week @ 2025-03-02 1823/week @ 2025-03-09 2471/week @ 2025-03-16 2013/week @ 2025-03-23

7,402 downloads per month
Used in 13 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

sentencepiece

This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.

libsentencepiece dependency

This crate depends on the sentencepiece C++ library. By default, this dependency is treated as follows:

  • If sentencepiece could be found with pkg-config, the crate will link against the library found through pkg-config. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions.
  • Otherwise, the crate's build script will do a static build of the sentencepiece library. This requires that cmake is available.

If you wish to override this behavior, the sentencepiece-sys crate offers two features:

  • system: always attempt to link to the sentencepiece library found with pkg-config.
  • static: always do a static build of the sentencepiece library and link against that.

No runtime deps