19 releases (10 breaking)
0.11.2 | Jul 22, 2023 |
---|---|
0.11.1 | Mar 19, 2023 |
0.10.0 | Oct 11, 2022 |
0.8.3 | Jul 30, 2022 |
0.1.3 | Feb 7, 2020 |
#2619 in Parser implementations
7,402 downloads per month
Used in 13 crates
(via sentencepiece)
2MB
25K
SLoC
sentencepiece
This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.
libsentencepiece
dependency
This crate depends on the sentencepiece
C++ library. By default,
this dependency is treated as follows:
- If
sentencepiece
could be found withpkg-config
, the crate will link against the library found throughpkg-config
. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions. - Otherwise, the crate's build script will do a static build of the
sentencepiece
library. This requires thatcmake
is available.
If you wish to override this behavior, the sentencepiece-sys
crate
offers two features:
system
: always attempt to link to thesentencepiece
library found withpkg-config
.static
: always do a static build of thesentencepiece
library and link against that.