Lib.rs

› Parser implementations | Programming languages › sentencepiece
#sentence-piece #bindings #tokenizer

sys sentencepiece-sys

by Daniël de Kok and 2 contributors

19 releases (10 breaking)

0.11.2 Jul 22, 2023
0.11.1 Mar 19, 2023
0.10.0 Oct 11, 2022
0.8.3 Jul 30, 2022
0.1.3 Feb 7, 2020

#126 in #tokenizer

Download history 349/week @ 2024-10-23 919/week @ 2024-10-30 451/week @ 2024-11-06 675/week @ 2024-11-13 638/week @ 2024-11-20 525/week @ 2024-11-27 1269/week @ 2024-12-04 907/week @ 2024-12-11 640/week @ 2024-12-18 432/week @ 2024-12-25 891/week @ 2025-01-01 1356/week @ 2025-01-08 1473/week @ 2025-01-15 1602/week @ 2025-01-22 2853/week @ 2025-01-29 1188/week @ 2025-02-05

7,297 downloads per month
Used in 12 crates (via sentencepiece)

Apache-2.0

2MB
25K SLoC

C++ 24K SLoC // 0.1% comments Bitbake 371 SLoC // 0.5% comments Rust 216 SLoC // 0.0% comments Shell 5 SLoC

Binding for the sentencepiece tokenizer

No runtime deps