Active direct dependers over time: total, gained/lost
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
1
1
1
2
2
4
4
4
4
4
4
4
5
5
5
5
5
6
6
7
7
10
11
12
13
13
14
15
31
32
31
38
44
43
42
52
49
53
51
48
51
54
57
57
59
2020 2021 2022 2023 2024 2025

Tokenizers is used at run time in 142 crates (of which 19 optionally). It is a direct run-time dependency in 93 crates. It's used only as a dev dependency in 5 crates (of which 4 directly).

Number of dependers Tokenizers version Downloads/month
69 0.21.1 99K
11 0.20.4 37K
22 0.19.1 35K
22 0.15.2 54K
9 0.14.1 4.5K
9 0.13.4 6.6K
4 0.11.3 47
1 0.10.1 40
Depender (with downloads and own rev deps) Tokenizers version
38K 15 text-splitter optional ^0.21
7.6K 1 toktrie_hf_tokenizers >=0.20.0, <1.0.0
6.2K 14 fastembed ^0.21
4.9K 19 rust-bert optional ^0.20
900 glowrs ^0.19.1
850 vectordb-cli ^0.21.1
800 sbv2_core ^0.20.0
800 4 candle-examples ^0.21.0
700 4 candle-datasets ^0.21.0
2.6K 5 lancedb optional ^0.19.1
700 gline-rs ^0.21.0
600 aleph-alpha-client ^0.21.0
600 2 gibberish-or-not ^0.14.1
490 gpt_sovits_rs ^0.20.0
440 rai-models ^0.15.0
430 r2md ^0.21.0
410 2 claude-tokenizer ^0.20.0
400 outlines-core =0.21.1
320 wimbd ^0.15.1
280 mpnet-rs ^0.15.1
260 simple-whisper ^0.19.1
250 ug-llama ^0.20.3
240 1 rbert ^0.21.0
230 embed_anything ^0.21.1
220 candle_embed ^0.19.1
220 1 kalosm-llama ^0.21.0
180 1 rwhisper ^0.21.0
180 ct2rs ^0.21.1
170 1 stable-diffusion ^0.15.0
140 moshi-backend ^0.15.2
130 1 kalosm-ocr ^0.21.0
120 gliclass-rs ^0.21.0
120 lsp-ai ^0.14.1
120 1 rwuerstchen ^0.21.0
110 usls ^0.15.2
100 gte-rs ^0.21.0
smartgpt ^0.13.3
4 alith-models ^0.21.1
rusty-dawg ^0.20.0
valentinus >=0.13.4
ragtime ^0.19
sbert ^0.11
bge ^0.15.2
oxidized-transformers ^0.15
callm ^0.19
reco-forge ^0.15.0
edge-transformers ^0.13
rphi ^0.19.1
promptbox ^0.15.0
fashion-clip-rs ^0.15
modeling ^0.11.0
1 wonnx-preprocessing ^0.13.3
enfer_core ^0.15.0
1 alith-inference ^0.21.1
3 llm_models ^0.21.0
glimpse ^0.21.0
tkn-cli ^0.21.1
dataflow_nlp ^0.11
aiochatgpt ^0.13.2
2 diffusion_rs_common ^0.21.0
musicgpt ^0.19.1
moz_genai ^0.15.0
1 clip_txt ^0.13.3
rsllm ^0.19.1
bm42 ^0.19
bundle_repo ^0.21
token-counter ^0.19.1
spoilers ^0
llms-from-scratch-rs ^0.21.0
2 diffuse_rs_common ^0.21.0
1 diffusion_rs_core ^0.21.0
coze ^0.15.2
4 sapiens ^0.14.1
cracken ~0.11.0
aleph-alpha-api ^0.15.0
anonymize_rs ^0.13.3
bridge-common ^0.19.1
candle-chat-types ^0.13.4
codegeex4-candle ^0.19.1
1 diffuse_rs_core ^0.21.0
heapswap ^0.15.1
inference_core ^0.15.0
prefrontal ^0.15
rmistral ^0.13.4
yolors ^0.15.0
260 code-splitter optional ^0.19.1
200 chonkier optional ^0.21.1
200 1 natural-tts optional ^0.21.0
180 norma optional ^0.20.0
110 1 scirs2-text optional ^0.15
aleph-alpha-tokenizer optional ^0.10.1
2 beet_ml optional ^0.19.1
sentinellm_ai optional ^0.20
348K 60 ort dev >=0.13.4
160 3 general-sam dev ^0.21.1
120 mtc-token-healing dev ^0.21.1
nano-vectordb-rs dev ^0.21.0