Cargo Features
ChonkieR has no features set by default.
[dependencies]
chonkier = { version = "0.0.2", features = ["all", "json", "tokenizers", "tiktoken"] }- all = json, tiktoken, tokenizers
- json all?
-
Enables serde_json
- tokenizers all?
-
Enables tokenizers
Affects
tokenizer::hftokenizers… - tiktoken all?
-
Enables tiktoken-rs