SmolToken

SmolToken is a fast library for tokenizing text using the Byte Pair Encoding (BPE) algorithm. Inspired by OpenAI's tiktoken, SmolToken is designed to fill a critical gap by enabling BPE training from scratch while maintaining high performance for encoding and decoding tasks.

Unlike tiktoken, SmolToken supports training tokenizers on custom data. Up to ~4x faster than the port of unoptimized educational implementation _educational.py in rust.

Benchmark Results

SmolToken is already faster than baseline educational implementation of BPE training:

Tested on:

• Vocabulary size: 500
• Dataset: Tiny Stories (~18 MB)

Installation

Add smoltoken to your Rust project via crates.io:

cargo add smoltoken

Or add smoltoken to your Python project via PyPI:

pip install smoltoken

Roadmap

• Concurrency: Add multi-threading support using rayon for faster training, encoding, and decoding.
• Python Bindings: Integrate with Python using PyO3 to make it accessible for Python developers.
• Serialization: Add serialization support to save/load trained tokenizer vocabulary.

Contributing

We very much welcome contributions to make Smoltoken fast, robust and efficient. Make a fork, create a feature branch if needed and sumbit your pull request. Since, the library itself is in its early release stage, I also expect to get community feedback to improve on. Just raise an issue here and we will fix them promptly.

License

SmolToken is open source and licensed under the MIT License.

Acknowledgements

Special thanks to OpenAI's tiktoken for inspiration and foundational ideas.