aleph-alpha-tokenizer

We at Aleph Alpha are big fans of huggingface's tokenizers crate. Kudos for this great library. There is only one downside: The interface is optimized for the bindings, not for working with it from within Rust.

So we took it as an inspiration and tried to improve on some things. First we wanted to see how fast we could make it while implementing the same Model trait. We based our implementation on the very good fst crate. Then we added our own interface to play to Rust's strengths (mainly avoiding needless allocation, re-using data, generics).

We are very happy with the improved performance. In our tests, we found our tokenizer performed mostly linearly with whatever data was thrown at it, while the huggingface wordpiece tokenizer performs quadratically worse with longer multi-token words. The following single-core runtimes in µs were measured for a set of benchmarks:

(This was measured on an Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz running on a Fedora kernel 5.6.15-300.fc32.x86_64 with all mitigations enabled)

As you can see, using our tokenizer as a model is faster than huggingface's wordpiece tokenizer by at least 13%, often more. Using the rustic interface, we can omit a lot of allocation and memory copying, so we are at least 60% faster.

To re-run the benchmark, call cargo bench --all-features. Otherwise only the AlephAlphaTokenizer will be benchmarked.

License

This package is licensed under MIT or Apache License Version 2, at your discretion.