1 unstable release
new 0.1.3 | Feb 14, 2025 |
---|---|
0.1.2 |
|
0.1.1 |
|
0.1.0 |
|
#213 in Compression
284 downloads per month
105KB
2K
SLoC
Vq
Vq (v[ector] q[uantizer]) is a Rust library that implements several popular vector quantization algorithms including binary, scalar, and product quantization algorithms. It provides a simple, efficient API for data compression that helps reduce memory usage and computational overhead.
Features
-
Implemented Algorithms:
-
Parallelized vector operations for large vectors using Rayon.
-
Flexible quantization algorithm implementations that support using various distance metrics such as Euclidean, Cosine, Manhattan distances.
-
Support for quantizing vectors of
f32
tof16
(using half) oru8
data types. -
Simple, intuitive, and uniform API for all quantization algorithms.
Installation
cargo add vq
Vq requires Rust 1.83 or later.
Documentation
Find the latest documentation on docs.rs.
Check out vq_examples.rs the tests directory for detailed examples of using Vq.
Quick Example
Here's a simple example using the SQ algorithm to quantize a vector:
use vq::sq::ScalarQuantizer;
use vq::vector::Vector;
fn main() {
// Create a scalar quantizer for values in the range [0.0, 1.0] with 256 levels.
let quantizer = ScalarQuantizer::fit(0.0, 1.0, 256);
// Create an input vector.
let input = Vector::new(vec![0.1, 0.5, -0.8, -0.3, 0.9]);
// Quantize the input vector.
let quantized_input = quantizer.quantize(&input);
println!("Quantized input vector: {}", quantized_input);
}
Performance
Check out the notebooks directory for information on how to evaluate the performance of the implemented algorithms. Additionally, see the content of src/bin directory for the scripts used for the evaluation.
On a ThinkPad T14 laptop with an Intel i7-1355U CPU and 32GB of RAM, the performance of the PQ algorithm for quantizing one million vectors of 128 dimensions (into 16 subspaces with 256 centroids per subspace) is as follows:
- Training Time: 232.5 seconds
- Quantization Time: 34.1 seconds
- Reconstruction Error: 0.02
- Recall@10: 0.19
Contributing
Contributions are welcome! Please see CONTRIBUTING.md for details on contributing.
License
Vq is available under the terms of either of the following licenses:
- MIT License (LICENSE-MIT)
- Apache License, Version 2.0 (LICENSE-APACHE)
Dependencies
~9.5MB
~178K SLoC