#sentence #vector #word #word2vec #file #partition #data

bin+lib sentence2vec

A tool to convert a sentence to a vector. It can be used to partition word2vec data. It can also be used to extract a list of words from a word2vec data file.

1 unstable release

0.1.0 Jul 1, 2023

#507 in Machine learning

MIT license

6.5MB
355 lines

Sentence2Vec

This computes the sentence vector of a sentence using word2vec for each word in the sentence.
The crate supports the txt format of word2vec and a custom format of word2vec for binary files.

Usage

use sentence2vec::Sentence2Vec;

let sentence = "This is a sentence.";
let model = Sentence2Vec::new("path/to/model.bin").unwrap();
let vector = model.sentence_vector(sentence);

The sentence vector is a sum up of the word vectors of the sentence. We then have a vector of the same dimension as the word vectors.

Example

Here is an example to compute the cosine similarity between two sentences.

use sentence2vec::Sentence2Vec;

let word2vec = Word2Vec::<300>::load_from_bytes("tests/common.en.300.bin").await.unwrap();
let sentence2vec = Sentence2Vec::new(Box::new(word2vec));
let similarity = sentence2vec.cosine("This is a test.", "This is a test.").await.unwrap();
assert_eq!(similarity, 1.0);

let similarity_fruits = sentence2vec.cosine("I love apples.", "I love pears").await.unwrap();

let similarity_fuits_cars = sentence2vec.cosine("I love apples.", "I love cars").await.unwrap();

assert!(similarity_fruits > similarity_fuits_cars);

License

MIT

Dependencies

~2.3–4.5MB
~78K SLoC