3 stable releases
new 1.0.2 | Nov 6, 2024 |
---|
#47 in Internationalization (i18n)
216 downloads per month
180KB
2.5K
SLoC
Sentence Segmentation
This is a regex rule-based sentence segmenter written in pure rust. This crate is inspired by the Ruby Pragmatic Segmenter by diasks2 (GitHub Repo).
Features
- Multilingual Support: The crate supports sentence segmentation for 76 languages (see list below).
processor::amharic
processor::afrikaans
processor::albanian
processor::arabic
processor::armenian
processor::azerbaijani
processor::basque
processor::belarusian
processor::bengali
processor::bosnian
processor::bulgarian
processor::catalan
processor::chinese
processor::croatian
processor::czech
processor::danish
processor::dutch
processor::english
processor::esperanto
processor::estonian
processor::finnish
processor::french
processor::ganda
processor::georgian
processor::german
processor::greek
processor::gujarati
processor::hausa
processor::hebrew
processor::hindi
processor::hungarian
processor::icelandic
processor::indonesian
processor::irish
processor::italian
processor::japanese
processor::kazakh
processor::korean
processor::latin
processor::latvian
processor::lithuanian
processor::macedonian
processor::malay
processor::maori
processor::marathi
processor::mongolian
processor::norwegian
processor::persian
processor::polish
processor::portuguese
processor::punjabi_eastern
processor::punjabi_western
processor::romanian
processor::russian
processor::serbian
processor::shona
processor::slovak
processor::slovenian
processor::somali
processor::sotho
processor::spanish
processor::swahili
processor::swedish
processor::tagalog
processor::tamil
processor::telugu
processor::tsonga
processor::tswana
processor::turkish
processor::ukrainian
processor::urdu
processor::vietnamese
processor::welsh
processor::xhosa
processor::yoruba
processor::zulu
- Modular Segmentation: Each language has a dedicated segmentation function, making it simple to use language-specific rules.
- Preprocessing: Through regex, the input text will also be preprocessed and denoised. The result will be a vector of cleaned strings which are ready for the normalization that you planned.
Installation
To use this crate, add it to your Cargo.toml
dependencies:
[dependencies]
sentence_segmentation = "1.0.2"
Usage
use sentence_segmentation::processor;
fn main() {
let text = "이 크레이트는 자연어 처리를 위해서 만들어진 크레이트입니다. 정규표현식 기반의 간단한 알고리즘을 통해 문장 단위로 분리하기에 매우 빠른 속도로 문장을 분리할 수 있습니다. 이 크레이트는 기본적인 전처리도 같이 수행하므로 문장 분리가 목적이 아니더라도 전처리를 위해서 사용할 수도 있습니다.";
let sentences = processor::korean(text);
for sentence in sentences {
println!("{}", sentence);
}
// 이 크레이트는 자연어 처리를 위해서 만들어진 크레이트입니다.
// 정규표현식 기반의 간단한 알고리즘을 통해 문장 단위로 분리하기에 매우 빠른 속도로 문장을 분리할 수 있습니다.
// 이 크레이트는 기본적인 전처리도 같이 수행하므로 문장 분리가 목적이 아니더라도 전처리를 위해서 사용할 수도 있습니다.
}
License
This project is licensed under the MIT License. See the LICENSE file for details.
Dependencies
~3–4.5MB
~82K SLoC