11 releases (6 stable)

1.5.0 May 28, 2024
1.4.0 Nov 16, 2023
1.3.2 Oct 3, 2023
1.2.1 Apr 18, 2023
0.0.1 Apr 29, 2021

#4 in Internationalization (i18n)

Download history 5714/week @ 2024-06-13 5342/week @ 2024-06-20 5271/week @ 2024-06-27 6849/week @ 2024-07-04 7893/week @ 2024-07-11 9849/week @ 2024-07-18 9060/week @ 2024-07-25 10299/week @ 2024-08-01 12641/week @ 2024-08-08 12267/week @ 2024-08-15 10520/week @ 2024-08-22 10139/week @ 2024-08-29 11075/week @ 2024-09-05 10853/week @ 2024-09-12 33300/week @ 2024-09-19 37221/week @ 2024-09-26

94,454 downloads per month
Used in 69 crates (14 directly)

Unicode-3.0

2.5MB
25K SLoC

icu_segmenter crates.io

Segment strings by lines, graphemes, words, and sentences.

This module is published as its own crate (icu_segmenter) and as part of the icu crate. See the latter for more details on the ICU4X project.

This module contains segmenter implementation for the following rules.

Examples

Line Break

Find line break opportunities:

use icu::segmenter::LineSegmenter;

let segmenter = LineSegmenter::new_auto();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(&breakpoints, &[0, 6, 13, 17, 23, 29, 36]);

See LineSegmenter for more examples.

Grapheme Cluster Break

Find all grapheme cluster boundaries:

use icu::segmenter::GraphemeClusterSegmenter;

let segmenter = GraphemeClusterSegmenter::new();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(
    &breakpoints,
    &[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
        19, 21, 22, 23, 24, 25, 28, 29, 30, 31, 34, 35, 36
    ]
);

See GraphemeClusterSegmenter for more examples.

Word Break

Find all word boundaries:

use icu::segmenter::WordSegmenter;

let segmenter = WordSegmenter::new_auto();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(
    &breakpoints,
    &[0, 5, 6, 11, 12, 13, 16, 17, 22, 23, 28, 29, 35, 36]
);

See WordSegmenter for more examples.

Sentence Break

Segment the string into sentences:

use icu::segmenter::SentenceSegmenter;

let segmenter = SentenceSegmenter::new();

let breakpoints: Vec<usize> = segmenter
    .segment_str("Hello World. Xin chào thế giới!")
    .collect();
assert_eq!(&breakpoints, &[0, 13, 36]);

See SentenceSegmenter for more examples.

More Information

For more information on development, authorship, contributing etc. please visit ICU4X home page.

Dependencies