#unicode-text #unicode #grapheme #word #boundary #text

no-std unicode-segmentation

This crate provides Grapheme Cluster, Word and Sentence boundaries according to Unicode Standard Annex #29 rules

23 releases (stable)

1.12.0 Sep 13, 2024
1.11.0 Feb 7, 2024
1.10.1 Jan 31, 2023
1.10.0 Sep 13, 2022
0.1.1 Jul 9, 2015

#6 in Text processing

Download history 1304923/week @ 2024-08-04 1339636/week @ 2024-08-11 1335129/week @ 2024-08-18 1350615/week @ 2024-08-25 1291476/week @ 2024-09-01 1344321/week @ 2024-09-08 1307490/week @ 2024-09-15 1416270/week @ 2024-09-22 1420897/week @ 2024-09-29 1456263/week @ 2024-10-06 1429135/week @ 2024-10-13 1474625/week @ 2024-10-20 1389320/week @ 2024-10-27 1421420/week @ 2024-11-03 1420965/week @ 2024-11-10 1351924/week @ 2024-11-17

5,664,795 downloads per month
Used in 14,499 crates (617 directly)

MIT/Apache

400KB
5K SLoC

Iterators which split strings on Grapheme Cluster or Word boundaries, according to the Unicode Standard Annex #29 rules.

Build Status

Documentation

use unicode_segmentation::UnicodeSegmentation;

fn main() {
    let s = "a̐éö̲\r\n";
    let g = s.graphemes(true).collect::<Vec<&str>>();
    let b: &[_] = &["", "", "ö̲", "\r\n"];
    assert_eq!(g, b);

    let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?";
    let w = s.unicode_words().collect::<Vec<&str>>();
    let b: &[_] = &["The", "quick", "brown", "fox", "can't", "jump", "32.3", "feet", "right"];
    assert_eq!(w, b);

    let s = "The quick (\"brown\")  fox";
    let w = s.split_word_bounds().collect::<Vec<&str>>();
    let b: &[_] = &["The", " ", "quick", " ", "(", "\"", "brown", "\"", ")", "  ", "fox"];
    assert_eq!(w, b);
}

no_std

unicode-segmentation does not depend on libstd, so it can be used in crates with the #![no_std] attribute.

crates.io

You can use this package in your project by adding the following to your Cargo.toml:

[dependencies]
unicode-segmentation = "1.10.1"

Change Log

1.11.0

  • #124 Update data to Unicode 15.1
  • #128 Add size_hint to iterators

1.10.1

  • #113 Use criterion.rs for word benchmarks
  • #112 Improve table search speed through lookups

1.10.0

  • #107 Upgrade to Unicode 15.0.0
  • #104 Supersedes and fixes #75

1.9.0

  • #101 Upgrade to Unicode 14.0.0

1.8.0

  • #100 * #100 - Increase #[inline] opportunities, resulting in 15-40% performance improvement.
  • #95 Implement debug for Graphemes
  • #94 Add Initial fuzzer for oss-fuzz integration
  • #93 Fix unused imports and deprecated pattern warnings
  • #91 Made local variable immutable by moving it into loop
  • #91 Add new iterator UnicodeWordIndices and unicode_word_indices

1.7.1

  • Update docs on version number

1.7.0

  • #87 Upgrade to Unicode 13
  • #79 Implement a special-case lookup for ascii grapheme categories
  • #77 Optimization for grapheme iteration

1.6.0

  • #72 Upgrade to Unicode 12

1.5.0

  • #68 Upgrade to Unicode 11

1.4.0

  • #56 Upgrade to Unicode 10

1.3.0

  • #24 Add support for sentence boundaries
  • #44 Treat gc=No as a subset of gc=N

1.2.1

  • #37: Fix panic in provide_context.
  • #40: Fix crash in prev_boundary.

1.2.0

  • New GraphemeCursor API allows random access and bidirectional iteration.
  • Fixed incorrect splitting of certain emoji modifier sequences.

1.1.0

  • Add as_str methods to the iterator types.

1.0.3

  • Code cleanup and additional tests.

1.0.1

  • Fix a bug affecting some grapheme clusters containing Prepend characters.

1.0.0

  • Upgrade to Unicode 9.0.0.

No runtime deps