token_stream_flatten-rs

Flattening iterator adaptor for token streams in rust. Allows processing rust source as a sequence of primitve tokens.

See the documentation for details.

License

MIT License.

lib.rs:

Flattening iterator adaptor for token streams.

Allows processing rust source as a sequence of primitve tokens.

Usage

The adaptor FlattenRec can be obtained by the flatten_rec method, provided by the trait IntoFlattenRec, implemented for proc-macro2's IntoIter. This is convenient for method chaining.

Alternatively, FlattenRec's From<IntoIter> implementation can be used directly.

#
let stream = r#"if n > 1 { foo[n - 2] } else { 11 }"#
    .parse::<proc_macro2::TokenStream>()
    .unwrap();
let flat = stream.into_iter().flatten_rec().collect::<Vec<_>>();

assert_eq!(
    flat.iter().map(|token| token.to_string()).collect::<Vec<_>>(),
    vec!["if", "n", ">", "1", "{", "foo", "[", "n", "-", "2", "]", "}", "else", "{", "11", "}"],
);

assert_eq!(flat[2].span().start(), proc_macro2::LineColumn { line: 1, column: 5 });
assert_eq!(flat[2].span().end(), proc_macro2::LineColumn { line: 1, column: 6 });
if let Token::Punct(punct) = &flat[2] {
    assert_eq!(punct.as_char(), '>');
} else {
    panic!("expected punct");
}

assert_eq!(flat[10].span().start(), proc_macro2::LineColumn { line: 1, column: 20 });
assert_eq!(flat[10].span().end(), proc_macro2::LineColumn { line: 1, column: 21 });
if let Token::Delimiter(delimiter) = &flat[10] {
    assert_eq!(delimiter.kind(), DelimiterKind::Bracket);
    assert_eq!(delimiter.position(), DelimiterPosition::Close);
} else {
    panic!("expected delimiter");
}

Features

• proc-macro: Enables feature proc-macro of proc-macro2 (enabled by default).
• span-locations: Enables feature span-locations of proc-macro2 (enabled by default).

a bc