48 releases
0.14.2 | Sep 18, 2024 |
---|---|
0.14.0 | Feb 7, 2024 |
0.13.0 | Apr 10, 2023 |
0.12.1 | Jun 8, 2022 |
0.7.3 | Nov 30, 2018 |
#3 in Parser tooling
490,838 downloads per month
Used in 759 crates
(263 directly)
1.5MB
693 lines
Logos
Create ridiculously fast Lexers.
Logos has two goals:
- To make it easy to create a Lexer, so you can focus on more complex problems.
- To make the generated Lexer faster than anything you'd write by hand.
To achieve those, Logos:
- Combines all token definitions into a single deterministic state machine.
- Optimizes branches into lookup tables or jump tables.
- Prevents backtracking inside token definitions.
- Unwinds loops, and batches reads to minimize bounds checking.
- Does all of that heavy lifting at compile time.
Example
use logos::Logos;
#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")] // Ignore this regex pattern between tokens
enum Token {
// Tokens can be literal strings, of any length.
#[token("fast")]
Fast,
#[token(".")]
Period,
// Or regular expressions.
#[regex("[a-zA-Z]+")]
Text,
}
fn main() {
let mut lex = Token::lexer("Create ridiculously fast Lexers.");
assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 0..6);
assert_eq!(lex.slice(), "Create");
assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 7..19);
assert_eq!(lex.slice(), "ridiculously");
assert_eq!(lex.next(), Some(Ok(Token::Fast)));
assert_eq!(lex.span(), 20..24);
assert_eq!(lex.slice(), "fast");
assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.slice(), "Lexers");
assert_eq!(lex.span(), 25..31);
assert_eq!(lex.next(), Some(Ok(Token::Period)));
assert_eq!(lex.span(), 31..32);
assert_eq!(lex.slice(), ".");
assert_eq!(lex.next(), None);
}
For more examples and documentation, please refer to the Logos handbook or the crate documentation.
How fast?
Ridiculously fast!
test identifiers ... bench: 647 ns/iter (+/- 27) = 1204 MB/s
test keywords_operators_and_punctators ... bench: 2,054 ns/iter (+/- 78) = 1037 MB/s
test strings ... bench: 553 ns/iter (+/- 34) = 1575 MB/s
Acknowledgements
- Pedrors for the Logos logo.
Thank you
Logos is very much a labor of love. If you find it useful, consider getting me some coffee. ☕
If you'd like to contribute to Logos, then consider reading the Contributing guide.
Contributing
Logos welcome any kind of contribution: bug reports, suggestions, or new features!
Please use the issues or pull requests tabs, when appropriate.
Releasing a new version
[!NOTE]
This section is only useful to Logos' maintainers.
First, make sure you are logged-in https://crates.io with: cargo login
.
If you don't have write access to Logos' crates, you can still
perform steps 1-4, and ask a maintainer with accesses to perform step 5.
This project uses cargo-release
to publish all packages with more ease.
Note that, by default, every command runs in dry mode, and you need to append --execute
to actually perform the action.
Here are the following steps to release a new version:
- create a branch
release-x.y.z
from themaster
branch; - run and commit
cargo release version --workspace <LEVEL>
; - run and commit
cargo release replace --workspace
; - push your branch and create a pull request;
- and, once your branch was merged to
master
, run the following:cargo release publish --package logos-codegen cargo release publish --package logos-derive cargo release publish --package logos-cli cargo release publish --package logos
And voilà!
License
This code is distributed under the terms of both the MIT license and the Apache License (Version 2.0), choose whatever works for you.
See LICENSE-APACHE and LICENSE-MIT for details.