Lexer module consumes text input and produces linear stream of tokens.

Tokens are defined in token

Javascript Lexer

Javscript lexer implements high performance lexer of javscript as defined by ECMAScript 9

Output of the lexer together with description can be found in docs

Installation

add to cargo.toml

[dependencies]
javascript_lexer = "0.1"

Example

Lexer is invoked by running

extern crate javascript_lexer;

use javascript_lexer::Lexer;

fn main() {
    Lexer::parse_tokens("javascript");
}