Generic lexer
A generic lexer base class
Generic lexer
Lexes ES modules returning their import/export metadata
Lexes CommonJS modules, returning their named exports metadata
The pug lexer (takes a string and converts it to an array of tokens)
lexing json
A tool set for CSS: fast detailed parser (CSS → AST), walker (AST traversal), generator (AST → CSS) and lexer (validation and matching) based on specs and browser implementations
Core components of Traqula
@ts-jison/common: ==== ----------------------------------------- A parser/lexer run-time library used by [@ts-jison/parser](../parser) and [@ts-jison/lexer](../lexer).
@ts-jison/lexer: ==== ----------------------------------------- A parser run-time library used by [@ts-jison/parser-generator](../parser-generator).
@ts-jison/parser: ==== A lightly-typescriptified API for creating parsers in JavaScript ----------------------------------------- A lexical analyzer generator used by [@ts-jison/lexer-generator](../lexer-generator).
Does the lexing (aka tokenizing) step of parsing source code.
MermaidJS parser
ES module parser powered by Rust
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
This is the Linux 64-bit binary for rs-module-lexer.
Edge parser to convert text markup to lexer tokens
This is the Linux 64-bit (musl) binary for rs-module-lexer.
XML Lexer
> A lossless Twig lexer
The lexer for Materialize's SQL dialect, with wasm build targets.
Does the lexing (tokenizing) step, used for parsing source text.
This is the Linux ARM 64-bit (musl) binary for rs-module-lexer.
Oedipus Lex is a lexer generator in the same family as Rexical and Rex. Oedipus Lex is my independent lexer fork of Rexical. Rexical was in turn a fork of Rex. We've been unable to contact the author of rex in order to take it over, fix it up, extend it, and relicense it to MIT. So, Oedipus was written clean-room in order to bypass licensing constraints (and because bootstrapping is fun). Oedipus brings a lot of extras to the table and at this point is only historically related to rexical. The syntax has changed enough that any rexical lexer will have to be tweaked to work inside of oedipus. At the very least, you need to add slashes to all your regexps. Oedipus, like rexical, is based primarily on generating code much like you would a hand-written lexer. It is _not_ a table or hash driven lexer. It uses StrScanner within a multi-level case statement. As such, Oedipus matches on the _first_ match, not the longest (like lex and its ilk). This documentation is not meant to bypass any prerequisite knowledge on lexing or parsing. If you'd like to study the subject in further detail, please try [TIN321] or the [LLVM Tutorial] or some other good resource for CS learning. Books... books are good. I like books.
noratext is a simple (and rather stupid) lexer and parser generator for loosly markuped text.
This gem implements a runtime for ANTLR4 in Ruby for lexers and parsers generated using the Ruby language target.
Full power of Java's ANTLR 3 language recognition tool brought to Ruby. Generate lexers, parsers, tree parsers, source translators, and language analysis tools for use in Ruby applications.
Generate all parts of a compiler, including the lexer, parser, syntax tree, symbol table, type checker, code generator, and optimizer.
Walrat is a Parsing Expression Grammar (PEG) parser generator that creates integrated lexers, "packrat" parsers, and Abstract Syntax Tree (AST) builders.
Rockit is a potent parser generator and gives you AST's (Abstract Syntax Tree's) which you can pattern match and pretty-print. Rockit does not distinghuish between lexing and parsing so the generated parsers are scanner-/lexer-less. The vision is to extend Rockit with more advanced compiler-related abilities including back-ends and code generation. However, currently the focus is on parsing and AST-related tasks such as transformation.