Results for @tokenizer/token

TypeScript definition for strtok3 token

A promise based streaming tokenizer

Common token types for decoding and encoding numeric and string values

Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

Tiny JavaScript tokenizer.

Tokenized zip support

Tokenize CSS

ProseMirror Markdown integration

Takes an array of GLSL tokens and determines whether or not they're a property of another identifier

Fast token estimation at 96% accuracy of a full tokenizer in a 2kB bundle

Algorithms to help you parse CSS from an array of tokens.

Infer the scope of each token in an array of GLSL tokens

safely inject a block of tokens into a shader

My JavaScript parser

Parses and stringifies CSS selectors

Multilingual tokenizer that automatically tags each token with its type

Trim the whitespace within an array of GLSL tokens

"Descope" an array of GLSL tokens such that they can be safely inlined alongside within another shader without causing any global variable conflicts.

Determine the scope depth of an array of GLSL tokens

Solve CSS math expressions

A tokenzier for Sass' SCSS syntax

Retrieve the values defined with preprocessor statements in a selection of GLSL tokens

A primitive to tokenize your solid-components to enable custom parsing.