lexer
The JavaScript lexer: `tokenize(src)` turns source text into a flat token array ending in an eof token, recursively lexing template expression bodies and disambiguating regex from division by the preceding token.
lexer
The JavaScript lexer: tokenize(src) turns source text into a flat token array ending in an eof token, recursively lexing template expression bodies and disambiguating regex from division by the preceding token.
Interface
What this asset declares: the schema it conforms to, what it exposes, and the rendered structured payload.
conforms to
zero/source-extract/v2strict
isDigit(c: string) → boolean
| arg | type | description |
|---|---|---|
| c | string |
isHexDigit(c: string) → boolean
| arg | type | description |
|---|---|---|
| c | string |
isIdentStart(c: string) → boolean
| arg | type | description |
|---|---|---|
| c | string |
isIdentPart(c: string) → boolean
| arg | type | description |
|---|---|---|
| c | string |
newLexer(src: string, sourceName: string) → Lexer
| arg | type | description |
|---|---|---|
| src | string | |
| sourceName | string |
peek(self: Lexer, ahead: number?) → string
| arg | type | description |
|---|---|---|
| self | Lexer | |
| ahead | number? |
advance(self: Lexer) → string
Advance one byte, tracking line/col. Line terminators latch nlBefore.
| arg | type | description |
|---|---|---|
| self | Lexer |
atEnd(self: Lexer) → boolean
| arg | type | description |
|---|---|---|
| self | Lexer |
raise(self: Lexer, msg: string, line: number, col: number) → never
| arg | type | description |
|---|---|---|
| self | Lexer | |
| msg | string | |
| line | number | |
| col | number |
prevSignificant(self: Lexer) → Token
| arg | type | description |
|---|---|---|
| self | Lexer |
regexAllowed(self: Lexer) → boolean
A `/` starts a regex when the previous token cannot end an expression.
| arg | type | description |
|---|---|---|
| self | Lexer |
skipTrivia(self: Lexer) → void
Consume whitespace and comments, latching nlBefore. Block comments may span lines; an unterminated block comment is an error.
| arg | type | description |
|---|---|---|
| self | Lexer |
readRadixDigits(self: Lexer, isValid: (string) → void
Read the digit run for a given radix, allowing separators between digits.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| isValid | (string |
readDecimalDigits(self: Lexer, out: { string }) → void
Read a decimal digit run (with separators) into the given buffer. A separator is valid only between two digits, so each `_` requires a digit both before and after it.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| out | { string } |
readNumber(self: Lexer) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer |
readEscape(self: Lexer, startLine: number, startCol: number) → string
Consume one escape sequence (backslash already consumed) and return its cooked text. Line continuation yields the empty string.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| startLine | number | |
| startCol | number |
readUnicodeCodepoint(self: Lexer, startLine: number, startCol: number) → number
Read one `\u` escape body (the leading backslash and `u` already consumed) and return its numeric code point, honouring both the `\u{...}` codepoint form and the fixed four-hex-digit form.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| startLine | number | |
| startCol | number |
readString(self: Lexer, quote: string) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer | |
| quote | string |
readTemplateExprSource(self: Lexer, startLine: number, startCol: number) → string
Capture the raw source of a brace-balanced `${...}` body, honouring nested braces, strings, templates, comments and regex literals so their inner characters (braces, quotes) do not derail the capture.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| startLine | number | |
| startCol | number |
captureRegexRaw(self: Lexer, startLine: number, startCol: number) → string
Copy a regex literal verbatim (class- and escape-aware) for expression capture.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| startLine | number | |
| startCol | number |
captureQuoted(self: Lexer, quote: string, startLine: number, startCol: number) → string
Copy a quoted string verbatim (including its escapes) for expression capture.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| quote | string | |
| startLine | number | |
| startCol | number |
captureTemplateRaw(self: Lexer, startLine: number, startCol: number) → string
Copy a nested template verbatim so its `${}` braces don't unbalance the outer capture.
| arg | type | description |
|---|---|---|
| self | Lexer | |
| startLine | number | |
| startCol | number |
readTemplate(self: Lexer) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer |
readRegex(self: Lexer) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer |
readIdentifier(self: Lexer) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer |
readPunct(self: Lexer) → RawToken
| arg | type | description |
|---|---|---|
| self | Lexer |
next(self: Lexer) → Token
| arg | type | description |
|---|---|---|
| self | Lexer |
tokenize(src: string, sourceName: string?) → void
| arg | type | description |
|---|---|---|
| src | string | |
| sourceName | string? |
Token = {Lexer = {Sub-parts
Everything contained inside this part. Assets are composite children (clickable cards). Files are leaf payloads. Expand any row to view its source.
Showing up to 3 levels deep. Open a part to see what it holds further down.
Problems
Everything affecting this asset right now: its own problems, anything wrong inside it, and problems on its direct dependencies.
agent_score is exposed.+ quality × 0.35
+ performance × 0.25
± compat factor
Usability ratings
Did the part work as advertised when consumers tried to drop it in. Separate from upvotes: those are taste; this is "did it function".
Scoped to this part · feeds back into the world's score.