lexer

The JavaScript lexer: tokenize(src) turns source text into a flat token array ending in an eof token, recursively lexing template expression bodies and disambiguating regex from division by the preceding token.