Lexer

MMO\doc\js · 1 symbol · generated 30 Sep 2026

class

Lexer

A small, tolerant javascript lexer -- just enough to let Parser walk a source file without ever mistaking a brace, a quote or a slash inside a string/comment/regular expression for real code.

class MMO\doc\js\Lexer

Namespace: MMO\doc\js

It is deliberately **not** a full ECMAScript tokenizer: it does not build an AST, does not validate anything, and never throws on malformed input (an unterminated string simply runs to the end of the file). What it guarantees is the one property a documentation extractor actually needs: every `{`, `}`, `(`, `)`, `;` and identifier it reports is really code, so brace-depth tracking is reliable. ```php foreach (Lexer::Tokenize($source) as $token) { if ($token['type'] === Lexer::T_DOC) {echo $token['value'];} } ``` A token is `['type' => Lexer::T_*, 'value' => string, 'pos' => int, 'end' => int, 'line' => int]`.

Index

	OPERATORS
	REGEX_KEYWORDS
	T_COMMENT
	T_DOC
	T_NAME
	T_NUMBER
	T_PUNCT
	T_REGEX
	T_STRING
	T_TEMPLATE
	RegexAllowed($previous) static	Whether a `/` appearing right after $previous opens a regular expression.
	SkipQuoted($source, $i, $quote) static
	SkipRegex($source, $i) static	Skip a regular expression literal's body and flags.
	SkipTemplate($source, $i) static	Skip a template literal's body, including any `${ ...
	Tokenize($source) static	Split a javascript source into tokens.

Constants10

OPERATORS

const OPERATORS = […33 items]

REGEX_KEYWORDS

const REGEX_KEYWORDS = […14 items]

T_COMMENT

const T_COMMENT = 'comment'

T_DOC

const T_DOC = 'doc'

T_NAME

const T_NAME = 'name'

T_NUMBER

const T_NUMBER = 'number'

T_PUNCT

const T_PUNCT = 'punct'

T_REGEX

const T_REGEX = 'regex'

T_STRING

const T_STRING = 'string'

T_TEMPLATE

const T_TEMPLATE = 'template'

Methods5

RegexAllowed($previous)

staticLexer.php:160

Whether a / appearing right after $previous opens a regular expression. Everything that can *end* an expression (a name, a number, a string, a closing )/]) means the / divides it; anything else means a new expression is starting, so the / opens a literal.

protected static function RegexAllowed(?array $previous): bool

$previous	null \| array	the last non-comment token, null at the start of the file
returns	bool

SkipQuoted($source, $i, $quote)

staticLexer.php:172

protected static function SkipQuoted(string $source, int $i, string $quote): int

$source	string
$i	int
$quote	string
returns	int	the offset just past the closing $quote (or the end of the source).

SkipRegex($source, $i)

staticLexer.php:219

Skip a regular expression literal's body and flags. A / inside a [...] character class does not close it -- which is exactly what makes a regex-unaware scanner desynchronise on real code.

protected static function SkipRegex(string $source, int $i): ?int

$source	string
$i	int
returns	null \| int	the offset just past the flags, or null when this is not a regex after all (no closing `/` before the end of the line, e.g. a stray division the heuristic mis-guessed)

SkipTemplate($source, $i)

staticLexer.php:188

Skip a template literal's body, including any ${ ... } substitution -- which can itself contain strings, braces and further template literals, so the braces are counted and nested literals recursed into.

protected static function SkipTemplate(string $source, int $i): int

$source	string
$i	int
returns	int	the offset just past the closing backtick.

Tokenize($source)

staticLexer.php:56

Split a javascript source into tokens.

static function Tokenize(string $source): array

$source	string
returns	array	string, value: string, pos: int, end: int, line: int}>