TokenDomain

MMO\core\helpers · 1 symbol · generated 5 Oct 2026

class

TokenDomain

A "grammar": a list of recognized literal tokens (each a plain string, e.g. ',' or '::') and/or nested TokenDomain instances (which the tokenizer can Tokenizer::openDomain into via their own open check). $mask is a fast pre-filter -- the set of first-characters any of this domain's tokens (including nested domains') could possibly start with, used by Tokenizer::next to quickly skip over plain text.

class MMO\core\helpers\TokenDomain

Namespace: MMO\core\helpers

new TokenDomain([',', ':', new BracketsDomain()]);

Index

	mask
	tokens	array<id: string>
	addChar($char)	Register a single-character token (id auto-derived from the character's ord() value).
	addDomains($domains)	Register one or more nested domains (their `open` is checked on every `Tokenizer::next` call while this domain is active).
	addToken($id, $token)	Register a single multi-character token.
	addTokens($tokens)	Register several tokens/domains at once.
	close($tokenizer)	Check whether the tokenizer's current position closes this domain (only ever called while this domain IS the `Tokenizer::currentDomain`).
	debugName()
	getAllowedTokens()
	getMask()
	open($tokenizer)	Check whether the tokenizer's current position starts this (nested) domain -- called on every candidate sub-domain while `Tokenizer::next` looks for the next token.
	removeToken($token)
	setTokens($tokens)	Replace every registered token/domain with $tokens.
	updateMask()	Recompute `getMask` from the current token/domain list.
	__construct($tokens)

Properties2

mask

protected string $mask = ''

Type: string

tokens

array<id: string>

protected array $tokens = []

Type: array

Methods13

addChar($char)

Tokenizer.php:597

function addChar(string $char): void

$char	string
returns	void

addDomains($domains)

Tokenizer.php:587

Register one or more nested domains (their open is checked on every Tokenizer::next call while this domain is active).

function addDomains(MMO\core\helpers\TokenDomain ...$domains): void

...$domains	TokenDomain
returns	void

addToken($id, $token)

Tokenizer.php:576

function addToken(int $id, string $token): void

$id	int
$token	string
returns	void

addTokens($tokens)

Tokenizer.php:632

function addTokens(array $tokens): void

$tokens	array	id => token-string entries, plain token strings (a single-char string's id is auto-derived via ord(); a multi-char string keeps its array key as its id), and/or TokenDomain instances (appended, not keyed)
returns	void
throws		if an entry is neither a string nor a TokenDomain

close($tokenizer)

Tokenizer.php:561

Check whether the tokenizer's current position closes this domain (only ever called while this domain IS the Tokenizer::currentDomain). The base implementation never closes (always null); a domain that's meant to be closable (e.g. quotes, brackets) overrides this.

function close(MMO\core\helpers\Tokenizer $tokenizer): ?MMO\core\helpers\Token

$tokenizer	Tokenizer
returns	null \| Token	the closing token, if this domain should be exited now

debugName()

Tokenizer.php:534

function debugName(): string

returns

string

the class name (override for a more specific description)

getAllowedTokens()

Tokenizer.php:539

function getAllowedTokens(): array

returns array id => (string|TokenDomain), see addTokens

getMask()

Tokenizer.php:566

function getMask(): string

returns

string

every first-character any token/nested-domain in this domain could start with

open($tokenizer)

Tokenizer.php:550

Check whether the tokenizer's current position starts this (nested) domain -- called on every candidate sub-domain while Tokenizer::next looks for the next token. The base implementation never opens (always null); override in a subclass that represents a nested grammar (e.g. quotes, brackets).

function open(MMO\core\helpers\Tokenizer $tokenizer): ?MMO\core\helpers\Token

$tokenizer	Tokenizer
returns	null \| Token	the opening token, if this domain should be entered now

removeToken($token)

Tokenizer.php:606

function removeToken(MMO\core\helpers\TokenDomain|string|int $token): MMO\core\helpers\TokenDomain|string|null

$token	TokenDomain \| string \| int	an id, a token string (found by value), or a nested domain (found by identity)
returns	TokenDomain \| string \| null	the removed entry, or null if not found

setTokens($tokens)

Tokenizer.php:619

Replace every registered token/domain with $tokens.

function setTokens(array $tokens): void

$tokens	array
returns	void

updateMask()

Tokenizer.php:649

Recompute getMask from the current token/domain list. Called automatically by every mutator above; only needs to be called directly if $tokens was somehow changed without going through them.

function updateMask(): void

returns

void

__construct($tokens)

Tokenizer.php:529

function __construct(array $tokens = [])

$tokens = []

array

id => token-string entries, plain token strings (id auto-derived), and/or TokenDomain instances