mathy.core.tokenizer

Tokenizer

Tokenizer(self)
The Tokenizer produces a list of tokens from an input string.

eat_token

Tokenizer.eat_token(self, context:mathy.core.tokenizer.TokenContext, typeFn)
Eat all of the tokens of a given type from the front of the stream until a different type is hit, and return the text.

identify_alphas

Tokenizer.identify_alphas(self, context:mathy.core.tokenizer.TokenContext) -> int
Identify and tokenize functions and variables.

identify_constants

Tokenizer.identify_constants(
    self,
    context: mathy.core.tokenizer.TokenContext,
) -> int
Identify and tokenize a constant number.

identify_operators

Tokenizer.identify_operators(
    self,
    context: mathy.core.tokenizer.TokenContext,
) -> bool
Identify and tokenize operators.

is_alpha

Tokenizer.is_alpha(self, c:str) -> bool
Is this character a letter

is_number

Tokenizer.is_number(self, c:str) -> bool
Is this character a number

tokenize

Tokenizer.tokenize(
    self,
    buffer: str,
    terms = False,
) -> List[mathy.core.tokenizer.Token]
Return an array of Tokens from a given string input. This throws an exception if an unknown token type is found in the input.