Class StringTokenizer<R>`Abstract`

Type Parameters

Hierarchy (view full)

Tokenizer<string, R>
- StringTokenizer

Index

Constructors

constructor

new StringTokenizer<R>(raw, path): StringTokenizer<R>
Type Parameters
- R
Parameters
- raw: string
- path: string
Returns StringTokenizer<R>
Overrides Tokenizer.constructor
- Defined in src/string-tokenizer.ts:8

Properties

length

length: number

Set to the length of the input values (vals) upon instantiation.

path

path: string

raw

raw: string

tokens

tokens: R[] = ...

An array of tokens productd by tokenize().

Usage Notes

Tokens are not pushed automatically into tokens within the body of tokenize(). You must do that manually.

vals

vals: string[]

The input values to be tokenized. This array is iterated over in onNextToken to generate tokens.

Accessors

position

get position(): number
Returns number
the zero-based position of the token at the top of vals.
Overrides Tokenizer.position
- Defined in src/string-tokenizer.ts:77

Methods

consume

consume(val?): {
until: ((predicate: ((val: string) => boolean)) => string[]);
while: ((predicate: ((val: string) => boolean)) => string[]);
}
Consumes a sequence of values from vals.
Parameters
- Optionalval: string
Returns {
until: ((predicate: ((val: string) => boolean)) => string[]);
while: ((predicate: ((val: string) => boolean)) => string[]);
}
An object that allows for the consumption of values, providing methods until and while. These methods determine when to stop or continue the consumption of the sequence.
- until: ((predicate: ((val: string) => boolean)) => string[])
  - - (predicate): string[]
    - Parameters
      predicate: ((val: string) => boolean)
      (val): boolean
      Parameters
      val: string
      Returns boolean
      Returns string[]
- while: ((predicate: ((val: string) => boolean)) => string[])
  - - (predicate): string[]
    - Parameters
      predicate: ((val: string) => boolean)
      (val): boolean
      Parameters
      val: string
      Returns boolean
      Returns string[]
Inherited from Tokenizer.consume
- Defined in src/tokenizer.ts:78

consumeNumber

consumeNumber(firstChar): string
Consumes chars until a non-numeric-character is encountered. If your use-case requires numbers to have non-numeric characters, you can register a new number regex with registerNumberRegex.
Parameters
- firstChar: string
Returns string
- Defined in src/string-tokenizer.ts:60

consumeWhitespace

consumeWhitespace(): string
Consumes vars until a non-whitespace token is shifted from vals, returning the unshifted non-whitespace char.

Returns string
- Defined in src/string-tokenizer.ts:20

consumeWord

consumeWord(firstChar): string[]
Consumes chars until a non-word-character is encountered. If your use-case requires words to have non-alphabetical characters, you can register a new word regex with registerWordRegex.
Parameters
- firstChar: string
Returns string[]
- Defined in src/string-tokenizer.ts:41

`Abstract`onNextToken

onNextToken(val): void
Abstract method to process the next token.

This method must be implemented by subclasses of Tokenizer. It defines how each value in vals should be converted into a token.
Parameters
- val: string
  The next value to process into a token.
Returns void
Inherited from Tokenizer.onNextToken
- Defined in src/tokenizer.ts:67

peek

peek(lookahead?): undefined | string
This method returns the next value in the sequence to be processed. If no more values are available, will throw an error.
Parameters
- lookahead: number = 0
Returns undefined | string
The next value from the input sequence

Throws
If the sequence is empty and an errMsg is provided, the method will throw an error with the given message; otherwise, t will throw Unexpected end of input.
Inherited from Tokenizer.peek
- Defined in src/tokenizer.ts:140

registerNumberRegex

registerNumberRegex(numberRegEx): void
Change the default number RegEx (/\d/i)
Parameters
- numberRegEx: RegExp
Returns void
- Defined in src/string-tokenizer.ts:69

registerWhitespaceRegex

registerWhitespaceRegex(whitespaceRegEx): void
Change the default whitespace RegEx (/\s|\n|\t/).
Parameters
- whitespaceRegEx: RegExp
Returns void
- Defined in src/string-tokenizer.ts:31

registerWordRegex

registerWordRegex(wordRegEx): void
Change the default word RegEx (/[a-z]/i)
Parameters
- wordRegEx: RegExp
Returns void
- Defined in src/string-tokenizer.ts:50

shift

shift(errMsg?): string
This method unshifts and returns the next value from vals If no more values are available, will throw an error.
Parameters
- OptionalerrMsg: string
  An optional error message
Returns string
The next value from the input sequence

Throws
If the sequence is empty and an errMsg is provided, the method will throw an error with the given message; otherwise, t will throw Unexpected end of input.
Inherited from Tokenizer.shift
- Defined in src/tokenizer.ts:122

tokenize

tokenize(): StringTokenizer<R>
Tokenizes the input values by processing them one at a time.

Each value in vals is processed by calling the abstract method onNextToken, which must be implemented by any subclass of Tokenizer.

Returns StringTokenizer<R>
Inherited from Tokenizer.tokenize
- Defined in src/tokenizer.ts:50

Class StringTokenizer<R>Abstract

Type Parameters

Hierarchy (view full)

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Type Parameters

Parameters

Returns StringTokenizer<R>

Properties

length

path

raw

tokens

Usage Notes

vals

Accessors

position

Returns number

Methods

consume

Parameters

Returns { until: ((predicate: ((val: string) => boolean)) => string[]); while: ((predicate: ((val: string) => boolean)) => string[]); }

until: ((predicate: ((val: string) => boolean)) => string[])

Parameters

Parameters

Returns boolean

Returns string[]

while: ((predicate: ((val: string) => boolean)) => string[])

Parameters

Parameters

Returns boolean

Returns string[]

consumeNumber

Parameters

Returns string

consumeWhitespace

Returns string

consumeWord

Parameters

Returns string[]

AbstractonNextToken

Parameters

Returns void

peek

Parameters

Returns undefined | string

Throws

registerNumberRegex

Parameters

Returns void

registerWhitespaceRegex

Parameters

Returns void

registerWordRegex

Parameters

Returns void

shift

Parameters

Returns string

Throws

tokenize

Returns StringTokenizer<R>

Settings

On This Page

Class StringTokenizer<R>`Abstract`

Returns {
until: ((predicate: ((val: string) => boolean)) => string[]);
while: ((predicate: ((val: string) => boolean)) => string[]);
}

`Abstract`onNextToken