Class Tokenizer<T, R>Abstract

constructor

• new Tokenizer<T, R>(vals): Tokenizer<T, R>

• Constructor for the Tokenizer class.

  Type Parameters

  • T
  • R

  Parameters

  • vals: T[]

    The input values to be tokenized. This array is iterated over in onNextToken to generate tokens.

  Returns Tokenizer<T, R>

length

tokens

vals

position

• get position(): number

• Calculates the current token's position as the difference between the initial length of the input values and the number of tokens that have already been produced.

  Returns number

consume

• consume(val?): {
      until: ((predicate: ((val: T) => boolean)) => T[]);
      while: ((predicate: ((val: T) => boolean)) => T[]);
  }

• Consumes a sequence of values from vals.

  Parameters

  • Optionalval: T

  Returns {
      until: ((predicate: ((val: T) => boolean)) => T[]);
      while: ((predicate: ((val: T) => boolean)) => T[]);
  }

  An object that allows for the consumption of values, providing methods until and while. These methods determine when to stop or continue the consumption of the sequence.

  • until: ((predicate: ((val: T) => boolean)) => T[])

    • • (predicate): T[]

      • Parameters

        • predicate: ((val: T) => boolean)
          • • (val): boolean

            • Parameters

              • val: T

              Returns boolean

        Returns T[]

  • while: ((predicate: ((val: T) => boolean)) => T[])

    • • (predicate): T[]

      • Parameters

        • predicate: ((val: T) => boolean)
          • • (val): boolean

            • Parameters

              • val: T

              Returns boolean

        Returns T[]

AbstractonNextToken

• onNextToken(val): void

• Abstract method to process the next token.

  This method must be implemented by subclasses of Tokenizer. It defines how each value in vals should be converted into a token.

  Parameters

  • val: T

    The next value to process into a token.

  Returns void

peek

• peek(lookahead?): undefined | T

• This method returns the next value in the sequence to be processed. If no more values are available, will throw an error.

  Parameters

  • lookahead: number = 0

  Returns undefined | T

  The next value from the input sequence

  Throws

  If the sequence is empty and an errMsg is provided, the method will throw an error with the given message; otherwise, t will throw Unexpected end of input.

shift

• shift(errMsg?): T

• This method unshifts and returns the next value from vals If no more values are available, will throw an error.

  Parameters

  • OptionalerrMsg: string

    An optional error message

  Returns T

  The next value from the input sequence

  Throws

  If the sequence is empty and an errMsg is provided, the method will throw an error with the given message; otherwise, t will throw Unexpected end of input.

tokenize

• tokenize(): Tokenizer<T, R>

• Tokenizes the input values by processing them one at a time.

  Each value in vals is processed by calling the abstract method onNextToken, which must be implemented by any subclass of Tokenizer.

  Returns Tokenizer<T, R>