Module lexbase

This module implements a base object of a lexer with efficient buffer handling. Only at line endings checks are necessary if the buffer needs refilling.

Imports

strutils, streams

Types

BaseLexer = object of RootObj
  bufpos*: int                 ## the current position within the buffer
  when defined(js):
      buf*: string

  else:
      buf*: cstring

  bufLen*: int                 ## length of buffer in characters
  input: Stream                ## the input stream
  lineNumber*: int             ## the current line number
  sentinel: int
  lineStart: int
  offsetBase*: int
  refillChars: set[char]

the base lexer. Inherit your lexer from this object.

Consts

EndOfFile = '\x00'

end of file marker

NewLines = {'\c', '\n'}

Procs

proc close(L: var BaseLexer) {...}{.raises: [Exception], tags: [].}

closes the base lexer. This closes L's associated stream too.

proc handleCR(L: var BaseLexer; pos: int): int {...}{.raises: [Exception],
    tags: [ReadIOEffect].}

Call this if you scanned over 'c' in the buffer; it returns the the position to continue the scanning from. pos must be the position of the 'c'.

proc handleLF(L: var BaseLexer; pos: int): int {...}{.raises: [Exception],
    tags: [ReadIOEffect].}

Call this if you scanned over 'L' in the buffer; it returns the the position to continue the scanning from. pos must be the position of the 'L'.

proc handleRefillChar(L: var BaseLexer; pos: int): int {...}{.raises: [Exception],
    tags: [ReadIOEffect].}

To be documented.

proc open(L: var BaseLexer; input: Stream; bufLen: int = 8192;
         refillChars: set[char] = NewLines) {...}{.raises: [Exception],
    tags: [ReadIOEffect].}

inits the BaseLexer with a stream to read from.

proc getColNumber(L: BaseLexer; pos: int): int {...}{.raises: [], tags: [].}

retrieves the current column.

proc getCurrentLine(L: BaseLexer; marker: bool = true): string {...}{.raises: [], tags: [].}

retrieves the current line.