ic/rodfiles

Source Edit

Low level binary format used by the compiler to store and load various AST and related data.

NB: this is incredibly low level and if you're interested in how the compiler works and less a storage format, you're probably looking for the ic or packed_ast modules to understand the logical format.

Overview

RodFile represents a Rod File (versioned binary format), and the associated data for common interactions such as IO and error tracking (RodFileError). The file format broken up into sections (RodSection) and preceded by a header (see: cookie). The precise layout, section ordering and data following the section are determined by the user. See ic.loadRodFile.

A basic but "wrong" example of the lifecycle:

1. create or open - create a new one or open an existing
2. storeHeader - header info
3. storePrim or storeSeq - save your stuff
4. close - and we're done

Now read the bits below to understand what's missing.

Issues with the Example

Missing Sections: This is a low level API, so headers and sections need to be stored and loaded by the user, see storeHeader & loadHeader and storeSection & loadSection, respectively.

No Error Handling: The API is centered around IO and prone to error, each operation checks or sets the RodFile.err field. A user of this API needs to handle these appropriately.

API Notes

Valid inputs for Rod files

ASTs, hopes, dreams, and anything as long as it and any children it may have support copyMem. This means anything that is not a pointer and that does not contain a pointer. At a glance these are:

• string
• objects & tuples (fields are recursed)
• sequences AKA seq[T]

Note on error handling style

A flag based approach is used where operations no-op in case of a preexisting error and set the flag if they encounter one.

Misc

• 'Prim' is short for 'primitive', as in a non-sequence type

Types

RodFile = object
  f*: File
  currentSection*: RodSection
  err*: RodFileError

Source Edit

RodFileError = enum
  ok, tooBig, cannotOpen, ioFailure, wrongHeader, wrongSection, configMismatch,
  includeFileChanged

Source Edit

RodSection = enum
  versionSection, configSection, stringsSection, checkSumsSection, depsSection,
  numbersSection, exportsSection, hiddenSection, reexportsSection,
  compilerProcsSection, trmacrosSection, convertersSection, methodsSection,
  pureEnumsSection, toReplaySection, topLevelSection, bodiesSection,
  symsSection, typesSection, typeInstCacheSection, procInstCacheSection,
  attachedOpsSection, methodsPerGenericTypeSection, enumToStringProcsSection,
  methodsPerTypeSection, dispatchersSection, typeInfoSection,
  backendFlagsSection, aliveSymsSection, sideChannelSection, namespaceSection,
  symnamesSection

Source Edit

Procs

proc close(f: var RodFile) {....raises: [], tags: [], forbids: [].}

Source Edit

proc create(filename: string): RodFile {....raises: [], tags: [], forbids: [].}

create the file and open it for writing Source Edit

proc loadHeader(f: var RodFile; cookie = defaultCookie) {....raises: [IOError],
    tags: [ReadIOEffect], forbids: [].}

Loads the header which is described by cookie. Source Edit

proc loadOrderedTable[K, T](f: var RodFile; s: var OrderedTable[K, T])

T must be compatible with copyMem, see loadPrim Source Edit

proc loadPrim(f: var RodFile; s: var string) {....raises: [IOError],
    tags: [ReadIOEffect], forbids: [].}

Read a string, the length was stored as a prefix Source Edit

proc loadPrim[T](f: var RodFile; x: var T)

Load a non-sequence/string T. Source Edit

proc loadSection(f: var RodFile; expected: RodSection) {....raises: [IOError],
    tags: [ReadIOEffect], forbids: [].}

read the bytes value of s, sets and error if the section is incorrect. Source Edit

proc loadSeq[T](f: var RodFile; s: var seq[T])

T must be compatible with copyMem, see loadPrim Source Edit

proc open(filename: string): RodFile {....raises: [], tags: [], forbids: [].}

open the file for reading Source Edit

proc storeHeader(f: var RodFile; cookie = defaultCookie) {....raises: [IOError],
    tags: [WriteIOEffect], forbids: [].}

stores the header which is described by cookie. Source Edit

proc storeOrderedTable[K, T](f: var RodFile; s: OrderedTable[K, T])

Source Edit

proc storePrim(f: var RodFile; s: string) {....raises: [IOError],
    tags: [WriteIOEffect], forbids: [].}

Stores a string. The len is prefixed to allow for later retreival. Source Edit

proc storePrim[T](f: var RodFile; x: T)

Stores a non-sequence/string T. If T doesn't support copyMem and is an object or tuple then the fields are written -- the user from context will need to know which T to load. Source Edit

proc storeSection(f: var RodFile; s: RodSection) {....raises: [IOError],
    tags: [WriteIOEffect], forbids: [].}

update currentSection and writes the bytes value of s. Source Edit

proc storeSeq[T](f: var RodFile; s: seq[T])

Stores a sequence of Ts, with the len as a prefix for later retrieval. Source Edit