gavo.grammars.directgrammar module

A grammar supporting C language boosters (or possibly other mechanisms bypassing internal dbtable).

These actually bypass most of our machinery and should only be used if performance is paramount. Otherwise, CustomGrammars play much nicer with the rest of the DC software.

Currently, only one kind of DirectGrammar is supported: C boosters.

class gavo.grammars.directgrammar.BinCodeGenerator(grammar, tableDef)

Bases: _CodeGenerator

a code generator for reading fixed-length binary records.

getFooter()

returns the code for the createDumpfile method.

You want to use the C fragments above for that.

The default returns something that bombs out.

getItemParser(item, index)

returns code that parses item (a Column instance) at column index index.

You’re free to ignore index.

getPreamble()

returns a list of lines that make up the top of the booster.

class gavo.grammars.directgrammar.CBooster(srcName, dataDesc, recordSize=4000, gzippedInput=False, autoNull=None, preFilter=None, ignoreBadRecords=False, customFlags='')

Bases: object

is a wrapper for an import booster written in C using the DC booster infrastructure.

Warning: If you change the booster description, you’ll need to touch the source to recompile.

getOutput(argName)

returns a pipe you can read the booster’s output from.

As a side effect, it also sets the attribute self.pipe. We need this to be able to retrieve the command status below.

getStatus()

silence_for_test = False

class gavo.grammars.directgrammar.ColCodeGenerator(grammar, tableDef)

Bases: _LineBasedCodeGenerator

getItemParser(item, index)

returns code that parses item (a Column instance) at column index index.

You’re free to ignore index.

getPrototype()

returns the prototype of the getTuple function.

class gavo.grammars.directgrammar.DirectGrammar(parent, **kwargs)

Bases: Structure, RestrictionMixin

A user-defined external grammar.

See the booster.html on user-defined code on more on direct grammars.

You will almost always use these in connection with C code generated by dachs mkboost.

Properties:

• dataset: For HDF5 boosters, the HDF path to the table to be imported (in vaex, not including “columns”). Certain, hopefully sensible, defaults apply depending on the HDF5 dialect.

• chunkSize: In HDF5 boosters, now many rows to pull into memory at one time (defaults to 5000).

attrSeq = [<gavo.base.attrdef.UnicodeAttribute object>, <gavo.rscdef.common.ResdirRelativeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.IntAttribute object>, <gavo.base.attrdef.BooleanAttribute object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.BooleanAttribute object>, <gavo.base.complexattrs.StructAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.complexattrs.PropertyAttribute object>, <gavo.rscdef.common.RDAttribute object>, <gavo.base.attrdef.IntAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.EnumeratedUnicodeAttribute object>]

property cBooster

clearProperty(name)

completedCallbacks = []

getBooster()

getFullId()

getProperty(name, default=<Undefined>)

hasProperty(name)

isDispatching = False

managedAttrs = {'autoNull': <gavo.base.attrdef.UnicodeAttribute object>, 'cBooster': <gavo.rscdef.common.ResdirRelativeAttribute object>, 'customFlags': <gavo.base.attrdef.UnicodeAttribute object>, 'extension': <gavo.base.attrdef.IntAttribute object>, 'gzippedInput': <gavo.base.attrdef.BooleanAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'ignoreBadRecords': <gavo.base.attrdef.BooleanAttribute object>, 'mapKeys': <gavo.base.complexattrs.StructAttribute object>, 'preFilter': <gavo.base.attrdef.UnicodeAttribute object>, 'properties': <gavo.base.complexattrs.PropertyAttribute object>, 'property': <gavo.base.complexattrs.PropertyAttribute object>, 'rd': <gavo.rscdef.common.RDAttribute object>, 'recordSize': <gavo.base.attrdef.IntAttribute object>, 'splitChar': <gavo.base.attrdef.UnicodeAttribute object>, 'type': <gavo.base.attrdef.EnumeratedUnicodeAttribute object>}

name_ = 'directGrammar'

notify = True

onElementComplete()

parse(sourceToken, targetData=None)

property rd

setProperty(name, value)

validate()

class gavo.grammars.directgrammar.FITSCodeGenerator(grammar, tableDef)

Bases: _NumpyMetaCodeGenerator

A code generator for reading from FITS binary tables.

getFooter()

returns the code for the createDumpfile method.

You want to use the C fragments above for that.

The default returns something that bombs out.

getItemParser(item, index)

returns code that parses item (a Column instance) at column index index.

You’re free to ignore index.

getPreamble()

returns a list of lines that make up the top of the booster.

getPrototype()

returns the prototype of the getTuple function.

class gavo.grammars.directgrammar.SplitCodeGenerator(grammar, tableDef)

Bases: _LineBasedCodeGenerator

a code generator for parsing files with lineas and separators.

getItemParser(item, index)

returns code that parses item (a Column instance) at column index index.

You’re free to ignore index.

getPreamble()

returns a list of lines that make up the top of the booster.

getSetupCode()

returns a sequence of C lines for code between an item parser.

gavo.grammars.directgrammar.buildSource(grammar, td)

returns (possibly incomplete) C source for a booster to read into td.

gavo.grammars.directgrammar.getCodeGen(grammar, tableDef)

returns the code generator suitable for making code for grammar.

gavo.grammars.directgrammar.getEnum(td, grammar)

gavo.grammars.directgrammar.getGetTuple(td, codeGen)

gavo.grammars.directgrammar.getGrammarAndTable(grammarId)

returns a pair of directGrammar and table being fed for a cross-rd reference.

gavo.grammars.directgrammar.getHDF5Generator(name)

a factory for HDF5 generators classes.

This is built like this because directhdf5 imports this module and we need to avoid circular imports. If this kind of construct becomes more common, we should pull basic code generation stuff into a boostercommon module. Meanwhile, this is a convenient place to produce a useful error message for when h5py is not available.

gavo.grammars.directgrammar.getNameForItem(item)

gavo.grammars.directgrammar.getSource(grammarId)

returns a bytestring containing C source to parse grammarId.

gavo.grammars.directgrammar.indent(stringList, indentChar)

gavo.grammars.directgrammar.main()

gavo.grammars.directgrammar.parseCmdLine()