Booster Structure

Once you have generated the booster source, you are free to change it in whatever way you fancy. On schema updates, unfortunately, you will have to merge in changes manually, as we have not found a sensible and general way to preserve arbitrary source changes when (re-)generating a booster. If you have a creative idea how better to separate generated and hand-made code, we're certainly interested. However, as usual: version control helps.

The code generated starts somewhat like this:

#include <math.h>
#include <string.h>
#include "boosterskel.h"

#define QUERY_N_PARS 33

enum outputFields {
  fi_localid,              /* Identifier, text */
  fi_pmra,                 /* PM (alpha), real */
  fi_pmde,                 /* PM (delta), real */

QUERY_N_PARS here is the number of columns in the destination table. It is also the number of items in the outputFields enum, which in turn defines the structure of the record written to the database. All this must perfectly match what DaCHS runs to create the table, and hence it is generally unwise to manually modify it. Conversely, again: If you add or remove a column, you will have to replace this part of your booster with what DaCHS generates then.

Within outputFields are the column names in the destination table prepended with a simple fi\_. If you only use these names to access fields, cutting and pasting on later schema changes should be fairly painless and safe.

While you should not need to change any of this, you in general have to change the getTuple function. What it looks like strongly depends on the sort of booster you're generating for; even the prototypes of this function depend on the sort of booster.

What is common is that getTuple needs to return a Field array. All boosters declare the return value like this:

static Field vals[QUERY_N_PARS];

– it needs to be static as a pointer to it is returned from the function; do not rely on anything in there to be stable across function calls, though, as the serialization to COPY material might mess around in that memory. The name vals is expected by, e.g., the F macro and must therefore not be changed.

Field is defined in boosterskel.h somewhat like this:

typedef struct Field_s {
  valType type;
  int length; /* ignored for anything but VAL_TEXT and VAL_ARRAY */
  union {
    char *c_ptr;
    unsigned char *arr_ptr;
    double c_double;
    float c_float;
    int64_t c_int64;
    int32_t c_int32;
    int16_t c_int16;
    int8_t c_int8;
    time_t time;
  } val;
} Field;

To see the definition your version of DaCHS has, run dachs adm dump src/boosterskel.h.

The type in there is one of:

typedef enum valType_e {
  VAL_NULL,
  VAL_BOOL,
  VAL_CHAR,
  VAL_SHORT,
  VAL_INT,
  VAL_BIGINT,
  VAL_FLOAT,
  VAL_DOUBLE,
  VAL_TEXT,
  VAL_JDATE,  /* a julian year ("J2000.0"); this is stored as a simple double */
  VAL_DATE,   /* date expressed as a time_t */
  VAL_DATETIME, /* date and time expressed as a time_t */
  VAL_ARRAY,  /* pre-serialised by pack*Array, length is # of *bytes* */
} valType;

(again, adm dumpDF might give you a more recent version).

JDATE is a julian day number to be dumped as a date (rather than a datetime). For other ways to represent dates and datetimes, see below.

You can, and frequently will, fill the vals by hand. There are, however, a couple of functions that take care of some standard situations:

• void linearTransform(Field *field, double offset, double factor) – changes field in place to offset+factor*oldValue. Handles NULL correctly, silently does nothing for anything non-numeric
• void parseFloatWithMagicNULL(char *src, Field *field, int start, int len, char *magicVal) – parses a float from a decimal serialisation in src[start:start+len] into field, writing NULL when magicVal is found in the field.
• void parseDouble(char *src, Field *field, int start, int len) – parses a double from src[start:start+len] into field, writing NULL if it's whitespace only.
• void parseInt(char *src, Field *field, int start, int len) – parses a 32-bit int from src[start:start+len] into field.
• void parseShort(char *src, Field *field, int start, int len) – parses a 16-bit int from src[start:start+len] into field.
• void parseBlankBoolean(char *src, Field *field, int srcInd) – parses a boolean such that field becomes true when src[srcInd] is nonempty.
• void parseBigint(char *src, Field *field, int start, int len) – parses a 64-bit int from src[start:start+len] into field.
• void parseString(char *src, Field *field, int start, int len, char *space) – copies len bytes starting at start from src into space (you are responsible for allocating that; usually, a static buffer should do, since the postgres input is generated before the next input line is parsed) and stuffs the whole thing into field.
• void parseChar(char *src, Field *field, int srcInd) – copies the value at *src into field, making that a char.
• MAKE_NULL(fi) – makes the column fi points to NULL
• MAKE_DOUBLE(fi, value) – puts a C double value into the column fi points to
• MAKE_BIGINT(fi, value) – puts a C long long value into the column fi points to
• MAKE_FLOAT(fi, value) – puts a C float value into the column fi points to
• MAKE_SHORT(fi, value) – puts a C short value into the column fi points to
• MAKE_CHAR(fi, value) – puts a C char value into the column fi points to
• MAKE_JDATE(fi, value) – writes a float/double value as a julian date (a postgres timestamp).
• MAKE_TEXT(fi, value) – note that you must manage the memory of value yourself. In particular, it must not be automatic memory of getTuple, since that will not be valid when the tuple actually is built. Most commonly, you'll be using a static buffer here.
• MAKE_CHAR_NULL(fi, value, nullvalue) – makes fi a char with value unless value==nullvalue; in that case, fi becomes a NULL
• MAKE_ARRAY(fi, bufptr, lenOfBuf) – see Arrays.
• double mjdToJYear(mjd) – returns a julian year for mjd
• AS2DEG(field) – turns a field value in arcsecs to degrees
• MAS2DEG(field) – turns a field value in milli-arcsecs to degrees

Of course, you can also manually copy or delimit data and use fieldscanf as documented in split boosters

As said above, dachs imp will compile and link your code together with the files boosterskel.c and boosterskel.h that come with the DaCHS distribution. If you insist on manually building a booster, use:

gavo admin dumpDF src/boosterskel.c # or .h

to pull these files into your working directory. Watch the build commands DaCHS executes when you change a booster source and run dachs imp to figure out the compile and link commands appropriate for your source.

Filling vals Manually

The F(index) macro lets you access the field info directly. So, you could enter a fixed-length piece of memory into fi_magic like this:

static char bufForMagic[8];

memcpy(bufForMagic, inputLine+20, 8);
F(fi_magic)->type = VAL_TEXT;
F(fi_magic)->val.c_ptr =  bufForMagic;
F(fi_magic)->length = 8;

Having static buffers in getTuple is usually ok since the COPY input is generated before getTuple is called again.

It is quite common to have to handle null values. In the example above, this could look like this if a NULL for magic were signified by a F in inputLine[19]:

static char bufForMagic[8];

if (inputLine[19]=='F') {
  F(fi_magic)->type = VAL_NULL;
} else {
  memcpy(bufForMagic, inputLine+20, 8);
  ...

Skipping a record

If you need to skip a record, do:

longjmp(ignoreRecord)

in getTuple. That works independently of the booster type.

Dates and times

The boosters treat “normal” dates and datetimes as struct tm-s. If you need a larger range, use VAL_JDATE, which lets you store julian dates in floats. Julian dates are serialized to dates rather than datetimes.

To parse VAL_DATE or VAL_DATETIME, you will write something like:

fieldscanf(curCont, fi_date, VAL_DATE, "%Y-%m-%d");

if parsing from date strings. If your input is something weird, figure out a way to generate a struct tm as defined in time.h. Then write:

struct tm timeParts;
timeParts.tm_sec = 12;
...
timeParts.tm_year = 1920;
F(fi_dt)->val.time = timeParts;
F(fi_dt).type = VAL_DATETIME;

(or VAL_DATE, as the case may be).

Having said all this, long experience has taught us it's ususally best do have dates and such in the database as MJD or julian years. You can format those to ISO strings (or, really, anything else you want) on output by using display hints on outpuField or even column itself.

MJDs are just so much easier to handle within ADQL queries. Support for timestamps, on the other hand, is extremely lousy.

Arrays

Boosters can write arrays, but for now it's a bit rudimentary. If you have ideas for abstracting this a bit, do contact the authors.

Note: If you have boosters still calling packFloatArray(buf, float_arr, nVals), replace that call with packArray(buf, vals, nVals, 700, sizeof(float), 1) (or make that final thing a 0 if you're on a big-endian platform).

The central (and mean) function has the signature:

size_t packArray(
        unsigned char *buf,  /* the target buffer */
        unsigned char *vals, /* the values to serialise */
        int nVals,           /* number of items in vals */
        int itemoid,         /* postgres UID of the target type */
        size_t itemsize,     /* size of each item in vals */
        int littleEndian     /* have this 1 when incoming data is little-endian */
        ) -> #bytes left in buf.

This writes a postgres binary array representation of nVals items of length itemsize into buf and returns the number of bytes put in there. buf needs to hold at least 36+nVals*(4+itemsize) bytes.

This needs the postgres oid for the the type to be written. These are fairly constant, so here are oids for the more common types (you can figure these out by running select oid, typname from pg_type):

Note that packArray only re-packs the bytes; hence, vals must already contain values appropriate for postgres' type.

The littleEndian parameter will be 1 in most common circumstances (dumping native data on x86); it makes packArray flip the byte order, which in postgres dumps is always big endian. If you are on a big-endian machine or have big-endian data in memory or have big-endian floats (which may happen on some little-endian architectures, I'm told), make it 0.

This function goes together with the MAKE_ARRAY(fi, buf, nbytes) macro that tells postgres that in the referenced field info there's nbytes of serialised array material. As with strings, you need to make buf live beyond the execution time of getTuple, and again we recommend doing that with static arrays. Hence, to write a 4-array collected from four consecutive array items starting at index 7 into a matarr database column, you could have in your getTuple:

#define ARRLEN 4
#define ARRBUFSZ (36+ARRLEN*8)

static unsigned char matbuf[ARRBUFSZ];
float tmparr[ARRLEN];  /* does not need to be static */
size_t tmplen;
...
for (i=0; i<ARRLEN; i++) {
  tmparr[i] = data[7+i][rowIndex];
}
tmplen = packArray(matbuf, tmparr, ARRLEN, 700, sizeof(float), 1);
MAKE_ARRAY(fi_matarr, matbuf, tmplen);

Line-based boosters

These are boosters that read from a text file, line by line. The maximumg line length is set by the direct grammar's recordSize attribute.

It is up to the parsing function to split and digest this text line.

Field *getTuple(char *inputLine)
{
  static Field vals[QUERY_N_PARS];

  parseWhatever(inputLine, F(fi_localid), start, len);
  parseFloat(inputLine, F(fi_pmra), start, len);
  parseFloat(inputLine, F(fi_pmde), start, len);
  parseFloat(inputLine, F(fi_raerr), start, len);

 char *curCont = strtok(inputLine, "\t");
fieldscanf(curCont, fi_objid, VAL_INT_64);
curCont = strtok(NULL, "\t");
fieldscanf(curCont, fi_run, VAL_SHORT);

#define strtok strtok_u

Bin boosters

When you get binary data of fixed record length, set the recordSize attribute on the DirectGrammar element:

<directGrammar type="bin" recordSize="300"...

Note that a recordSize larger than INPUT_LINE_MAX will cause a buffer overflow.

You are mainly on your own in terms of segmentation, but for entering values, you can use the MAKE_* macros discussed above.

For these in particular, use the the portable type specifiers for integral types, viz., int8_t, int16_t, int32_t, and int64_t and these names with a u in front.

In particular with binary boosters, it is essential you always properly cast what you read, e.g.,:

MAKE_DOUBLE(fi_dej2000, -90+*(int32_t*)(line+4)/1e6); /* SPD */

when a declination is given as mas of south polar distance.

FITS boosters

These read from FITS binary tables, which come with enough metadata that it might just be possible to make DaCHS generate a booster working without editing any C code.

To build one of those, DaCHS inspects the first file matched by the parent data's sources element (which also means these won't work outside of a data element). DaCHS where there is a match for a table column after lowercasing the name in the FITS table, DaCHS will automatically add a suitable mapping instruction. FITS table columns without a match in the database table are ignored, table columns without matches in the FITS are set to NULL.

FITS binary tables are organized by columns rather than by rows, bearing witness to their FORTRAN heritage. The way the boosters are currently generated, all these columns are completely read into memory, which means you cannot ingest FITS binary tables that do not fit into your machine's memory. Fixing this would be fairly straightforward (patches are welcome, but we'll also fix this if you ask for it).

In case your column names in the database are different from the FITS input (modulo case), you can use Element mapKeys to inform the direct grammar. For instance, <mapKeys> raj2000:RA, dej2000:DEC </mapKeys> will map column named RA in your source file to column named raj2000 in your database table and analoguosly for DEC. If you don't do this, only column names from your DB table will be read and imported.

If you need to postprocess the items, we recommend you do that again in the getTuple function (note how that gets passed the row index) for maintainability, rather than directly after reading the rows.

Attention: The system will not warn you if the type of a column in the table is not compatible with what you have in the database. If it is, the program will probably silently dump garbage into the db, though if you're lucky it'll crash. This is almost on purpose. It will let you do manual type conversions like, for example, making a 64 bit integer from a string as follows:

if (nulls[18][rowIndex]) {
  MAKE_NULL(fi_ppmxl);
} else {
   parseBigint(((char**)(data[18]))[rowIndex], F(fi_ppmxl), 0, 19);
}
return vals;

(we could admittedly warn you if this kind of thing becomes necessary, and we'll gratefully accept patches for that).

HDF5 boosters

HDF5 is a fairly complex binary format that allows any number of different serialisations of tables. The Element hdf5Grammar can parse some of these.

Directgrammar can generate boosters for a few HDF5 dialects. list above.

As with FITS, there is some basic metadata in HDF5 files that lets DaCHS infer mappings when HDF5 and column names match or when mapKeys provides some. Hence, again, to generate the booster, you will have to have at least one data file so DaCHS can infer this mapping.

The getTuple for hdf5vaex boosters has two arguments: rowIndex (for informational purposes) and data, which is of type InRec*. This type is a record that DaCHS defines for you and that has the input file's columns as names. Hence, in the ideal case, you can directly build the vals entries from data members, somewhat like this:

MAKE_BIGINT(fi_source_id, ((uint64_t)(data->source_id)));
MAKE_FLOAT(fi_a0_best, ((float)(data->A0_best)));

– this is what DaCHS generates.

Note that HDF5 does not define portable NULL representations (or if it does, then DaCHS knows nothing about them). Hence, you will have to worry about NULLs yourself.