Package Layout

The following rules should be followed as regards subpackages of gavo in order to keep the modules dependency graph manageable (and facilitate factoring out libraries).

• Each functionality block is in a subpackage, the __init__ for which contains the main functions, classes, etc., of the sub-package interface most clients will be concerned with. Clients needing special tricks may still import individual modules (but then they're in much larger danger of breaking). What's in __init__ should be considered "public interface" and hence changed very carefully if at all.
• Within each subpackage, no module imports the sub-package, i.e., a module in base never says "from gavo import base"
• A subpackage may have a module common, containing objects that multiple modules within that subpackage requires. common may not import any module from the subpackage, but may be imported from all of them. No rules wrt importing modules from the same subpackage exist of other modules. Just apply common sense here to avoid circular imports.
• Don't use import *. It interferes with our static checking. For relative imports, we will probably be slowly migrating towards from . import over the current absolute imports (from gavo.base import...).
• There is a hierarchy of subpackages, where subpackages lower in the hierarchy may not import anything from the higher or equal levels, but only from lower levels. This hierarchy currently looks like this: imp [<] utils < stc < (votable, adql) < base < dm < rscdef < grammars < formats < rsc < svcs < registry < protocols < web < rscdesc < (helpers, user) utils should never assume anything from imp is present, i.e., it may attempt to import from there, but it should not fail hard if the import doesn't work. Of course, concrete functions (e.g., from utils.fitstools) won't work if the base libraries are not present.

Getting Table Metadata and Querying Tables

The preferred way to do simple queries against tables in DaCHS these days is:

1. get the table metadata:

   td = base.resolveCrossId("resdir/q#mytable")
   

2. use the td's doSimpleQuery(selectClause, fragments, params) method to get dicts of rows; all arguments are optional and default to pulling all; selectClause is just a list of column names:

   for row in td.doSimpleQuery(["col1", "col2"],
       "col1<%(lim1)s AND col2=%(foo)s",
       {'lim1': 23, 'foo': 42}):
     print row
   

When you need explicit connection management or want to do more complex operations, use the context managers (typically getTableConn when querying, getWritableAdminConn when writing) using a context manager:

with base.getTableConn() as conn:
  for row in conn.queryToDicts(myComplexQuery,
      {'arg1': 23, 'pat': 'M32'}):
    ...

Besides queryToDicts, there's also query, which yields tuples. Both are iterators, which means that the queries contained will not be executed unless you fetch at least one row.

Hence, for queries that don't return anything (DDL, inserts, etc), use conn.execute.

In contrast to doSimpleQuery, all these will do case folding of the select list items.

In ”user” code, get these symbols from api instead of from base.

Another feature you might like to know about is the connections's parameters context manager. It takes postgres settings keys and values and will reset them at the end of the controlled block. This is useful for things like timeouts and the like, e.g.,:

with conn.parameters([
              ("statement_timeout", "%s ms"%int(timeout*1000))]):
              whatever

Versioning Issues

DaCHS itself is versioned such that minor versions (e.g., 2.6) are releases, which technically means they get a bit more pre-release testing, and they get properly announced on dachs-users. Micro versions on to of that are then beta releases including new features up to the next stable release. Hence, 2.6.1 will in general be less stable than 2.6. Perhaps we should change this to something less flamboyant one of these days.

When we support protocols, we treat major versions as separate standards, i.e., there are separate RDs for, say, siap and siap2. Within each such RD, mixins or other evolving material may be tagged with the minor version. For instance a mixin table-0 would correspond to version 1.0 (or 2.0) of a standard. This hasn't been done consistently in DaCHS' past, so you'll see all kinds of other experiments. But the minor-version tagging is what should happen with future developments.

Error handling, logging

from gavo.base import ObserverBase, listensTo

class PlainUI(ObserverBase):
  @listensTo("NewSource")
  def announceNewSource(self, srcString):
    print "Starting %s"%srcString

from gavo import base

def handleNewSource(srcToken):
  pass

base.ui.subscribeNewSource(handleNewSource)

from gavo.user import logui
logui.LoggingUI(base.ui)

raise base.ui.logOldExc(GavoError(...))

Testing

In an installed checkout of DaCHS, you can go to the tests subdirectory and run:

python3 runAllTests.py

for a fairly extensive set of unit tests.

This needs to create a test database, and that will only work if whoever runs this is postgres superuser. dachsroot from the Debian package already is. If you want to run tests as another user, you'll have to make yourself a suitable account, typically with:

sudo -u postgres createuser -s `id -nu`

The suite will not tear down and build up everything each time it's called. To make it rebuild everything, remove ~/_gavo_test and dropdb dachstest.

Also, you'll need the extra package python3-testresource (which the dachs packages don't declare as a dependency), and you'll need build-essentials as well as libcfitsio-dev

I'm testing against concrete error messages, and DaCHS sometimes hands through messages from the database. Hence, some tests will fail when lc_messages in postgresql.conf isn't C.

This uses some management of test scaffolds; when something is severely wrong, generating these scaffolds can fail and the execution of the suite will stop. I'm not decided whether to regard that as a bug or a feature, but I'll not fix it any time soon. So, if this bites you, find out why resource generation fails and fix it.

export CLASSPATH=/usr/share/doc/libxerces2-java-doc/examples/xercesSamples.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar
exec java dom.Counter -n -v -s -f $@

import os
os.environ["GAVO_OOTTEST"] = "dontcare"

from gavo.helpers import testhelpers

pyannotate -w -3 --type-info=/path/to/types.json dir/file.py

# pragma: no cover

openssl genrsa -out server.key 2048
openssl req -new -x509 -key server.key -out server.pem -days 2000
cat server.key server.pem > bundle.pem

Configuration

DaCHS has far too many different configuration hooks: gavo.rc, defaultmeta.txt, the database profiles, vanitynames.txt, userconfig.rd, as well as locally-overridden system RDs and templates. At least defaultmeta.txt was a mistake, as was probably vanitynames.txt. We should be working on getting rid of it.

New configuration should preferably go into userconfig.rd, while there's always going to be room for gavo.rc, too.

Configuration items for userconfig.rd typically are going to be STREAMs. To provide fallbacks for those if the user hasn't defined any, there's //userconfig, which also serves as built-in documentation for what's there. As an identifier is resolved in //userconfig, the system first looks in a etc/userconfig.rd and then, even if that file exists (but has no element with the id in question), in //userconfig.

When using the elements, always use the canonical abbreviation for userconfig, %, as in <FEED source="%#registry-interfacerecords"/>.

if "dali-interface-in-tap-1" in base.getConfig("future"):
  ...

future: dali-interface-in-tap-1, some-other-new-feature-if-necessary

Structures

Resource description within DaCHS works via instances of base.Structure. These parse themselves from XML strings, do validation, etc. All compound RD elements correspond to a structure class (well, almost; meta is an exception).

A structure instance has the following callbacks:

• completeElement(ctx) -- called when the element's closing tag is encountered, used to fill in computed defaults. ctx is a parse context that you can use to, e.g. resolve XML ids.
• validate() -- called after completeElement, used to raise errors if some gross ("syntactic") mistakes are in the element
• onElementComplete() -- called after validate, i.e., elementCompleted can rely on seeing a "valid" structure

In addition, structures can define onParentCompleted methods. These are called after they parent's onElementComplete callbacks.

This processing is done automatically when parsing elements from XML. When building elements manually, you must call the structure's finishElement method when done to arrange for these methods being called; to make sure this happens, you usually want to construct Structures using base.makeStruct.

If you override these methods, you (almost) always want to call the corresponding superclasses' methods using super().methname([ctx]). Structures in DaCHS sometimes use multiple inheritance, and hence there's really no alternative to using super here. To make sure this works as expected, any (python) mixin for structures must inherit from base.StructCallbacks.

The user.docgen module makes documentation out of these structures. There are several catches. One of the more striking is that element names in the entire DaCHS code must be unique, since docgen generates section heading from those names and actually checks that these headings are unique; hence, only one (essentially randomly selected) of two identically-named elements would be documented, and parent links would both point there.

Since there are cases when that limitation is a real pain (e.g., the publish element of services and data), there's a workaround: you can set a docName_ class attribute on a structure that contains the name used for the documentation. See rscdef.common.Registration for an example.

Metadata

"Open" metadata (as opposed to the attributes of columns and the like) is kept in a meta_ structure added by base.meta.MetaMixin. You should probably not access that attribute directly if at all possible since the current implementation is incredibly messy and liable to change.

For this kind of metadata, a simple inheritance exists. MetaMixins have a setMetaParent method that declares another structure as the current's meta parent. Any request for metadata that cannot be satisfied from self will then be propagated up to this parent (unless propagation is suppressed). Usually, parents will call their children's setMetaParent methods.

The metdata is organized in a tree with MetaItem``s as nodes.  Each MetaItem contains one or more children that are instances ``MetaValue (or more specialized classes). A MetaValue in turn can have more MetaItem children.

Memoization

The base.caches module should be the central point for all kinds of memoization/caching tasks; in particular, if you use base.caches, your caches will automatically be cleared on dachs serve reload. To keep dependencies and risks of recursive imports low, it is the providing modules' responsibility to register caching functions. The idea is that, e.g., rscdesc wants a cache of resource descriptors. Therefore, it says:

base.caches.makeCache("getRD", getRD)

Clients then say:

base.caches.getRD(id).

This mechanism for now is restricted to items that come with a unique id (the argument). It would be easy to extend this to multiple-argument functions, but I don't think that's a good idea -- the "identities" of the cached objects should be kept simple.

No provision is made to prevent accidental overwriting of function names.

And, of course, individual functions can do functools.lru_cache-ing to their heart's delight but should keep in mind that dachs serve reload will not clear this.

Profiling

If you want to profile server actions, try a script like this:

"""
Make a profile of server responses.

Call as

trial --profile createProfile.py
"""

import sys

from gavo import api
from gavo.web import dispatcher

sys.path.append("/home/msdemlei/gavo/trunk/tests")

import trialhelpers


class ProfileThis(trialhelpers.RenderTest):
  renderer = dispatcher.ArchiveService()

  def testOneService(self):
    self.assertGETHasStrings("/ppmx/res/ppmx/scs/form",
      {"hscs_pos": "12 2", "hscs_sr": "20.0"},
      ["PPMX"])

After running, you can use pstats on the file profile.data.

To profile actually running DaCHS operations, use the --profile-to <profile file> option of the dachs program. For the server, you must make sure in cleanly exists in order to have meaningful stats. Do this by accessing /test/exit on a debug server.

Debugging

Just insert lines like:

import pdb;pdb.Pdb(nosigint=True).set_trace()

whereever required to have python dump you into the debugger and let you look around, single-step, etc.

When you want to inspect what's going on within the server, in particular when something only manifests itself after a long time, you may want to have a look at twisted's manhole; quite a bit easier, however, is to use the debug/q rd that you can get from http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/debug and adapt it to your needs.

The idea here is that within q.rd#1 you create customDFs or customRFs exposing what you're interested in. You can then use those in res/page1.html. You can edit both files "live", they will both be reloaded as necessary.

if MEM_DEBUG:
        from gavo.utils import codetricks
        import gc
        gr = gc.get_referrers
        if hasattr(base, "getNewStructs"):
                ns = base.getNewStructs()
                print ">>>>>> new structs:", len(ns)

    if len(ns)==147:
            from gavo.helpers import testtricks
            ob = ns[0]
            del ns
testtricks.debugReferenceChain(ob)

Delimited SQL identifiers

Although it may look like it, we do not really support delimited identifiers (DIs) as column names (and not at all as table names). I happen to regard them as an SQL misfeature and really only want to keep them out of my software.

However, TAP forces me to deal with them at least superficially. That means that using them elsewhere will lead to lots of mysterious error messages from inside of DaCHS's bowels. There still should not be any remote exploits possible when using them.

Here's the deal on them:

They are represented as utils.misctricks.QuotedName objects. These QuotedNames have some methods to control the impact the partial support for delimited identifiers has on the rest of the software. In particular, when you stringify them, they result in string ready for inclusion into SQL (i.e., hopefully properly escaped). The hash to the name, i.e., there are no implied quotes, and, unfortunately, hash(di)!=hash(str(di)).

The one real painful thing is the representation of result rows with DIs -- I did not want to have lots of these ugly QuotedNames in the result rows, so they end up as SQL-escaped strings when used as keys. This is extra sad since in this way for a DI column foo, rec[QName("foo")] raises a KeyError. To work around this, fields have a key attribute, and rec[f.key] should never bomb.

Grammars

Grammars are DaCHS' means of turning some external data to rowdicts, i.e., dictionaries that map grammar keys to values that are usually strings. They are fed to rowmakers to come up with rows suitable for ingestion (or formatting).

A grammar consists of a Grammar object, which is a structure inheriting from grammars.Grammar. It contains all the "configuration" (e.g., rules). Grammars have a parse method receiving some kind of source token (typically, a file name). You will normally not need to override it.

The real action happens in the row iterator, which is declared in the rowIterator class attribute of the grammar. Row iterators should inherit from grammars.RowIterator.

TODO: yieldsTyped, rowfilters, sourceFields, targetData

Do not import modules from the grammars subpackage directly. Instead, use rscdef.getGrammar with the name of the grammar you want. If you define a new grammar, add a line in rscdef.builtingrammars.grammarRegistry. To inspect what grammars are available, consult the keys from rscdef.grammarRegistry.

Procedures

To embed actual (python) code into RDs, you should use the infrastructure given in rscdef.procdef. It basically leads up to ProcApp, which is what's usually embedded in RDs.

ProcApp inherits from ProcDef, a procedure definition. Such a definition gives some (python) code that is executed when the procedure is applied. To set up the execution environment of this code, there's the definition's setup child.

The setup contains code and parameters. The code is executed to set up the namespace that the procedure will run in; it is thus executed once -- at construction -- per procedure. The parameters allow configuration of the procedure. This is the place to do relatively expensive operations like I/O or imports.

For example, //procs#resolveObject creates the resolver in its setup code; this happens only once per creation of the embedding RD:

<procDef type="apply" id="resolveObject">
  <setup>
    <par key="ignoreUnknowns">True</par>
    <par key="identifier" late="True"/>
    <code>
      from gavo.protocols import simbadinterface
      resolver = simbadinterface.Sesame(saveNew=True)
    </code>
  </setup>
  <doc>...</doc>
   <code>
    ra, dec = None, None
    try:
      ra, dec = resolver.getPositionFor(identifier)
    except KeyError:
      if not ignoreUnknowns:
        raise base.Error("resolveObject could not resolve object"
          " %s."%identifier)
    vars["simbadAlpha"] = ra
    vars["simbadDelta"] = dec
  </code>
</procDef>

The setup definition introduced two parameters. One is ignoreUnknowns, which is "immediate" and just lets the code see a name ignoreUnknowns. As with all par elements, the content of the element is a python expression providing a default.

The other parameter, identifier, is a "late" identifier. This means that it is evaluated on each application of the procedure, much like a function argument. These are just translated into assignments at the top of the function body, which means that everything available in the procedure code is available; e.g., for rowmaker procedures (i.e., type="apply"), you can access vars here.

Taken together, late and immediate par allow for all kinds of configuration of procedures. This is particularly convenient together with macros.

To actually execute the code, you need some kind of procedure application. These always inherit from procdef.ProcApp and add bindings. The bind element lets you give python expressions for all names defined using par in the setup child of the ProcDef given in the procDef attribute. You can also define just a procedure application without a procDef by giving setup and code.

Procedure application have "types" -- these give where they can be used. In particular, the type determines the signature of the python callable that the procedure application is compiled into. procdef.ProcApp has no type, and thus is "abstract"; it should never be a child factory of any StructAttribute.

Instead, inherit from it and give

• name_ -- the element name, as always in structures. This is "apply" for rowmaker applys, "rowfilter" for grammar rowfilters, etc
• formalArgs -- a python argument list that gives the arguments of the callable a ProcApp of this type is compiled into. Thus, this defines the signature.
• requiredType -- a type name that specifies what kind of ProcDef the application will accept. This will in general be the same as name_. None would mean accept all, which probably is useless.

So, all you need to do to define a new sort of ProcApp is write something like:

class EmbeddedIterator(rscdef.ProcApp):
  name_ = "iterator"
  formalArgs = "self"

(of course, here, documentation as to what the code is supposed to do is particularly important, so don't leave out the docstring when actually doing anything.

Then, you could have:

_iterator = base.StructAttribute("iterator", default=base.Undefined,
 childFactory=EmbeddedIterator,
 description="Code yielding row dictionaries", copyable=True)

in some structure. To produce something you can execute, then say:

theIterator = self.iterator.compile()
for row in theIterator(self):
  print row

or somesuch.

ADQL User Defined Functions

ADQL user defined functions currently all live in adql.ufunctions, and their tests are centralised in ufunctest. We should probably have a canonical place where individual operators can add them from reliably.

To write a UDF, write a function matching the signature explained in adql.ufunctions.userFunction and apply that decorator. For the names, you must use something starting with either gavo_ or ivo_ as per ADQL 2.1 (where you can only use ivo_ if you've got someone else also implementing it).

If you can, produce nodes and raise a ReplaceNode (as in, e.g., gavo_transform). Most existing UDFs admittedly return strings, which leads to lousy tree annotation and makes it impossible to later morph the result – but I'll give you it's much simpler to write functions returning fixed strings. If you do that, be sure to never include unparsed literals; remember: args is quite strongly under user control. Hence, make it a habit of always writing notes.flatten(args[n]) whenever you use args. This also has the advantage that expressions in the arguments of your UDFs will be flattened, too.

UDFs will typically start their existence as gavo_whatever. If, as is rather common, this later becomes an interoperable UDF (i.e., listed in the UDF catalogue), this should then become ivo_whatever. However, existing queries using the gavo-prefixed forms should keep working. To make that happen, do:

• pass the ivo_ version as userFunction's first argument
• add the gavo_ version in the additionalNames keyword argument.

Users will no longer see the gavo_ version in the capabilities, and hence TOPCAT will mark a syntax error if folks type the old name. I'd say that's ok. People should change their queries after all.

There's an example for that migration technique in ufunctions._ivo_histogram. In that case, there is the additional complication that that's just handing through to a SQL function created in //adql; this is still called gavo_histogram, and to unify the names this is using a custom node (which is convenient here because we want to fiddle with the node's annotation anyway).

Schema updates

If you need to change the on-disk schema, you must provide an updater in gavo.user.upgrade. See the docstring on Upgrade on what you can and should do in there, and read on.

The basic idea is that each upgrade step is written as a class inheriting Upgrader. Its version attribute must be the value of upgrader.CURRENT_SCHEMAVERSION (defined near the top) when you start working. After you have defined your upgrader, increase CURRENT_SCHEMAVERSION by one.

The upgrader has attributes and class methods with magic names; if these are string-values, they are directly executed, if they are methods, they are called with a connection argument. Do not use any other connections in upgraders or you'll break the atomicity of the upgrades.

The magic names can either be u_<nn>_<name> or s_<nn>_<name>. use nn to determine the action sequence. The difference between u and s is that when upgrading over multiple versions, all s methods are being executed before the first u method. The idea is that schema-changing changes should be in s methods, content updates and similar should be in u methods.

When defining upgrades, it pays to make sure upgraders don't break if what they're doing has been done already; that reduces the requirements on upgrade atomicity and prevents upgrade crashes (which are always ugly) when people do odd things. For this, use the relationExists(tableName, connection) and getColumnNamesFor(rdTableId), connection) methods to figure things out. There's also _updateTAP_SCHEMA(connection), which you should use on anything influencing TAP_SCHEMA (which might include, say, changes in column serialisation).

At Heidelberg, once an upgrade is defined, test the upgrader using:

testgavo upgrade

The effects should be visible in the dachstest database. To be able to roll back changes effected by the upgrade, you may want to backup the cluster first. Markus has a script backup_postgres for that.

If you follow the rules, upgrade should be atomic, i.e., either the upgrade succeeds or the database is untouched, letting operators downgrade and continue operations until a problem is figured out. To selectively re-run upgraders (and they should be idempotent), use dachs upgrade's --force-dbversion option.

XML Schema updates

According to the new schema versioning policy of the VO, for minor updates, the target namespace of XSD does not change any more. Still, the file names are versioned upstream.

DaCHS, in general, only knows one version of each schema. Therefore, we remove the version names, and when there's a new schema, you just overwrite the corresponding schema.

There are a few exceptions; in particular, because several minor versions of VOTable have been out there and in common use, we keep the schemas for VOTable 1.1 and 1.2 around with their own custom prefixes.

In case you actually need a new schema file, this is what you need to do:

• install the new file in gavo/resources/schemata and svn add it
• in gavo/helpers/testtricks.py, locate VO_SCHEMATA and add your new file name.
• add a registerPrefix call for your new schema, allocating a new prefix for it while doing so. See registry/model.py for a large selection of such registerPrefix calls; but do the registration whereever the schema actually belongs.
• write a unit test exercising your schema.

Javascript

While it's our goal to let people operate the web-based part of DaCHS without javascript enabled, it's ok if fancier functionality depends on javascript.

After some hesitation, we decided to use the jquery javascript library (we used to have MochiKit but left that when we wanted nice in-browser plotting; so, if you still see MochiKit somewhere, please disregard). We also include some of jquery-ui.

We keep all javascript in "full" source form (in resources/web/js). DaCHS performs on-the-fly minimisaton (unless [web]jsSource is False).

For development of that, it's much more convenient if the stuff that gets served out is in source. To enable that, set [web]jsSource to true. This needs actual code support; right now this only works for files served out in commonhead. You need to restart the server for the setting to take effect.

<script type="text/html" id="tmpl_authorHeader">
  <li>
    <a class="arrow-e"
      onclick="toggleAuthorResources(this)" name="$author"/>
      $author ($nummatch)
  </li>
</script>

renderTemplate("tmpl_authorHeader", {
  author: 'Thor, A. U',
  nummatch: 8})

<script type="text/html" id="tabbar_store">
<ul id="tabset_tabs">
  <li class="selected"><a name="by-title">By Title</a></li>
  <li><a name="by-subject">By Subject</a></li>
  <li><a name="by-author">By Author</a></li>
</ul>
</script>

<p id="tab_placeholder" style="border:2pt dashed #bb9999;padding: 0.5ex">
  Enable Javascript for more choices.</p>

$(document).ready(function() {
  $("#tab_placeholder").replaceWith(
    $(document.getElementById("tabbar_store").innerHTML));
  $("#tabset_tabs li").bind("click", makeTabCallback({
    'by-subject': func1,
    'by-author': func2,
    'by-title': func3,
  }));
}

position: relative;
background-color: #EAEBEE;
margin-top: 0px;
min-height:70ex;

sudo apt install libjs-jquery libjs-jquery-flot
python3 ../../../web/ifpages.py

Stuff in gavo.imp

gavo.imp has some external dependencies of DaCHS. Shortly after release 1.0, many were dropped in favour of their packaged/native counterparts (argparse, pyparsing...). What's currently left is:

• rjsmin -- Debian packaged, and the Debian version will be picked up automatically if installed (so, this should not go into the Debian package)

Different Database Backends

A request we get fairly regularly is to make DaCHS work with database engines other than Postgres, with MySQL and Oracle being the most popular alternatives for external requests and SQLite something we personally would like to see for ease of deployment.

The short answer to all this: It's tricky. You might get away with using foreign data wrappers in some cases; a group at Paris Observatory reports fairly good results with them.

Here's the longer answer: DaCHS does a lot of inspection of the database, while at the same time worrying about different access levels, reconnection on database restarts, and similar; it also creates extension types. We are not aware of any abstraction layer that would let us keep all this code generic, and that's why we let DaCHS slide into a fairly deep entanglement with psycopg2 and Postgres.

Seeing such an entanglement reduces the scope of DaCHS, we'd certainly help pulling it out of the entanglement. We probably won't do it ourselves. Here's a list of things that would need to be done for un-entanglement, that's probably somewhat incomplete and also contains some project mines (innocuous-looking things that blow up into a lot of refactoring once you step on them):

1. separate what's specific to postgresql+psycopg2 from sqlsupport, put that into a module (backend_postgres, say), devise some sort of dispatcher to backends, and have, to work out things, a second backend, that would then contain different implementations for tableExists, indexExists, and so on. Actually, throwing out some cruft from sqlsupport that should have gone ages ago would be a good thing, too.
2. figure out what other hidden dependencies exist; the most worrisome part probably is the extension types DaCHS uses and registers as well as the pgSphere interface; this is built into typesystems and used left and right. If there's no way to hide DB-specific differences, there'll have to be some major redesign. Also, DaCHS implicitely assumes TEXT in the database is cheap. If that's not true of a DB (and I think in Oracle TEXT can't be properly indexed) and you'll want much more VARCHARs and similar, minor adjustments might be in order.
3. The ADQL translator would need to get another "morpher" (the thing that turns ADQL parse trees into the language of the backend database) That's already forseen, but figuring out how to enable maximum reuse of code between the different morphers might take some thought. Also, again the question of spherical geometry in the backend will have to be looked at.
4. Some mixins directly depend on postgres features (//scs#q3cindex is an obvious example). I believe it'd be ok to say "well, don't use these on non-Postgres", and we'd provide similar things for the other DBs. But that would make RDs non-portable, which I don't like too much either.
5. The C boosters generate material for Postgres binary copy. Obviously, one would need to figure out the analogon on other databases (which may not be well-documented; I had to check the Postgres source for some details, too) and then split up boosterskel.c into generic and postgres-specific parts. Or there'd be no support for C boosters on different databases, which might not be unreasonable, either.

Writing Documentation

Documentation on DaCHS is maintained in ReStructuredText format with some minor extensions (see below). While there's documentation in the tarball and the main SVN, in order to encourage external contributions (including, but not restricted to, typo fixes and the like) the main copy now is at https://github.com/gavodachs/dachs-doc.git.

When authoring, you can use some extra RST features (the price: stock rst2pdf and friends don't work properly; use dachs gendoc latex or dachs gendoc html, or a special sphinx configuration). These include:

• dachsref: Give a reference documentation heading (as in The //obscore#publishSIAP Mixin), and you'll get a link there.
• dachsdoc: Works like normal links, including explicit targets, but it prepends the root URL of the DaCHS documentation
• bibcode: Adds an ADS link to a bibcode

Random Stuff

strace dachs imp -h 2>&1 | grep 'open' | grep -v ENOENT | grep -v "pyc" | sed -e 's/.*"\(.*\)".*/\1/'

import matplotlib
matplotlib.use("Agg")
from matplotlib import pyplot