gavo.utils.codetricks module

Functions dealing with compilation and introspection of python and external code.

class gavo.utils.codetricks.AllEncompassingSet

Bases: set

a set that contains everything.

Ok, so this doesn’t exist. Yes, I’ve read my Russell. You see, this is a restricted hack for a reason. And even the docstring is contradictory.

Sort-of. This now works for intersection and containing. Should this reject union? Also, unfortunately this only works as a left operand; I don’t see how to override whatever set does with this as a right operand.

>>> s = AllEncompassingSet()
>>> s & set([1,2])
{1, 2}
>>> "gooble" in s
True
>>> s in s
True
>>> s not in s
False

intersection(other)

Return self&value.

class gavo.utils.codetricks.CachedGetter(getter, *args, **kwargs)

Bases: object

A cache for a callable.

This is basically memoization, except that these are supposed to be singletons; CachedGetters should be used where the construction of a resource (e.g., a grammar) should be deferred until it is actually needed to save on startup times.

The resource is created on the first call, all further calls just return references to the original object.

You can also leave out the getter argument and add an argumentless method impl computing the value to cache.

Using a CachedGetter also serializes generation, so you can also use it when getter isn’t thread-safe.

At construction, you can pass a f(thing) -> bool in an isAlive keyword argument. If you do, the function will be called with the cache before the cache is being returned. If it returns false, the resource is re-made (no concurrency control is enforced here).

class gavo.utils.codetricks.CachedResource(arg)

Bases: object

is like CachedGetter but with a built-in getter.

Here, you define your class and have a class method impl returning what you want.

cache = None

class gavo.utils.codetricks.DeferredImport(moduleName, loadingCode)

Bases: object

A trivial deferred module loader.

Use this to delay the actual import of a module until it’s actually needed.

It is constructed with a module name (that will be inserted into the calling module’s globals() as a side effect) and some literal code that, when executed in the caller’s global namespace, actually imports the module, for instance:

utils.DeferredImport("wcs", "from astropy import wcs")

As a service for static code checking, you’ll usually want to repeat the module name, though:

wcs = utils.DeferredImport(“wcs”, “from astropy import wcs”)

loadedModule = None

class gavo.utils.codetricks.EqualingRE(pattern)

Bases: object

A value that compares equal based on RE matches.

This is a helper mainly for GetHasXPathsTests. Use an instance of this class to check against an RE rather than a plain string.

>>> EqualingRE("(ab)+") == "ababab"
True
>>> EqualingRE("(ab)+$") == "ababa"
False
>>> EqualingRE("(ab)+$") != "ababa$"
True
>>> "ababa" == EqualingRE("(ab)+$")
False

class gavo.utils.codetricks.IdManagerMixin

Bases: object

A mixin for objects requiring unique IDs.

The primary use case is XML generation, where you want stable IDs for objects, but IDs must be unique over an entire XML file.

The IdManagerMixin provides some methods for doing that:

• makeIdFor(object) – returns an id for object, or None if makeIdFor has

  already been called for that object (i.e., it presumably already is in the document).

• getIdFor(object) – returns an id for object if makeIdFor has already

  been called before. Otherwise, a NotFoundError is raised

• getOrMakeIdFor(object) – returns an id for object; if object has

  been seen before, it’s the same id as before. Identity is by equality for purposes of dictionaries.

• getForId(id) – returns the object belonging to an id that has

  been handed out before. Raises a NotFoundError for unknown ids.

• cloneFrom(other) – overwrites the self’s id management dictionaries

  with those from other. You want this if two id managers must work on the same document.

cloneFrom(other)

takes the id management dictionaries from other.

getForId(id)

getIdFor(ob)

getOrMakeIdFor(ob, suggestion=None)

makeIdFor(ob, suggestion=None)

class gavo.utils.codetricks.Infimum(*args, **kwargs)

Bases: gavo.utils.codetricks._Comparer

is a class smaller than anything.

This will only work as the first operand.

>>> Infimum<-2333
True
>>> Infimum<""
True
>>> -2333<Infimum
False

class gavo.utils.codetricks.NocaseString

Bases: str

A string that compares case-insensitively.

This is my way to work around the crazy requirement that all kinds of VO protocol parameters need to be case-insensitive. This won’t work with dictionaries. It will work with cgi.FieldStorage, though, because it does a linear search.

Normal DaCHS code doesn’t need this because of various hacks in contextgrammar and elsewhere. If you’re touching request.fields manually, you’ll have to look at this, though.

Case insensitivity is evil. Let’s get rid of it and then get rid of this nasty mess. >>> NocaseString(“aBc”)==”abc” True >>> “aBc”==NocaseString(“abc”) True >>> NocaseString(“axc”)==”abc” False >>> NocaseString(“axc”)!=”abc” True >>> NocaseString(“axc”)!=NocaseString(“abc”) True

class gavo.utils.codetricks.NullObject

Bases: object

A Null object, i.e. one that accepts any method call whatsoever.

This mainly here for use in scaffolding.

class gavo.utils.codetricks.Supremum(*args, **kwargs)

Bases: gavo.utils.codetricks._Comparer

is a class larger than anything.

This will only work as the first operand.

>>> Supremum>1e300
True
>>> Supremum>""
True
>>> Supremum>None
True
>>> Supremum>Supremum
True

gavo.utils.codetricks.addDefaults(dataDict, defaultDict)

adds key-value pairs from defaultDict to dataDict if the key is missing in dataDict.

gavo.utils.codetricks.buildClassResolver(baseClass, objects, instances=False, key=<function <lambda>>, default=None)

returns a function resolving classes deriving from baseClass in the sequence objects by their names.

This is used to build registries of Macros and RowProcessors. The classes in question have to have a name attribute.

objects would usually be something like globals().values()

If instances is True the function will return instances instead of classes.

key is a function taking an object and returning the key under which you will later access it. If this function returns None, the object will not be entered into the registry.

class gavo.utils.codetricks.bytelist(iterable=(), /)

Bases: list

gavo.utils.codetricks.compileFunction(src, funcName, useGlobals=None, debug=False)

runs src through exec and returns the item funcName from the resulting namespace.

This is typically used to define functions, like this:

>>> resFunc = compileFunction("def f(x): print(x)", "f")
>>> resFunc(1); resFunc("abc")
1
abc

class gavo.utils.codetricks.complexlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.devnull()

returns a (string-) writable /dev/null.

This always returns the same object, and to placate resource warnings, the file will be closed before exiting the programme; the close method of the returned thing is a no-op.

gavo.utils.codetricks.document(origFun)

is a decorator that adds a “buildDocsForThis” attribute to its argument.

This attribute is evaluated by documentation generators.

gavo.utils.codetricks.ensureExpression(expr, errName='unknown')

raises a LiteralParserError if expr is not a parseable python expression.

>>> ensureExpression("4+4")
>>> ensureExpression("'/'.join([str(x) for x in range(10)])")
>>> ensureExpression("junk")

class gavo.utils.codetricks.floatlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.forgetMemoized(ob)

clears things memoizeOn-ed on ob or @utils.memoize-ed.

This is sometimes necessary to let the garbage collector free ob, e.g., when closures have been memoized.

gavo.utils.codetricks.getTracebackAsString()

gavo.utils.codetricks.identity(x)

gavo.utils.codetricks.importModule(modName)

imports a module from the module path.

Use this to programmatically import “normal” modules, e.g., dc-internal ones. It uses python’s standard import mechanism and returns the module object.

We’re using exec and python’s normal import, so the semantics should be identical to saying import modName except that the caller’s namespace is not changed.

The function returns the imported module.

gavo.utils.codetricks.in_dir(destDir)

executes the controlled block within destDir and then returns to the previous directory.

Think “within dir”. Haha.

gavo.utils.codetricks.intToFunnyWord(anInt, translation=b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,z./aeiousmnth:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`wblpgdghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff')

returns a sometimes funny (but unique) byte string from an arbitrary integer.

>>> intToFunnyWord(3829901938)
'guunwwni'

class gavo.utils.codetricks.intlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.iterConsecutivePairs(sequence)

yields pairs of consecutive items from sequence.

If the last item cannot be paired, it is dropped.

>>> list(iterConsecutivePairs(range(6)))
[(0, 1), (2, 3), (4, 5)]
>>> list(iterConsecutivePairs(range(5)))
[(0, 1), (2, 3)]

gavo.utils.codetricks.iterDerivedClasses(baseClass, objects)

iterates over all subclasses of baseClass in the sequence objects.

gavo.utils.codetricks.iterDerivedObjects(baseClass, objects)

iterates over all instances of baseClass in the sequence objects.

gavo.utils.codetricks.iterRanges(separators)

yields (left, right) pairs for a sequence of separators.

>>> list(iterRanges(range(6)))
[(0, 1), (1, 2), (2, 3), (3, 4), (4, 5)]

gavo.utils.codetricks.loadInternalObject(relativeName, objectName)

gets a name from an internal module.

relativeName is the python module path (not including “gavo.”), objectName the name of something within the module.

This is used for “manual” registries (grammars, cores,…).

gavo.utils.codetricks.loadPythonModule(fqName, relativeTo=None)

imports fqName and returns the (module, spec).

Do not use this function to import DC-internal modules; this may mess up singletons since you could bypass python’s mechanisms to prevent multiple imports of the same module.

fqName is a fully qualified path to the module without the .py, unless relativeTo is given, in which case it is interpreted as a relative path. This for letting modules in resdir/res import each other by saying:

mod, _ = api.loadPythonModule("foo", relativeTo=__file__)

The python path is temporarily amended with the path part of the source module.

If the module is in /var/gavo/inputs/foo/bar/mod.py, Python will know the module as foo_bar_mod (the last two path components are always added). This is to keep Python from using the module when someone writes import mod.

gavo.utils.codetricks.memoizeOn(onObject, generatingObject, generatingFunction, *args)

memoizes the result of generatingFunction on onObject.

This is for caching things that adapt to onObjects; see procdefs and rowmakers for examples why this is useful.

gavo.utils.codetricks.memoized(origFun)

a trivial memoizing decorator.

This is a legacy wrapper for functools.lru_cache. Don’t use in new code

gavo.utils.codetricks.printFrames()

prints a compact list of frames.

This is an aid for printf debugging.

gavo.utils.codetricks.runInSandbox(setUp, func, tearDown, *args, **kwargs)

runs func in a temporary (“sandbox”) directory.

func is called with args and kwargs. setUp and tearDown are two functions also called with args and kwargs; in addition, they are passed the path of the tempdir (setUp) or the path of the original directory (teardown) in the first argument.

setUp is called after the directory has been created, but the process is still in the current WD.

tearDown is called before the temp dir is deleted and in this directory. Its return value is the return value of runInSandbox, which is the preferred way of getting data out of the sandbox.

If any of the handlers raise exceptions, the following handlers will not be called. The sandbox will be torn down, though.

This is only present for legacy code. Use the sandbox context manager now.

gavo.utils.codetricks.sandbox(tmpdir=None, debug=False, extractfunc=None)

sets up and tears down a sandbox directory within tmpdir.

This is is a context manager. The object returned is the original path (which allows you to copy stuff from there). The working directory is the sandbox created while in the controlled block.

If tmpdir is None, the system default is used (usually /tmp), rather than dachs’ tmpdir. So, you will usually want to call this as sandbox(base.getConfig(“tempDir”))

This is obviously not thread-safe – you’ll not usually want to run this in the main server process. Better fork before running this.

You can pass in a function extractfunc(owd) that is executed in the sandbox just before teardown. It receives the original working directory and can, e.g., move files there from the sandbox.

gavo.utils.codetricks.silence(errToo=False)

a context manager to temporarily redirect stdout to /dev/null.

This is used to shut up some versions of pyparsing and pyfits that insist on spewing stuff to stdout from deep within in relatively normal situations.

Note that this will acquire a lock while things are silenced; this means that silenced things cannot run concurrently.

gavo.utils.codetricks.stealVar(varName)

returns the first local variable called varName in the frame stack above my caller.

This is obviously abominable. This is only used within the DC code where the author deemed the specification ugly. Ah. Almost.