gavo.utils.codetricks module

Functions dealing with compilation and introspection of python and external code.

class gavo.utils.codetricks.AllEncompassingSet

Bases: set

a set that contains everything.

Ok, so this doesn’t exist. Yes, I’ve read my Russell. You see, this is a restricted hack for a reason. And even the docstring is contradictory.

Sort-of. This now works for intersection and containing. Should this reject union? Also, unfortunately this only works as a left operand; I don’t see how to override whatever set does with this as a right operand.

>>> s = AllEncompassingSet()
>>> s & set([1,2])
{1, 2}
>>> "gooble" in s
True
>>> s in s
True
>>> s not in s
False

intersection(other: Set) → Set

Return the intersection of two sets as a new set.

(i.e. all elements that are in both sets.)

class gavo.utils.codetricks.CachedGetter(getter: Callable, *args, **kwargs)

Bases: object

A cache for a callable.

This is basically memoization, except that these are supposed to be singletons; CachedGetters should be used where the construction of a resource (e.g., a grammar) should be deferred until it is actually needed to save on startup times.

The resource is created on the first call, all further calls just return references to the original object.

You can also leave out the getter argument and add an argumentless method impl computing the value to cache.

Using a CachedGetter also serializes generation, so you can also use it when getter isn’t thread-safe.

At construction, you can pass a f(thing) -> bool in an isAlive keyword argument. If you do, the function will be called with the cache before the cache is being returned. If it returns false, the resource is re-made (no concurrency control is enforced here).

class gavo.utils.codetricks.DeferredImport(moduleName: str, loadingCode: str)

Bases: object

A trivial deferred module loader.

Use this to delay the actual import of a module until it’s actually needed.

It is constructed with a module name (that will be inserted into the calling module’s globals() as a side effect) and some literal code that, when executed in the caller’s global namespace, actually imports the module, for instance:

utils.DeferredImport("wcs", "from astropy import wcs")

As a service for static code checking, you’ll usually want to repeat the module name, though:

wcs = utils.DeferredImport("wcs", "from astropy import wcs")

loadedModule = None

class gavo.utils.codetricks.EqualingRE(pattern: str)

Bases: object

A value that compares equal based on RE matches.

This is a helper mainly for GetHasXPathsTests. Use an instance of this class to check against an RE rather than a plain string.

>>> EqualingRE("(ab)+") == "ababab"
True
>>> EqualingRE("(ab)+$") == "ababa"
False
>>> EqualingRE("(ab)+$") != "ababa$"
True
>>> "ababa" == EqualingRE("(ab)+$")
False

class gavo.utils.codetricks.IdManagerMixin

Bases: object

A mixin for objects requiring unique IDs.

The primary use case is XML generation, where you want stable IDs for objects, but IDs must be unique over an entire XML file.

The IdManagerMixin provides some methods for doing that:

• makeIdFor(object) – returns an id for object, or None if makeIdFor has

  already been called for that object (i.e., it presumably already is in the document).

• getIdFor(object) – returns an id for object if makeIdFor has already

  been called before. Otherwise, a NotFoundError is raised

• getOrMakeIdFor(object) – returns an id for object; if object has

  been seen before, it’s the same id as before. Identity is by equality for purposes of dictionaries.

• getForId(id) – returns the object belonging to an id that has

  been handed out before. Raises a NotFoundError for unknown ids.

• cloneFrom(other) – overwrites the self’s id management dictionaries

  with those from other. You want this if two id managers must work on the same document.

cloneFrom(other)

takes the id management dictionaries from other.

getForId(id: str) → Hashable

returns the object associated with id.

This will raise a KeyError for unknown ids.

getIdFor(ob: Hashable) → str

returns the id for ob if it is known.

If ob does not yet have an id, the method raises a NotFoundError.

getOrMakeIdFor(ob: Hashable, suggestion: Optional[str] = None) → str

returns the id of the (hashable) ob or creates one if it does not already have one.

This is the function you would normally use with the id manager.

makeIdFor(ob: Hashable, suggestion: Optional[str] = None) → str

returns a new id for ob (which must be Hashable).

suggestion can be a string giving what id ought to be; the function will change it as necessary to make it a usable and unique id.

If the object already has an id, the method will raise a ValueError (since DaCHS 2.8.1; returned None before).

class gavo.utils.codetricks.Infimum(*args, **kwargs)

Bases: _Comparer

is a class smaller than anything.

This will only work as the first operand.

>>> Infimum<-2333
True
>>> Infimum<""
True
>>> -2333<Infimum
False

class gavo.utils.codetricks.NS(**kwargs)

Bases: object

An object that has its kwargs as attributes.

class gavo.utils.codetricks.NocaseString

Bases: str

A string that compares case-insensitively.

This is my way to work around the crazy requirement that all kinds of VO protocol parameters need to be case-insensitive. This won’t work with dictionaries. It will work with cgi.FieldStorage, though, because it does a linear search.

Normal DaCHS code doesn’t need this because of various hacks in contextgrammar and elsewhere. If you’re touching request.args manually, you’ll have to look at this, though.

Case insensitivity is evil. Let’s get rid of it and then get rid of this nasty mess. >>> NocaseString(“aBc”)==”abc” True >>> “aBc”==NocaseString(“abc”) True >>> NocaseString(“axc”)==”abc” False >>> NocaseString(“axc”)!=”abc” True >>> NocaseString(“axc”)!=NocaseString(“abc”) True

class gavo.utils.codetricks.NullObject

Bases: object

A Null object, i.e. one that accepts any method call whatsoever.

This mainly here for use in scaffolding.

class gavo.utils.codetricks.Supremum(*args, **kwargs)

Bases: _Comparer

is a class larger than anything.

This will only work as the first operand.

>>> Supremum>1e300
True
>>> Supremum>""
True
>>> Supremum>None
True
>>> Supremum>Supremum
True

gavo.utils.codetricks.addDefaults(dataDict: Dict, defaultDict: Dict) → None

adds key-value pairs from defaultDict to dataDict if the key is missing in dataDict.

gavo.utils.codetricks.buildClassResolver(baseClass: type, objects: ~typing.List[~typing.Any], instances: bool = False, key: ~typing.Callable[[~typing.Hashable], ~typing.Optional[str]] = <function <lambda>>, default=None) → Callable[[str], Any]

returns a function resolving classes deriving from baseClass in the sequence objects by their names.

This is used to build registries of, for instance, Macros and RowProcessors. The classes in question have to have a name attribute.

objects would usually be something like globals().values()

If instances is True the function will return instances instead of classes.

key is a function taking an object and returning the key under which you will later access it. If this function returns None, the object will not be entered into the registry.

class gavo.utils.codetricks.bytelist(iterable=(), /)

Bases: list

gavo.utils.codetricks.compileFunction(src: str, funcName: str, useGlobals: Optional[Dict[str, Any]] = None, debug: bool = False, uniqueName: Optional[str] = None)

runs src through exec and returns the funcName from the resulting namespace.

This takes care to preserve src in the line cache so it is available in tracebacks or in the debugger.

useGlobals can be a namespace; if not passed, the globals of the utils.codestricks is used.

If debug=True is passed in, additional code is produced to give halfway useful tracebacks.

uniqueName, if given, is the identifier for the code. If passed in, no automatic cleanup for the linecache is done under the assumption that reloads (or whatever) will overwrite the linecache. Otherwise code is autonumbered (which is not really desirable for user-provided code, as they won’t know where their failing code comes from).

This is typically used to define functions, like this:

>>> resFunc = compileFunction("def f(x): print(x)", "f")
>>> resFunc(1); resFunc("abc")
1
abc

class gavo.utils.codetricks.complexlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.devnull() → TextIO

returns a (string-) writable /dev/null.

This always returns the same object, and to placate resource warnings, the file will be closed before exiting the program; the close method of the returned thing is a no-op.

gavo.utils.codetricks.document(origFun: Any)

is a decorator that adds a “buildDocsForThis” attribute to its argument.

This attribute is evaluated by documentation generators.

gavo.utils.codetricks.ensureExpression(expr: str, errName: str = 'unknown') → None

raises a LiteralParserError if expr is not a parseable python expression.

>>> ensureExpression("4+4")
>>> ensureExpression("'/'.join([str(x) for x in range(10)])")
>>> ensureExpression("junk")

class gavo.utils.codetricks.floatlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.forgetMemoized(ob: object) → None

clears things memoizeOn-ed on ob or @utils.memoize-ed.

This is sometimes necessary to let the garbage collector free ob, e.g., when closures have been memoized.

gavo.utils.codetricks.getTracebackAsString() → str

gavo.utils.codetricks.identity(x: Any) → Any

gavo.utils.codetricks.importModule(modName: str) → module

imports a module from the module path.

Use this to programmatically import “normal” modules, e.g., dc-internal ones. It uses python’s standard import mechanism and returns the module object.

We’re using exec and python’s normal import, so the semantics should be identical to saying import modName except that the caller’s namespace is not changed.

The function returns the imported module.

gavo.utils.codetricks.in_dir(destDir: Union[str, Path]) → Generator

executes the controlled block within destDir and then returns to the previous directory.

Think “within dir”. Haha.

gavo.utils.codetricks.intToFunnyWord(anInt: int, translation=b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,z./aeiousmnth:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`wblpgdghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff') → str

returns a sometimes funny (but unique) byte string from an arbitrary integer.

>>> intToFunnyWord(3829901938)
'guunwwni'

class gavo.utils.codetricks.intlist(iterable=(), /)

Bases: list

gavo.utils.codetricks.iterConsecutivePairs(sequence: Sequence)

yields pairs of consecutive items from sequence.

If the last item cannot be paired, it is dropped.

>>> list(iterConsecutivePairs(range(6)))
[(0, 1), (2, 3), (4, 5)]
>>> list(iterConsecutivePairs(range(5)))
[(0, 1), (2, 3)]

gavo.utils.codetricks.iterDerivedClasses(baseClass: type, objects: List[Any])

iterates over all subclasses of baseClass in the sequence objects.

gavo.utils.codetricks.iterDerivedObjects(baseClass: type, objects: List[Any])

iterates over all instances of baseClass in the sequence objects.

gavo.utils.codetricks.iterRanges(separators: Sequence[T]) → Generator[Tuple[T, T], None, None]

yields (left, right) pairs for a sequence of separating indexes.

This is when you want to partition a sequence based on cut points.

>>> list(iterRanges(range(6)))
[(0, 1), (1, 2), (2, 3), (3, 4), (4, 5)]

gavo.utils.codetricks.loadInternalObject(relativeName: str, objectName: str) → Any

gets a name from an internal module.

relativeName is the python module path (not including “gavo.”), objectName the name of something within the module.

This is used for “manual” registries (grammars, cores,…).

gavo.utils.codetricks.loadPythonModule(fqName: Union[str, Path], relativeTo: Optional[Union[str, Path]] = None) → Tuple[module, Any]

imports fqName and returns the (module, spec).

Do not use this function to import DC-internal modules; this may mess up singletons since you could bypass python’s mechanisms to prevent multiple imports of the same module.

fqName is a fully qualified path to the module without the .py, unless relativeTo is given, in which case it is interpreted as a relative path. This for letting modules in resdir/res import each other by saying:

mod, _ = api.loadPythonModule("foo", relativeTo=__file__)

The python path is temporarily amended with the path part of the source module.

If the module is in /var/gavo/inputs/foo/bar/mod.py, Python will know the module as foo_bar_mod (the last two path components are always added). This is to keep Python from using the module when someone writes import mod.

gavo.utils.codetricks.memoizeOn(onObject: object, generatingObject: object, generatingFunction: Callable, *args) → Any

memoizes the result of generatingFunction on onObject and returns it.

This is for caching things that adapt to onObjects; see procdefs and rowmakers for examples why this is useful.

args is passed to generatingFunction; generatingObject is only needed for identification.

gavo.utils.codetricks.memoized(origFun: Callable)

a trivial memoizing decorator.

This is a legacy wrapper for functools.lru_cache. Don’t use in new code

gavo.utils.codetricks.printFrames() → None

prints a compact list of frames.

This is an aid for printf debugging.

gavo.utils.codetricks.sandbox(tmpdir: Optional[Union[str, Path]] = None, debug: Optional[bool] = False, extractfunc: Optional[Callable[[Union[str, Path]], None]] = None)

sets up and tears down a sandbox directory within tmpdir.

This is is a context manager. The object returned is the original path (which allows you to copy stuff from there). The working directory is the sandbox created while in the controlled block.

If tmpdir is None, the system default is used (usually /tmp), rather than dachs’ tmpdir. So, you will usually want to call this as sandbox(base.getConfig(“tempDir”))

This is obviously not thread-safe – you’ll not usually want to run this in the main server process. Better fork before running this.

You can pass in a function extractfunc(owd) that is executed in the sandbox just before teardown. It receives the original working directory and can, e.g., move files there from the sandbox.

gavo.utils.codetricks.silence(errToo: bool = False) → Generator

a context manager to temporarily redirect stdout to /dev/null.

This is used to shut up some versions of pyparsing and pyfits that insist on spewing stuff to stdout from deep within in relatively normal situations.

Note that this will acquire a lock while things are silenced; this means that silenced things cannot run concurrently.

gavo.utils.codetricks.stealVar(varName: str) → Any

returns the first local variable called varName in the frame stack above my caller.

This is obviously abominable. This is only used within the DC code where the author deemed the specification ugly. Ah. Almost.

If no variable with varName is found anywhere in the stack, this raises a ValueError.