gavo.helpers.processing module

An abstract processor and some helping code.

Currently, I assume a plain text interface for those. It might be a good idea to use the event mechanism here.

class gavo.helpers.processing.AnetHeaderProcessor(opts, dd)

Bases: HeaderProcessor

A file processor for calibrating FITS frames using astrometry.net.

It might provide calibration for “simple” cases out of the box. You will usually want to override some solver parameters. To do that, define class attributes sp_<parameter name>, where the parameters available are discussed in helpers.anet’s docstring. sp_indices is one thing you will typically need to override.

To use SExtractor rather than anet’s source extractor, override sexControl, to use an object filter (see anet.getWCSFieldsFor), override the objectFilter attribute.

To add additional fields, override _getHeader and call the parent class’ _getHeader method. To change the way astrometry.net is called, override the _solveAnet method (it needs to return some result anet.of getWCSFieldsFor) and call _runAnet with your custom arguments for getWCSFieldsFor.

See processors#astrometry-net for details.

static addOptions(optParser)

commentFilter(value)

returns true if the comment value should be preserved.

You may want to override this.

historyFilter(value)

returns true if the history item value should be preserved.

indexPath = '/home/msdemlei/gavo/astrometry-indexes'

noCopyHeaders = {'bitpix', 'datamax', 'datamin', 'date', 'imageh', 'imagew', 'naxis', 'naxis1', 'naxis2', 'simple'}

objectFilter = None

property solverParameters

sourceExtractorControl = None

exception gavo.helpers.processing.CannotComputeHeader

Bases: Exception

is raised when no FITS header was generated by a HeaderProcessor.

Specifically, this is what gets raised when _getHeader returns None.

class gavo.helpers.processing.FileProcessor(opts, dd)

Bases: object

An abstract base for a source file processor.

In concrete classes, you need to define a process(fName) method receiving a source as returned by the dd (i.e., usually a file name).

You can override the method _createAuxiliaries(dataDesc) to compute things like source catalogues, etc. Thus, you should not need to override the constructor.

These objects are usually constructed thorough api.procmain as discussed in processing.html.

addClassification(fName)

static addOptions(parser)

classify(fName)

getProductKey(srcName)

inputsDir = '/home/msdemlei/gavo/inputs'

iterIdentifiers()

iterates over all identifiers that should be processed.

This is usually the paths of the files to be processed. You can, however, override it to do something else if that fits your problem (example: Previews in SSA use the accref).

iterJobs(nParallel)

executes process() in parallel for all sources and iterates over the results.

We use this rather than multiprocessing’s Pool, as that cannot call methods. I’m working around this here.

printReport(processed, ignored)

printTableSize()

printVerboseReport(processed, ignored)

process(fName)

processAll()

calls the process method of processor for all sources of the data descriptor dd.

class gavo.helpers.processing.HeaderProcessor(opts, dd)

Bases: FileProcessor

A base for processors doing FITS header manipulations.

The processor builds naked FITS headers alongside the actual files, with an added extension .hdr (or whatever is in the headerExt attribute). The presence of a FITS header indicates that a file has been processed. The headers on the actual FITS files are only replaced if necessary.

The basic flow is: Check if there is a header. If not, call _getNewHeader(srcFile) -> hdr. Store hdr to cache. Insert cached header in the new FITS if it’s not there yet.

You have to implement the _getHeader(srcName) -> pyfits header object function. It must raise an exception if it cannot come up with a header. You also have to implement _isProcessed(srcName) -> boolean returning True if you think srcName already has a processed header.

This basic flow is influenced by the following opts attributes:

• reProcess – even if a cache is present, recompute header values

• applyHeaders – actually replace old headers with new headers

• reHeader – even if _isProcessed returns True, write a new header

• compute – perform computations

The idea is that you can:

• generate headers without touching the original files: proc

• write all cached headers to files that don’t have them

  proc –apply –nocompute

• after a bugfix force all headers to be regenerated:

  proc –reprocess –apply –reheader

All this leads to the messy logic. Sorry ‘bout this.

static addOptions(optParser)

cardSequence = None

commentFilter(value)

returns true if the comment value should be preserved.

You may want to override this.

static getPrimaryHeader(srcName)

returns the primary header of srcName.

This is a convenience function for user derived classes.

headerExt = '.hdr'

historyFilter(value)

returns true if the history item value should be preserved.

keepKeys = {'BITPIX', 'BSCALE', 'BZERO', 'EXTEND', 'NAXIS', 'NAXIS1', 'NAXIS2', 'SIMPLE'}

maxHeaderBlocks = 40

process(srcName)

class gavo.helpers.processing.ImmediateHeaderProcessor(opts, dd)

Bases: FileProcessor

An base for processors doing simple FITS manipulations to the primary FITS header.

To define these, override _isProcessed(self, srcName, hdr) and _changeHeader(self, hdr).

_changeHeader can change the pyfits header hdr in place. It will then be replaced on the actual file.

For complex operations, it is probably advisable to use HeaderProcessor which gives you a two-step process of first having the detached headers that you can check before applying them.

static addOptions(optParser)

process(srcName)

class gavo.helpers.processing.PreviewMaker(opts, dd)

Bases: FileProcessor

A file processor for generating previews.

For these, define a method getPreviewData(accref) -> string returning the raw preview data.

static addOptions(optParser)

classify(path)

getPreviewPath(accref)

iterIdentifiers()

iterates over the accrefs in the first table of dd.

process(accref)

class gavo.helpers.processing.SpectralPreviewMaker(opts, dd)

Bases: PreviewMaker

connectPoints = True

fluxColumn = 'flux'

static get2DPlot(tuples, linear=False, connectPoints=True)

returns a png-compressed pixel image for a 2D plot of (x,y) tuples.

getPreviewData(accref)

linearFluxes = False

spectralColumn = 'spectral'

gavo.helpers.processing.matplotlibLock()

gavo.helpers.processing.procmain(processorClass, rdId, ddId)

The “standard” main function for processor scripts.

The function returns the instantiated processor so you can communicate from your processor back to your own main.

See processors.html for details.