gavo.utils.texttricks module

Formatting, text manipulation, string constants, and such.

class gavo.utils.texttricks.NameMap(src: str, missingOk: bool = False, enc: str = 'utf-8')

Bases: object

is a name mapper fed from a simple text file.

The text file format simply is:

<target-id> “TAB” <src-id>{whitespace <src-id>}

src-ids have to be encoded quoted-printable when they contain whitespace or other “bad” characters (“=”!). You can have #-comments and empty lines.

The file is supposed to be ASCII, with non-ASCII encoded quoted-printable. The qp-decoded strings are assumed to be utf-8 encoded, but there’s a constructor argument to change that.

resolve(name: str) → str

gavo.utils.texttricks.bytify(s: Union[str, bytes]) → bytes

returns s utf-8 encoded if it is a string, unmodified otherwise.

gavo.utils.texttricks.datetimeToRFC2616(dt: datetime) → str

returns a UTC datetime object in the format required by http.

This may crap when you fuzz with the locale. In general, when handling “real” times within the DC, prefer unix timestamps over datetimes and use the other *RFC2616 functions.

gavo.utils.texttricks.debytify(b: Union[str, bytes], enc: str = 'ascii')

returns a bytestring b as a normal string.

This will return b unless it’s bytes. If it is bytes, it will be decoded as enc (and the thing will fail when the encoding is wrong).

gavo.utils.texttricks.defuseFileName(fName: Union[str, bytes], replaceSlash: bool = True)

returns fName without any non-ASCII or slashes but in a way that people can still work out what the file name has been.

This is basically a quoted-printable encoding. What’s returned is a string that’s guaranteed to be ASCII only. With replaceSlash=False, it can also double as a reasonable asciification.

gavo.utils.texttricks.degToDms(deg: float, sepChar: str = ' ', secondFracs: int = 2, preserveLeading: bool = False, truncate: bool = False, addSign: bool = True) → str

converts a float angle in degrees to a sexagesimal string.

This takes a lot of optional arguments:

• sepChar is the char separating the components

• secondFracs is the number for fractional seconds to generate

• preserveLeading can be set to True if leading zeroes should be preserved

• truncate can be set to True if fractional seconds should be truncated rather then rounded (as necessary for building IAU identifiers)

• addSign, if true, makes the function return a + in front of positive values (the default)

>>> degToDms(-3.24722, "", 0, True, True)
'-031449'
>>> degToDms(0)
'+0 00 00.00'
>>> degToDms(0, addSign=False)
'0 00 00.00'
>>> degToDms(-0.25, sepChar=":")
'-0:15:00.00'
>>> degToDms(-23.50, secondFracs=4)
'-23 30 00.0000'
>>> "%.4f"%dmsToDeg(degToDms(-25.6835, sepChar=":"), sepChar=":")
'-25.6835'

gavo.utils.texttricks.degToHms(deg: float, sepChar: str = ' ', secondFracs: int = 3, truncate: bool = False) → str

converts a float angle in degrees to an time angle (hh:mm:ss.mmm).

This takes a lot of optional arguments:

• sepChar is the char separating the components

• secondFracs is the number for fractional seconds to generate

• truncate can be set to True if fractional seconds should be truncated rather then rounded (as necessary for building IAU identifiers)

>>> degToHms(0, sepChar=":")
'00:00:00.000'
>>> degToHms(122.057, secondFracs=1)
'08 08 13.7'
>>> degToHms(122.057, secondFracs=1, truncate=True)
'08 08 13.6'
>>> degToHms(-0.055, secondFracs=0)
'-00 00 13'
>>> degToHms(-0.055, secondFracs=0, truncate=True)
'-00 00 13'
>>> degToHms(-1.056, secondFracs=0)
'-00 04 13'
>>> degToHms(-1.056, secondFracs=0)
'-00 04 13'
>>> degToHms(359.9999999)
'24 00 00.000'
>>> degToHms(359.2222, secondFracs=4, sepChar=":")
'23:56:53.3280'
>>> "%.4f"%hmsToDeg(degToHms(256.25, secondFracs=9))
'256.2500'

gavo.utils.texttricks.dmsToDeg(dmsAngle: str, sepChar: Optional[str] = None) → float

returns the degree minutes seconds-specified dmsAngle as a float in degrees.

>>> "%3.8f"%dmsToDeg("45 30.6")
'45.51000000'
>>> "%3.8f"%dmsToDeg("45:30.6", ":")
'45.51000000'
>>> "%3.8f"%dmsToDeg("-45 30 7.6")
'-45.50211111'
>>> dmsToDeg("junk")
Traceback (most recent call last):
ValueError: Invalid dms value with sepChar None: 'junk'

gavo.utils.texttricks.ensureOneSlash(s: str) → str

returns s with exactly one trailing slash.

gavo.utils.texttricks.fixIndentation(code: str, newIndent: str, governingLine: int = 0) → str

returns code with all whitespace from governingLine removed from every line and newIndent prepended to every line.

governingLine lets you select a line different from the first one for the determination of the leading white space. Lines before that line are left alone.

>>> fixIndentation("  foo\n  bar", "")
'foo\nbar'
>>> fixIndentation("  foo\n   bar", " ")
' foo\n  bar'
>>> fixIndentation("  foo\n   bar\n    baz", "", 1)
'foo\nbar\n baz'
>>> fixIndentation("  foo\nbar", "")
Traceback (most recent call last):
gavo.utils.excs.StructureError: Bad indent in line 'bar'

gavo.utils.texttricks.formatFloat(f: float) → str

returns floating-point numbers somewhat suitable for human consumption.

The idea of this function is to slowly migrate ad-hoc formatting of this kind that’s in all kind of different places in DaCHS here and thus have a central knob we can eventually use to adapt to tastes.

>>> formatFloat(1)
'1'
>>> formatFloat(1/3)
'0.333333'
>>> formatFloat(-1/3e20)
'-3.33333e-21'
>>> import math;formatFloat(math.pi)
'3.14159'
>>> formatFloat(20000000.23)
'2e+07'

gavo.utils.texttricks.formatISODT(dt: datetime) → str

returns some ISO8601 representation of a datetime instance.

The reason for preferring this function over a simple str is that datetime’s default representation is too difficult for some other code (e.g., itself); hence, this code suppresses any microsecond part and always adds a Z (where strftime works, utils.isoTimestampFmt produces an identical string).

The behaviour of this function for timezone-aware datetimes is undefined.

For convenience, None is returned as None.

Also for convenience, you can pass in a string; this will then be parsed first, which provides both some basic format validation and guaranteed DALI-compliant serialisation.

>>> formatISODT(datetime.datetime(2015, 10, 20, 12, 34, 22, 250))
'2015-10-20T12:34:22Z'
>>> formatISODT(datetime.datetime(1815, 10, 20, 12, 34, 22, 250))
'1815-10-20T12:34:22Z'
>>> formatISODT(datetime.datetime(2018, 9, 21, 23, 59, 59, 640000))
'2018-09-22T00:00:00Z'

gavo.utils.texttricks.formatRFC2616Date(secs: Optional[float] = None) → str

returns an RFC2616 date string for UTC seconds since unix epoch.

gavo.utils.texttricks.formatSimpleTable(data: list[list[Any]], stringify: bool = True, titles: Optional[list[str]] = None) → str

returns a string containing a text representation of tabular data.

All columns of data are simply stringified, then the longest member determines the width of the text column. The behaviour if data does not contain rows of equal length is unspecified; data must contain at least one row.

If you have serialised the values in data yourself, pass stringify=False.

If you pass titles, it must be a sequence of strings; they are then used as table headers; the shorter of data[0] and titles will determine the number of columns displayed.

gavo.utils.texttricks.formatSize(val: float, sf: int = 1) → str

returns a human-friendly representation of a file size.

gavo.utils.texttricks.fracHoursToDeg(fracHours: float) → float

returns the time angle fracHours given in decimal hours in degrees.

gavo.utils.texttricks.getFileStem(fPath: str)

returns the file stem of a file path.

The base name is what remains if you take the base name and split off extensions. The extension here starts with the last dot in the file name, except up to one of some common compression extensions (.gz, .xz, .bz2, .Z, .z) is stripped off the end if present before determining the extension.

>>> getFileStem("/foo/bar/baz.x.y")
'baz.x'
>>> getFileStem("/foo/bar/baz.x.gz")
'baz'
>>> getFileStem("/foo/bar/baz")
'baz'

gavo.utils.texttricks.getRandomString(length: int) → str

returns a random string of harmless printable characters.

gavo.utils.texttricks.getRelativePath(fullPath: str, rootPath: str, liberalChars: bool = True) → str

returns rest if fullPath has the form rootPath/rest and raises an exception otherwise.

Pass liberalChars=False to make this raise a ValueError when URL-dangerous characters (blanks, amperands, pluses, non-ASCII, and similar) are present in the result. This is mainly for products.

gavo.utils.texttricks.hmsToDeg(hms: str, sepChar: Optional[str] = None) → float

returns the time angle (h m s.decimals) as a float in degrees.

>>> "%3.8f"%hmsToDeg("22 23 23.3")
'335.84708333'
>>> "%3.8f"%hmsToDeg("22:23:23.3", ":")
'335.84708333'
>>> "%3.8f"%hmsToDeg("222323.3", "")
'335.84708333'
>>> hmsToDeg("junk")
Traceback (most recent call last):
ValueError: Invalid time with sepChar None: 'junk'

gavo.utils.texttricks.hoursToHms(decimal_hours: float, sepChar: str = ':', secondFracs: int = 0) → str

returns a time span in hours in sexagesmal time (h:m:s).

The optional arguments are as for degToHms.

>>> hoursToHms(0)
'00:00:00'
>>> hoursToHms(23.5)
'23:30:00'
>>> hoursToHms(23.55)
'23:33:00'
>>> hoursToHms(23.525)
'23:31:30'
>>> hoursToHms(23.553, secondFracs=2)
'23:33:10.80'
>>> hoursToHms(123.553, secondFracs=2)
'123:33:10.80'

gavo.utils.texttricks.iterSimpleText(f: TextIO) → Generator[Tuple[int, str], None, None]

iterates over (physLineNumber, line) in f with some usual conventions for simple data files.

You should use this function to read from simple configuration and/or table files that don’t warrant a full-blown grammar/rowmaker combo. The intended use is somewhat like this:

with open(rd.getAbsPath("res/mymeta")) as f:
        for lineNumber, content in iterSimpleText(f):
                try:
                        ...
                except Exception, exc:
                        sys.stderr.write("Bad input line %s: %s"%(lineNumber, exc))

The grammar rules are, specifically:

• leading and trailing whitespace is stripped

• empty lines are ignored

• lines beginning with a hash are ignored

• lines ending with a backslash are joined with the following line; to have intervening whitespace, have a blank in front of the backslash.

gavo.utils.texttricks.makeEllipsis(aStr: str, maxLen: int = 60, ellChars: str = '...') → str

returns aStr cropped to maxLen if necessary.

Cropped strings are returned with an ellipsis marker.

gavo.utils.texttricks.makeIAUId(prefix: str, long: float, lat: float, longSec: int = 0, latSec: int = 0) → str

returns an (equatorial) IAU identifier for an object at long and lat.

The rules are given on https://cds.unistra.fr/Dic/iau-spec.html

The prefix, including the system identifier, you have to pass in. You cannot build identifiers using only minutes precision. If you want to include sub-arcsec precision, pass in longSec and/or latSec (the number of factional seconds to preserve).

gavo.utils.texttricks.makeLeftEllipsis(aStr: str, maxLen: int = 60)

returns aStr shortened to maxLen by dropping prefixes if necessary.

Cropped strings are returned with an ellipsis marker.

>>> makeLeftEllipsis("0123456789"*2, 11)
'...23456789'

gavo.utils.texttricks.makeSourceEllipsis(sourceToken: Any) → str

returns a string hopefully representative for a source token.

These are, in particular, passed around within rsc.makeData. Usually, these are (potentially long) strings, but now and then they can be other things with appallingly long reprs. When DaCHS messages need to refer to such sources, this function is used to come up with representative strings.

gavo.utils.texttricks.parseAccept(aString: str) → Dict[str, str]

parses an RFC 2616 accept header and returns a dict mapping media type patterns to their (unparsed) parameters.

If aString is None, an empty dict is returned

If we ever want to do fancy things with http content negotiation, this will be further wrapped to provide something implementing the complex RFC 2616 rules; this primitive interface really is intended for telling apart browsers (which accept text/html) from other clients (which hopefully do not) at this point.

>>> sorted(parseAccept("text/html, text/*; q=0.2; level=3").items())
[('text/*', 'q=0.2; level=3'), ('text/html', '')]
>>> parseAccept(None)
{}

gavo.utils.texttricks.parseAssignments(assignments: str) → Dict[str, str]

returns a name mapping dictionary from a list of assignments.

This is the preferred form of communicating a mapping from external names to field names in records to macros – in a string that contains “:”-seprated pairs separated by whitespace, like “a:b b:c”, where the incoming names are leading, the desired names are trailing.

If you need defaults to kick in when the incoming data is None, try _parseDestWithDefault in the client function.

This function parses a dictionary mapping original names to desired names.

>>> parseAssignments("a:b  b:c")
{'a': 'b', 'b': 'c'}

gavo.utils.texttricks.parseDefaultDate(literal: Optional[Union[str, date]]) → Optional[date]

parseDefaultDatetime’s little sister.

gavo.utils.texttricks.parseDefaultDatetime(literal: Optional[Union[str, datetime]]) → Optional[datetime]

returns a datetime from string or passes through datetimes and Nones.

The function will try to parse a string in various ways; we will try not to drop formats from one minor version to the next.

gavo.utils.texttricks.parseDefaultTime(literal: Optional[Union[str, time]]) → Optional[time]

parseDefaultDatetime’s other little sister.

gavo.utils.texttricks.parseISODT(literal: str, useTime: bool = False) → datetime

returns a datetime object for a ISO time literal.

There’s no real timezone support yet, but we accept and ignore various ways of specifying UTC.

By default, this uses plain python datetime because it usually covers a large date range than the time module. The downside is that it does not know about leap seconds. Pass useTime=True to go through time tuples, which know how to deal with them (but may not deal with dates far in the past or future).

>>> parseISODT("1998-12-14")
datetime.datetime(1998, 12, 14, 0, 0)
>>> parseISODT("1998-12-14T13:30:12")
datetime.datetime(1998, 12, 14, 13, 30, 12)
>>> parseISODT("1998-12-14T13:30:12Z")
datetime.datetime(1998, 12, 14, 13, 30, 12)
>>> parseISODT("1998-12-14T13:30:12.224Z")
datetime.datetime(1998, 12, 14, 13, 30, 12, 224000)
>>> parseISODT("19981214T133012Z")
datetime.datetime(1998, 12, 14, 13, 30, 12)
>>> parseISODT("19981214T133012+00:00")
datetime.datetime(1998, 12, 14, 13, 30, 12)
>>> parseISODT("2016-12-31T23:59:60")
Traceback (most recent call last):
ValueError: second must be in 0..59
>>> parseISODT("2016-12-31T23:59:60", useTime=True)
datetime.datetime(2017, 1, 1, 1, 0)
>>> parseISODT("junk")
Traceback (most recent call last):
ValueError: Bad ISO datetime literal: junk (required format: yyyy-mm-ddThh:mm:ssZ)

gavo.utils.texttricks.parsePercentExpression(literal: str, format: str) → dict

returns a dictionary of parts in the %-template format.

format is a template with %<conv> conversions, no modifiers are allowed. Each conversion is allowed to contain zero or more characters matched stingily. Successive conversions without intervening literals aren’t really supported. There’s a hack for strptime-type times, though: H, M, and S just eat two characters each if there’s no separator.

This is really only meant as a quick hack to support times like 25:33.

>>> r=parsePercentExpression("12,xy:33,","%a:%b,%c"); r["a"], r["b"], r["c"]
('12,xy', '33', '')
>>> sorted(parsePercentExpression("2357-x", "%H%M-%u").items())
[('H', '23'), ('M', '57'), ('u', 'x')]
>>> r = parsePercentExpression("12,13,14", "%a:%b,%c")
Traceback (most recent call last):
ValueError: '12,13,14' cannot be parsed using format '%a:%b,%c'

gavo.utils.texttricks.parseRFC2616Date(s: str) → float

returns seconds since unix epoch representing UTC from the HTTP-compatible time specification s.

gavo.utils.texttricks.replaceXMLEntityRefs(unicodeString: str) → str

retplaces all known HTML entities in unicodeString with actual unicode chars.

(and dies on unknown entities).

TODO: this is unused and probably not very useful to clients. Discard?

gavo.utils.texttricks.resolvePath(rootPath: str, relPath: str) → str

joins relPath to rootPath and makes sure the result really is in rootPath.

gavo.utils.texttricks.roundToSeconds(dt: datetime) → datetime

returns a datetime instance rounded to whole seconds.

This also recklessly clears any time zone marker. So, don’t pass in anything with a meaningful time zone.

gavo.utils.texttricks.safe_str(val: Any) → str

returns a reasonable string from pretty much anything.

gavo.utils.texttricks.timegm(tm: struct_time, epoch: float = 25203)