I nternational
V irtual
O bservatory
A lliance
This document describes an XML encoding standard for metadata about services implementing the table access protocol TAP [TAP], referred to as TAPRegExt. Instance documents are part of the service's registry record or can be obtained from the service itself. They deliver information to both humans and software on the languages, output formats, and upload methods supported by the service, as well as data models implemented by the exposed tables, optional language features, and certain limits enforced by the service.
This is a Working Draft of the DAL WG in cooperation with the Registry WG. The first release of this document was 2011 Jan 27. This is the second public WD.
This is an IVOA Working Draft for review by IVOA members and other interested parties. It is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use IVOA Working Drafts as reference materials or to cite them as other than "work in progress".
A list of current IVOA Recommendations and other technical documents can be found at http://www.ivoa.net/Documents/.
This document has been developed with support from the German Astronomical Virtual Observatory (BMBF Bewilligungsnummer 05A08VHA). Anyone else wants to be here?
This document has borrowed extensively from StandardsRegExt [SRE] working drafts.
This document defines the TAPRegExt schema using XML Schema (see [XSD]). The full Schema document is listed in Appendix A. Parts of the schema appear within the main sections of this document; however, documentation nodes have been left out for the sake of brevity.
Reference to specific elements and types defined in the VOResource
schema include the namespace prefix, vr
, as in
vr:Resource
(a type defined in the VOResource schema).
Reference to specific elements and types defined in the TAPRegExt
schema include the namespaces prefix, tr
, as in
tr:TableAccess
(a type defined in the TAPRegExt schema).
Use of the tr
prefix in compliant instance documents is
strongly recommended, particularly in the applications that involve
IVOA Registries.
The Table Access Protocol TAP (see [TAP]) allows VO clients to send queries to remote database servers and receive the results in standard formats. In addition, it defines means to discover database schemata on the remote side, to upload data from the local disk or third-party hosts, and more. TAP builds upon a variety of other standards, premier among which is the Universal Worker Service (see [UWS]), which describes how client and server can negotiate the execution of a query and the retrieval of results without having to maintain a continuous connection.
To accomodate a wide variety of requirements, the TAP specification offers implementors many choices on optional features, resource limits, or locally defined functionality. One purpose of TAPRegExt is to allow the service to communicate such choices to remote clients using the mechanisms laid down in the VO Service Interfaces standard (see [VOSI]).
Clients also need to discover TAP services offering certain kinds of data.
Central to this is the concept of a registry where resources can be
described and thus discovered by users and applications in the VO. Registries
receive resource descriptions as defined in the IVOA standard
[VOR]. In this
schema, support for a standard service protocol is described as a service's
capability; the associated metadata is contained within the service resource
description's <capability>
element.
TAPRegExt defines this capability element for TAP services. In the context of registering TAP services, an important role filled by TAPRegExt is the communication of supported data models to the registry.
Figure 1: IVOA Architecture diagram with TAPRegExt and the related standards marked up.
This specification directly relates to other IVOA standards in the following ways:
This standard also relates to other IVOA standards:
capabilities
endpoint described by VOSI.The namespace associated with TAPRegExt VOResource extensions will be
http://www.ivoa.net/xml/TAPRegExt/v1.0
.
Just like the namespace URI for the VOResource schema, the
TAPRegExt namespace URI can be interpreted as a URL. Resolving it
will return the XML Schema document (given in Appendix A)
that defines the TAPRegExt schema.
Authors of VOResource instance documents may choose to
provide a location for the VOResource XML Schema document and its
extensions using the
xsi:schemaLocation
attribute. While the author is
free to choose any schema location, this specification
recommends using the TAPRegExt namespace URI as its location URL
(as illustrated in the example above), as in,
xsi:schemaLocation="http://www.ivoa.net/xml/TAPRegExt/v1.0 http://www.ivoa.net/xml/TAPRegExt/v1.0"
The IVOA Registry Interface standard [RI]
actually requires that the VOResource records it
shares with other registries provide location URLs via
xsi:schemaLocation
for the VOResource schema and
all legal extension schemas that are used in the records. This
rule would apply to the TAPRegExt schema.
The IVOA defines certain data models that can be instantiated in database tables exposed by a TAP service. This allows a query built exclusively on a data model or a set of data models to work on all TAP services exposing tables instantiating the data model(s).
In TAPRegExt, a data model is identified by its IVORN. The first example for such a data model is ObsCore (see [OT]).
<xs:complexType name="DataModelType" > <xs:simpleContent > <xs:extension base="xs:token" > <xs:attribute name="ivo-id" type="vr:IdentifierURI" use="required" /> </extension> </simpleContent> </complexType>
tr:DataModelType Attributes | |||||||
---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||
ivo-id |
|
TAP services may offer a variety of query languages. In TAPRegExt, the
language
element allows the communication of what languages are
available on a service. TAP defines values of the LANG
parameter
to have either the form <name>-<version>
or the form
<name>
, where the latter form leaves the choice of the
version to the server. Therefore, a language is defined using a name and one
or more versions.
The recommended way to associate larger amounts of documentation with a language entry in a capability element is via registration of the language using the mechanisms defined in [SRE] and associating the registry record with the language element through the latter's ivo-id attribute. The only language mandatory for TAP services, ADQL 2.0, has a registry record at ivo://ivoa.net/ADQL/2.0
<xs:complexType name="Language" > <xs:sequence > <xs:element name="name" type="xs:NCName" /> <xs:element name="version" type="tr:Version" minOccurs="1" maxOccurs="unbounded" /> <xs:element name="description" type="xs:token" minOccurs="0" /> <xs:element name="languageFeatures" type="tr:LanguageFeatureList" minOccurs="0" maxOccurs="unbounded" /> </sequence> </complexType>
tr:Language Metadata Elements | |||||||||
---|---|---|---|---|---|---|---|---|---|
Element | Definition | ||||||||
name |
| ||||||||
version |
| ||||||||
description |
| ||||||||
languageFeatures |
|
<xs:complexType name="Version" > <xs:simpleContent > <xs:extension base="xs:token" > <xs:attribute name="ivo-id" type="vr:IdentifierURI" /> </extension> </simpleContent> </complexType>
tr:Version Attributes | |||||||||
---|---|---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||||
ivo-id |
|
Query languages may support optional features. For ADQL, the most prominent
of those are user-defined functions, i.e. functions not defined in the language
standard but added by the operators of the service, and geometry functions.
Such optional features may be communicated to the service client by use of one
or more tr:languageFeatures
elements.
Each such list is labelled with a type
attribute
indicating the type of language option being described.
This string should be an IVORN whose semantics in this context,
along with the semantics of the content of its descendant
feature/form
elements,
can be documented in association with the language in question.
TAPRegExt itself defines the following feature types:
ivo://ivoa.net/TAPRegExt#features-udf
Each feature declares a user-defined ADQL (or similar) function supported.
The content of the form
element
must be the signature of the function, written to match the
signature
nonterminal in the following grammar:
signature ::= <funcname> <arglist> "->" <type_name> funcname ::= <regular_identifier> arglist ::= "(" <arg> { "," <arg> } ")" arg ::= <regular_identifier> <type_name>
The type_name
nonterminal is not defined by the ADQL
grammar. For the purposes of TAPRegExt, it is sufficient to assume
it expands to "some sort of SQL type specifier" (which may
include spaces and parentheses).
ivo://ivoa.net/TAPRegExt#features-adqlgeo
Each feature declares support for one of the geometry functions defined by ADQL (support for these functions is in general optional for ADQL implementations, though TAP imposes some constraints on what combinations of support are permitted).
The signature of these functions, where supported, is fixed by ADQL;
the content of the form
element
is just the name of the function.
Example:
<feature> <form>CONTAINS</form> </feature>
We could go on to define what's allowed in a region and the coordinate systems supported. However, I'd hope both features won't become very prominent, and thus I'd rather not talk about them too much. If you disagree, it's not hard to sway me, but at least for REGION I'd need input as to what "form" could be for it. -- MD
<xs:complexType name="LanguageFeatureList" > <xs:sequence > <xs:element name="feature" type="tr:LanguageFeature" minOccurs="0" maxOccurs="unbounded" /> </sequence> <xs:attribute name="type" type="xs:anyURI" use="required" /> </complexType>
tr:LanguageFeatureList Metadata Elements | |||||||
---|---|---|---|---|---|---|---|
Element | Definition | ||||||
feature |
|
tr:LanguageFeatureList Attributes | |||||||||
---|---|---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||||
type |
|
<xs:complexType name="LanguageFeature" > <xs:sequence > <xs:element name="form" type="xs:token" /> <xs:element name="description" type="xs:string" minOccurs="0" /> </sequence> </complexType>
tr:LanguageFeature Metadata Elements | |||||||||
---|---|---|---|---|---|---|---|---|---|
Element | Definition | ||||||||
form |
| ||||||||
description |
|
A TAP service may offer a variety of output formats.
What output formats are available is defined using
outputFormat
elements. They
declare a MIME type (see [RFC2045]) as well
as aliases (the shorthand forms the server also accepts in the
FORMAT parameter). If desired, the format can be further described
using an IVORN; at the time of writing, the TAPRegExt provides keys for
tabledata encoded and binary encoded VOTables.
<xs:complexType name="OutputFormat" > <xs:sequence > <xs:element name="mime" type="xs:token" /> <xs:element name="alias" type="xs:token" minOccurs="0" maxOccurs="unbounded" /> </sequence> <xs:attribute name="ivo-id" type="xs:anyURI" /> </complexType>
tr:OutputFormat Metadata Elements | |||||||||
---|---|---|---|---|---|---|---|---|---|
Element | Definition | ||||||||
mime |
| ||||||||
alias |
|
tr:OutputFormat Attributes | |||||||||
---|---|---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||||
ivo-id |
|
TAP services should allow the upload of VOTables. They can support various methods to do this: HTTP upload, retrieval by URL, but also VOSpace or possibly retrieval using Grid protocols. Since an actual specification of the details of such protocols is far beyond the scope of a registry document and probably would not benefit clients anyway, the upload methods are given as IVORNs.
Keys for the standard upload methods are provided within the
resource record
ivo://ivoa.net/TAPRegExt
(while the registry record is not in place,
it can be examined in the volute repository).
The keys are built by using the keys as fragment identifiers within the
TAPRegExt IVORN.
It is permitted to register upload methods under authorities other than ivoa.net. The registry records can then provide more in-depth information. For the upload methods defined in the TAP specification, however, you must use the IVORNs of the keys in the TAPRegExt resource record to enable clients to identify supported methods using string comparisons.
At the time of writing, the set of protocol identifiers includes:
upload-inline
-- HTTP upload as per section 2.5.2 of
[TAP].upload-http
-- retrieval from an http URL.upload-https
-- retrieval from an https URL.upload-ftp
-- retrieval from an ftp URL.Thus, a service offering upload by retrieving from ftp and http URLs would say:
<uploadMethod ivo-id="ivo://ivoa.net/TAPRegExt#upload-http"/> <uploadMethod ivo-id="ivo://ivoa.net/TAPRegExt#upload-ftp"/>
<xs:complexType name="UploadMethod" > <xs:complexContent > <xs:restriction base="xs:anyType" > <xs:attribute name="ivo-id" type="xs:anyURI" /> </restriction> </complexContent> </complexType>
tr:UploadMethod Attributes | |||||||
---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||
ivo-id |
|
TAP services usually impose certain limits on resource usage by clients,
e.g., a maximum run time per query, or a maximum number of rows in the result
set. Services assing such limits to newly created jobs and may
allow raising the limits by means of queries or query parameters (e.g., the
size of the result set is limited by the MAXREC
parameter, whereas
the date of job destruction may be changed by adjusting the
destruction
parameter). Services may put some limit to how
far the resource limitations may be raised.
The capabilities element allows the declaration of such limits. These declarations are primarily intended for human consumption and should give guidelines. If a service supports authentication and has different limits depending on what user is authenticated, it should return the limits applying to the logged user.
The resource limits applying to newly created jobs are given in
default
elements, the limits beyond which users cannot
raise the limits are given in hard
elements.
Note that the absence of a specification of limits does not imply that no limits are enforced.
This document defines two time-like resource limits:
retentionPeriod
-- the time from job creation until
destruction
; services are free to give the maximum time
the destruction time may be set in the future here.executionDuration
-- the maximal run time given to
a query.All values in time-like limits are given in seconds. Both
retentionPeriod
and executionDuration
are of type
tr:TimeLimits
.
<xs:complexType name="TimeLimits" > <xs:sequence > <xs:element name="default" type="xs:integer" minOccurs="0" maxOccurs="1" /> <xs:element name="hard" type="xs:integer" minOccurs="0" maxOccurs="1" /> </sequence> </complexType>
tr:TimeLimits Metadata Elements | |||||||
---|---|---|---|---|---|---|---|
Element | Definition | ||||||
default |
| ||||||
hard |
|
Limits on data work like time limits, including having a
default
and a hard
value, except that
both those values have a unit attribute that can either be bytes
or rows
.
This document defines two resource limits on data:
outputLimit
-- if unit
is rows
here,
the default
gives the
value of TAP's MAXREC
parameter the service will use when none
is specified.uploadLimit
-- the maximum size of uploads. This
is not a TAP adjustable parameter. The default
value
advises clients about the server's wishes as to the limit above which
some sort of acknowledgement should be requested from the user. The
hard
limit gives the maximum size of an upload to the
server.Data limits are defined using the tr:DataLimits
and tr:DataLimit
types:
<xs:complexType name="DataLimits" > <xs:sequence > <xs:element name="default" type="tr:DataLimit" minOccurs="0" maxOccurs="1" /> <xs:element name="hard" type="tr:DataLimit" minOccurs="0" maxOccurs="1" /> </sequence> </complexType>
tr:DataLimits Metadata Elements | |||||||
---|---|---|---|---|---|---|---|
Element | Definition | ||||||
default |
| ||||||
hard |
|
<xs:complexType name="DataLimit" > <xs:simpleContent > <xs:extension base="xs:integer" > <xs:attribute name="unit" use="required" > <xs:simpleType > <xs:restriction base="xs:token" > <xs:enumeration value="bytes" /> <xs:enumeration value="rows" /> </restriction> </simpleType> </attribute> </extension> </simpleContent> </complexType>
tr:DataLimit Attributes | |||||||||
---|---|---|---|---|---|---|---|---|---|
Attribute | Definition | ||||||||
unit |
|
Using the types defined above, the
tr:TableAccess
type can be defined. Note that
it is a type, not a (global) element. In instance documents, you
will typically place it in a capability element with an explicit
type specification, like this:
<capability xmlns:tr="http://www.ivoa.net/xml/TAP/v1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" standardID="ivo://ivoa.net/std/TAP" xsi:type="tr:TableAccess"> ...
<xs:complexType name="TableAccess" > <xs:complexContent > <xs:extension base="tr:TAPCapRestriction" > <xs:sequence > <xs:element name="dataModel" type="tr:DataModelType" minOccurs="0" maxOccurs="unbounded" /> <xs:element name="language" type="tr:Language" minOccurs="1" maxOccurs="unbounded" /> <xs:element name="outputFormat" type="tr:OutputFormat" minOccurs="1" maxOccurs="unbounded" /> <xs:element name="uploadMethod" type="tr:UploadMethod" minOccurs="0" maxOccurs="unbounded" /> <xs:element name="retentionPeriod" type="tr:TimeLimits" minOccurs="0" maxOccurs="1" /> <xs:element name="executionDuration" type="tr:TimeLimits" minOccurs="0" maxOccurs="1" /> <xs:element name="outputLimit" type="tr:DataLimits" minOccurs="0" maxOccurs="1" /> <xs:element name="uploadLimit" type="tr:DataLimits" minOccurs="0" maxOccurs="1" /> </sequence> </extension> </complexContent> </complexType>
tr:TableAccess Extension Metadata Elements | |||||||||
---|---|---|---|---|---|---|---|---|---|
Element | Definition | ||||||||
dataModel |
| ||||||||
language |
| ||||||||
outputFormat |
| ||||||||
uploadMethod |
| ||||||||
retentionPeriod |
| ||||||||
executionDuration |
| ||||||||
outputLimit |
| ||||||||
uploadLimit |
|
As an example, here is an instance document as it might be returned from a VOSI capability endpoint: