<resource schema="tap_schema" resdir="__system">
	<meta name="creationDate">2009-12-01T10:00:00</meta>
	<meta name="subject">virtual-observatories</meta>
	<meta name="subject">catalogs</meta>
	<meta name="schema-rank">10</meta>

	<meta name="description">
		\getConfig{web}{sitename}'s Table Access Protocol
		(TAP) service with table metadata.
	</meta>


<!--**************************** the TAP schema *********************-->

	<table id="schemas" onDisk="True" system="True"
			dupePolicy="overwrite" primary="schema_name"
			adql="True">
		<meta name="description">Schemas containing tables available for ADQL
			querying.</meta>
	<!-- since schemata may be shared between RDs, nothing will ever
	     get deleted from here (automatically); in particular, there is no
	     sourceRD column here because that might need to have multiple
	     values.  In principle, based on dc.rdmeta one could drop schemas
	     here when they are no longer used. -->

		<column name="schema_name" type="text"
			description="Fully qualified schema name">
			<property name="std">1</property>
		</column>
		<column name="description" type="unicode"
			description="Brief description of the schema">
			<property name="std">1</property>
		</column>
		<column name="utype" type="text"
			description="utype if schema corresponds to a data model">
			<property name="std">1</property>
		</column>
		<column name="schema_index" type="integer"
			description="Suggested position this schema should take in a sorted
				list of schemas from this data center.">
			<property name="std">1</property>
			<values nullLiteral="-1"/>
		</column>
	</table>
	
	<table id="tables" onDisk="True" system="True" primary="table_name"
			adql="True">
		<meta name="description">Tables available for ADQL querying.</meta>
		<meta name="table-rank">1</meta>
		<foreignKey source="schema_name" inTable="schemas"/>

		<column name="schema_name" type="text"
			tablehead="Schema"
			description="Fully qualified schema name">
			<property name="std">1</property>
		</column>
		<column name="table_name" type="text"
			tablehead="Table"
			description="Fully qualified table name">
			<property name="std">1</property>
		</column>
		<column name="table_type" type="text"
			tablehead="Type"
			description="One of: table, view">
			<property name="std">1</property>
		</column>
		<column name="description" type="unicode"
			description="Brief description of the table">
			<property name="std">1</property>
		</column>
		<column name="utype" type="text"
			description="utype if the table corresponds to a data model">
			<property name="std">1</property>
		</column>
		<column name="table_index" type="integer"
			description="Suggested position this table should take in
				a sorted list of tables from this data center">
			<property name="std">1</property>
			<values nullLiteral="-1"/>
		</column>
		<column name="sourceRD" type="text"
			description="Id of the originating rd (local information)"/>
		<column name="nrows" type="bigint"
			description="The approximate size of the table in rows">
			<property name="std">1</property>
			<values nullLiteral="-1"/>
		</column>
	</table>

	<table id="columns" onDisk="True" system="True"
			primary="table_name,column_name" adql="True">
		<meta name="description">Columns in tables available for ADQL querying.
		</meta>
		<meta name="table-rank">1</meta>
		<foreignKey source="table_name" inTable="tables"/>

		<column name="table_name" type="text"
			tablehead="Table"
			description="Fully qualified table name">
			<property name="std">1</property>
		</column>
		<column name="column_name" type="text"
			tablehead="Name"
			description="Column name">
			<property name="std">1</property>
		</column>
		<column name="description" type="unicode"
			description="Brief description of column">
			<property name="std">1</property>
		</column>
		<column name="unit" type="text"
			description="Unit in VO standard format">
			<property name="std">1</property>
		</column>
		<column name="ucd" type="text"
			tablehead="UCD"
			description="UCD of column if any">
			<property name="std">1</property>
		</column>
		<column name="utype" type="text"
			description="Utype of column if any">
			<property name="std">1</property>
		</column>
		<column name="datatype" type="text"
			tablehead="Type"
			description="ADQL datatype">
			<property name="std">1</property>
		</column>
		<column name="arraysize" type="text"
			tablehead="Arr."
			description="Arraysize in VOTable notation">
			<property name="std">1</property>
		</column>
		<column name="xtype" type="text"
			tablehead="xtype"
			description="VOTable extended type information (for special
				interpretation of data content, e.g., timestamps or points)">
			<property name="std">1</property>
		</column>
		<column name="quoted/size" type="integer"
				description="Legacy length (ignore if you can).">
			<values nullLiteral="-1"/>
			<property name="std">1</property>
		</column>
		<column name="principal" type="integer" required="True"
			tablehead="Princ?"
			description="Is column principal?">
			<property name="std">1</property>
		</column>
		<column name="indexed" type="integer" required="True"
			tablehead="Indexed?"
			description="Is there an index on this column?">
			<property name="std">1</property>
		</column>
		<column name="std" type="integer" required="True"
			tablehead="Std?"
			description="Is this a standard column?">
			<property name="std">1</property>
		</column>
		<column name="sourceRD" type="text"
			description="Id of the originating rd (local information)"/>
		<column name="column_index" type="smallint"
			tablehead="Ind."
			description="1-based index of the column in database order.">
			<property name="std">1</property>
			<values nullLiteral="-1"/>
		</column>
	</table>

	<table id="keys" onDisk="True" system="True"
			primary="key_id" adql="True">
		<meta name="description">Foreign key relationships between tables
			available for ADQL querying.
		</meta>
		<foreignKey source="from_table" dest="table_name" inTable="tables"/>
		<foreignKey source="target_table" dest="table_name" inTable="tables"/>

		<column name="key_id" type="text"
			tablehead="Id"
			description="Unique key identifier">
			<property name="std">1</property>
		</column>
		<column name="from_table" type="text"
			tablehead="From table..."
			description="Fully qualified table name">
			<property name="std">1</property>
		</column>
		<column name="target_table" type="text"
			tablehead="To table..."
			description="Fully qualified table name">
			<property name="std">1</property>
		</column>
		<column name="description" type="unicode"
			description="Description of this key">
			<property name="std">1</property>
		</column>
		<column name="utype" type="text"
			description="Utype of this key">
			<property name="std">1</property>
		</column>
		<column name="sourceRD" type="text"
			description="Id of the originating rd (local information)"/>
	</table>

	<table id="key_columns" onDisk="True" system="True" adql="True">
		<meta name="description">Columns participating in foreign key
			relationships between tables available for ADQL querying.
		</meta>
		<foreignKey source="key_id"
			dest="key_id" inTable="keys"/>

		<column name="key_id" type="text"
			tablehead="Id"
			description="Key identifier from TAP_SCHEMA.keys">
			<property name="std">1</property>
		</column>
		<column name="from_column" type="text"
			tablehead="Src. Column"
			description="Key column name in the from table">
			<property name="std">1</property>
		</column>
		<column name="target_column" type="text"
			tablehead="Tgt. Column"
			description="Key column in the target table">
			<property name="std">1</property>
		</column>
		<column name="sourceRD" type="text"
			description="Id of the originating rd (local information)"/>
	</table>

	<table id="groups" onDisk="True" system="True" adql="True">
		<meta name="description">Columns that are part of groups
			within tables available for ADQL querying.
		</meta>
		<foreignKey source="table_name" inTable="tables"/>
		<!-- this is slightly denormalized, but normalizing it by introducing
		two new tables IMHO isn't worth it at all -->
		<column original="columns.table_name"/>
		<column original="columns.column_name"
			description="Name of a column belonging to the group"/>
		<column original="columns.utype" name="column_utype"
			description="utype the column within the group"/>
		<column name="group_name" type="text"
			description="Name of the group"/>
		<column name="group_utype" type="text"
			description="utype of the group"/>
		<column name="sourceRD" type="text"
			description="Id of the originating rd (local information)"/>
	</table>

	<table id="supportedmodels" onDisk="True" primary="dmivorn"
			dupePolicy="overwrite" system="True">
		<meta name="description">
			Standard data models supported by this service.

			This is a non-standard tap_schema table used by DaCHS in the creation
			of registry records.  It is manipulated through gavo imp on tables
			with supportsModel and supportsModelURI properties.
		</meta>
		<column original="tables.sourceRD"/>
		<column name="dmname" type="text"
			description="Human-readable name of the data model"/>
		<column name="dmivorn" type="text"
			description="IVOID of the data model (sorry for the legacy name)."/>
	</table>

	<data id="importTablesFromRD" auto="False">
		<embeddedGrammar isDispatching="True">
			<iterator>
				<code>
					rd = self.sourceToken
					yield ("schemas", {
						"schema_name": rd.schema,
						"schema_description": base.getMetaText(rd, "description"),
						"schema_utype": base.getMetaText(rd, "utype"),
						"schema_index": base.getMetaText(rd, "schema-rank")})

					# the moribund property is set by external code if the
					# rd is to be removed from TAP_SCHEMA.  The removal
					# is already done by the the newSource script in make
					# below, thus we only need to do nothing here.
					if rd.getProperty("moribund", False):
						return
					for table in rd.tables:
						if not table.adql or table.adql=="hidden":
							continue
						if table.viewStatement:
							tapTableType = "view"
						else:
							tapTableType = "table"
						
						yield ("tables", {
							"schema_name": rd.schema,
							"table_name": table.getQName().lower(),
							"table_description": base.getMetaText(
								table, "description",
								propagate=True),
							"table_type": tapTableType,
							"table_utype": base.getMetaText(
								table, "utype",
								propagate=False),
							"sourceRD": rd.sourceId,
							"nrows": table.nrows,
							"table_index": base.getMetaText(table, "table-rank"),
							})
				</code>
			</iterator>
		</embeddedGrammar>

		<make table="schemas" role="schemas">
			<rowmaker>
				<simplemaps>
					schema_index: schema_index,
					schema_name: schema_name,
					description: schema_description,
					utype: schema_utype
				</simplemaps>
			</rowmaker>
		</make>

		<make table="tables" role="tables">
			<script type="newSource" lang="python" id="removeStale"
					notify="False" name="delete stale TAP_SCHEMA entries">
				table.deleteMatching("sourceRD=%(sourceRD)s",
					{"sourceRD": sourceToken.sourceId})
			</script>

			<rowmaker idmaps="*">
				<simplemaps>
					utype: table_utype,
					description: table_description
				</simplemaps>
			</rowmaker>
		</make>
	</data>

	<data id="importDMsFromRD" auto="False">
		<!-- note: we rely on tables removing entries from that list
		themselves rather than doing it here.  The reason is that
		multiple different tables might claim the same data model, and
		if I unconditionally tore them down, that's about as broken
		as leaving them.  The whole dataModel thing in TAPRegExt was
		a mistake. -->
		<embeddedGrammar>
			<iterator>
				<code>
					rd = self.sourceToken
					if rd.getProperty("moribund", False):
						return

					for table in rd.tables:
						if not table.adql or table.adql=="hidden":
							continue
						yield {
							"sourceRD": rd.sourceId,
							"table_name": table.getQName().lower(),
							"dmname": table.getProperty("supportsModel", None),
							"dmivorn": table.getProperty("supportsModelURI", None),
						}
				</code>
			</iterator>
			<rowfilter>
				<code>
				if @dmname:
					for name, uri in (
							zip(@dmname.split(","), @dmivorn.split())):
						yield {
							"sourceRD": @sourceRD,
							"dmname": name.strip(),
							"dmivorn": uri}
				</code>
			</rowfilter>
		</embeddedGrammar>
		<make table="supportedmodels"/>
	</data>

	<data id="importColumnsFromRD" auto="False">
		<embeddedGrammar>
			<iterator>
				<setup>
					<code>
						from gavo.utils.typeconversions import sqltypeToVOTable
					</code>
				</setup>
				<code>
					rd = self.sourceToken
					if rd.getProperty("moribund", False):
						return
					for table in rd.tables:
						if not table.adql or table.adql=="hidden":
							continue
						
						for index, col in enumerate(table):
							if col.hidden:
								continue
							indexed = 0
							if col.isIndexed():
								indexed = 1
							type, arraysize, xtype = sqltypeToVOTable(col.type)
							# xtype overrides -- if this becomes more common, we'll
							# have to change sqltypeToVOTable to take local xtypes
							# into account as well.
							# There's analogous code in registry.tableset.
							if col.xtype=='adql:REGION':
								type, arraysize, xtype = "char", "*", "adql:REGION"

							try:
								legacy_size = int(arraysize)
							except (TypeError, ValueError):
								legacy_size = None

							yield {
								"table_name": table.getQName().lower(),
								"column_name": col.getTAPName(),
								"description": col.description,
								"unit": col.unit,
								"ucd": col.ucd,
								"utype": col.utype,
								"xtype": xtype,
								"arraysize": arraysize,
								"datatype": type,
								"size": legacy_size,
								"principal": col.verbLevel&lt;=20,
								"indexed": indexed,
								"std": parseInt(col.getProperty("std", "0")),
								"sourceRD": rd.sourceId,
								"column_index": index+1,
							}
				</code>
			</iterator>
		</embeddedGrammar>
		<make table="columns">
			<script original="removeStale"/>
		</make>
	</data>

	<data id="importGroupsFromRD" auto="False">
		<embeddedGrammar>
			<iterator>
				<code>
					rd = self.sourceToken
					if rd.getProperty("moribund", False):
						return
					for table in rd.tables:
						if not table.adql or table.adql=="hidden":
							continue
						for group in table.groups:
							for colRef in group.columnRefs:
								yield {
									"table_name": table.getQName().lower(),
									"column_name": colRef.resolve(table).getTAPName(),
									"column_utype": colRef.utype,
									"group_name": group.name,
									"group_utype": group.utype,
									"sourceRD": rd.sourceId,
								}
				</code>
			</iterator>
		</embeddedGrammar>
		<make table="groups"/>
	</data>

	<data id="importFkeysFromRD" auto="False">
		<embeddedGrammar>
			<iterator>
				<code>
					rd = self.sourceToken
					if rd.getProperty("moribund", False):
						return
					for table in rd.tables:
						if not table.adql or table.adql=="hidden":
							continue
						for fkey in table.foreignKeys:
							if not fkey.isADQLKey:
								continue

							fkeyId = rd.sourceId+utils.intToFunnyWord(id(fkey))
							yield {
								"key_id": fkeyId.lower(),
								"from_table": table.getQName().lower(),
								"target_table": fkey.inTable.getQName().lower(),
								"description": None,
								"utype": None,
								"sourceRD": rd.sourceId,
								"dest": "keys",
							}
							for src, dst in zip(fkey.source, fkey.dest):
								yield {
									"key_id": fkeyId.lower(),
									"from_column": src.lower(),
									"target_column": dst.lower(),
									"sourceRD": rd.sourceId,
									"dest": "cols",
								}
				</code>
			</iterator>
		</embeddedGrammar>

		<!-- XXX TODO: We should let the DB do our work here by properly
			defining the fkey relationship between keys and key_columns -->

		<rowmaker id="build_keys" idmaps="*">
			<ignoreOn><not><keyIs key="dest" value="keys"/></not></ignoreOn>
		</rowmaker>

		<rowmaker id="build_key_columns" idmaps="*">
			<ignoreOn><not><keyIs key="dest" value="cols"/></not></ignoreOn>
		</rowmaker>

		<make table="keys" rowmaker="build_keys">
			<script original="removeStale"/>
		</make>
		<make table="key_columns" rowmaker="build_key_columns">
			<script original="removeStale"/>
		</make>
	</data>

	<data id="createSchema">
		<!-- run this to create the TAP schema initially, or to recreate it
		later.  To update the TAP schema contents after a bugfix or standards
		update, use gavo.user.upgrade._updateTAP_SCHEMA.
		 -->
		<make table="schemas"/>
		<make table="tables"/>
		<make table="columns"/>
		<make table="keys"/>
		<make table="groups"/>
		<make table="supportedmodels"/>
		<make table="key_columns">
			<!-- this script is for bootstrapping.  Since TAP_SCHEMA isn't
			finished when the tables are created, they cannot be added
			to them.  There's special code to let tap.py ignore them
			in that situation, and this code here adds them when TAP_SCHEMA
			is done.
			-->
			<script type="postCreation" lang="python"
					name="Add TAP_SCHEMA to TAP_SCHEMA" notify="False">
				from gavo.protocols import tap
				rd = table.tableDef.rd
				tap._insertRDIntoTAP_SCHEMA(rd, table.connection)
			</script>
		</make>
	</data>

	<data id="refresh" updating="True" auto="False">
		<sources items="refresh"/>
		<nullGrammar/>
		<make table="schemas">
			<script name="Refresh TAP schema" type="newSource" lang="python"
					notify="False">
				from gavo.protocols import tap

				toDo = [r[0] for r in
					table.connection.query(
						"SELECT DISTINCT sourceRD FROM tap_schema.tables")]

				for rdId in toDo:
					if rdId=="__system__/tap":
						# TAP_SCHEMA needs special handling because of bootstrapping
						tap._insertRDIntoTAP_SCHEMA(
							base.caches.getRD(rdId), table.connection)
						continue

					try:
						tap.publishToTAP(
							base.caches.getRD(rdId, doQueries=False), table.connection)
					except Exception as msg:
						base.ui.notifyWarning("RD %s couldn't be loaded or ingested to"
							" TAP_SCHEMA (%s).  Fix and run dachs imp -m on it to have"
							" up-to-date metadata in TAP_SCHEMA"%(rdId, msg))
						continue

			</script>
		</make>
	</data>


<!--********************* TAP UWS job table ******************-->

<table id="tapjobs" system="True">
	<meta name="description">
		A non-standard (and not tap-accessible) table used for managing
		asynchronous TAP jobs.  It is manipulated through TAP job creation
		and destruction internally.  Under very special circumstances, operators
		can use the gavo admin cleantap command to purge jobs from this table.

		Note that such jobs have corresponding directories in $STATEDIR/uwsjobs,
		which will be orphaned if this table is manipulated through SQL.
	</meta>
	<FEED source="//uws#uwsfields"/>
	<column name="pid" type="integer"
			description="A unix pid to kill to make the job stop">
		<values nullLiteral="-1"/>
	</column>
</table>

<data id="createJobTable">
	<make table="tapjobs"/>
</data>


<!--********************* The TAP Service *********************-->
	<service id="run" allowed="sync,async,dali,user_tables"
			defaultRenderer="info">
		<meta name="identifier">ivo://\getConfig{ivoa}{authority}/tap</meta>
		<meta name="shortName">\metaString{authority.shortName} TAP</meta>
		<meta name="title">\getConfig{web}{sitename} TAP service</meta>


		<meta name="_longdoc" format="rst"><![CDATA[
		This service speaks TAP, a protocol designed to allow the exchange of
		queries and data between clients (that's normally something running on your
		computer) and servers (e.g., us).

		You will want to use some sort of client to query TAP services;
		examples for those include:

		* TOPCAT_ (see in the "VO" menu)
		* `TAPHandle`_ (following this link should bring you to a page
		  that lets you query this server) works completely within your
		  browser,
		* pyVO_ (for VO support in python programs)
		
		You can, in a pinch, use our service in an XML-enabled browser, too.
		Under `Overview <#overview>`_, look for the bullet point on tap and
		follow the link to "this service". Then, click on "New job..."
		in the job list, enter your query, click "Set query", then
		"Execute query".  Reload the page you are redirected to now
		and then to see when your job is finished, then retrieve the result.

		The queries this service executes are written an a dialect of SQL called
		ADQL.  You need to learn it to use this service.  See
		`our ADQL tutorial`_.  Also do not miss the `local examples`_.
		
		By the way, for quick ad-hoc queries from within a browser,
		our `ADQL form service`_ may be more convenient than TAP.

		Also see the `table metadata`_ of the tables exposed here.

		Local information: This service may put a much lower limit on
		uploads in sync queries than advertised in the table metadata.
		Unfortunately, current TOPCAT versions don't give terribly helpful error
		messages when the server rejects a query because the attached upload is
		too large (something like “Error writing request body to server“).
		If that happens to you, just switch to asynchronous querying (in TOPCAT,
		that's in the “Mode” combo box just above the query editor).
		

		.. _TOPCAT: http://www.star.bris.ac.uk/~mbt/topcat/
		.. _GAVO VOTable library: http://soft.g-vo.org/subpkgs
		.. _TAP shell: http://soft.g-vo.org/tapsh
		.. _pyVO: https://github.com/pyvirtobs/pyvo
		.. _table metadata: \internallink{__system__/dc_tables/list/form}
		.. _ADQL form service: \internallink{__system__/adql/query/form}
		.. _service doc of our ADQL form service: \internallink{__system__/adql/query/info}
		.. _our ADQL tutorial: http://docs.g-vo.org/adql
		.. _local examples: \internallink{__system__/tap/run/examples}
		.. _TAPHandle: http://saada.unistra.fr/taphandle/?url=\internallink{/tap}


		Issues
		======

		For information on our ADQL implementation, see the
		\RSTservicelink{__system__/adql/query/info}{ADQL service info}.

		If multiple output columns in a query would end up having the same name,
		in the output VOTable they are disambiguated by appending underscores.
		This is in violation of the specification, but since fixing it would
		require rather intrusive changes into our software and it is not
		clear why one should want to use names when they are not unique to begin
		with, this will probably not be fixed.
		]]>
		</meta>
		<publish render="dali" sets="ivo_managed,local">
			<meta name="accessURL">\internallink{tap}</meta>
		</publish>

		<tapCore/>

		<FEED source="%#tapdescription"/>
		<FEED source="%#tapexamples"/>
	</service>
</resource>
