Package org.apache.spark.sql.catalog

Class Catalog

Object

org.apache.spark.sql.api.Catalog<Dataset>

org.apache.spark.sql.catalog.Catalog

public abstract class Catalog
extends Catalog<Dataset>

Inheritdoc:

Constructor Summary

Constructors

Constructor	Description
Catalog()

Method Summary

Modifier and Type	Method	Description
Dataset<Row>	createExternalTable(String tableName, String path)	Creates a table from the given path and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String path, String source)	Creates a table from the given path based on a data source and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String source, Map<String,String> options)	Creates a table from the given path based on a data source and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, Map<String,String> options)	Create a table from the given path based on a data source, a schema and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Create a table from the given path based on a data source, a schema and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Creates a table from the given path based on a data source and a set of options.
abstract Dataset<Row>	createTable(String tableName, String path)	Creates a table from the given path and returns the corresponding DataFrame.
abstract Dataset<Row>	createTable(String tableName, String path, String source)	Creates a table from the given path based on a data source and returns the corresponding DataFrame.
Dataset<Row>	createTable(String tableName, String source, String description, Map<String,String> options)	Creates a table based on the dataset in a data source and a set of options.
abstract Dataset<Row>	createTable(String tableName, String source, String description, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Creates a table based on the dataset in a data source and a set of options.
Dataset<Row>	createTable(String tableName, String source, Map<String,String> options)	Creates a table based on the dataset in a data source and a set of options.
Dataset<Row>	createTable(String tableName, String source, StructType schema, String description, Map<String,String> options)	Create a table based on the dataset in a data source, a schema and a set of options.
abstract Dataset<Row>	createTable(String tableName, String source, StructType schema, String description, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Create a table based on the dataset in a data source, a schema and a set of options.
Dataset<Row>	createTable(String tableName, String source, StructType schema, Map<String,String> options)	Create a table based on the dataset in a data source, a schema and a set of options.
abstract Dataset<Row>	createTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Create a table based on the dataset in a data source, a schema and a set of options.
abstract Dataset<Row>	createTable(String tableName, String source, scala.collection.immutable.Map<String,String> options)	(Scala-specific) Creates a table based on the dataset in a data source and a set of options.
abstract Dataset<CatalogMetadata>	listCatalogs()	Returns a list of catalogs available in this session.
abstract Dataset<CatalogMetadata>	listCatalogs(String pattern)	Returns a list of catalogs which name match the specify pattern and available in this session.
abstract Dataset<Column>	listColumns(String tableName)	Returns a list of columns for the given table/view or temporary view.
abstract Dataset<Column>	listColumns(String dbName, String tableName)	Returns a list of columns for the given table/view in the specified database under the Hive Metastore.
abstract Dataset<Database>	listDatabases()	Returns a list of databases (namespaces) available within the current catalog.
abstract Dataset<Database>	listDatabases(String pattern)	Returns a list of databases (namespaces) which name match the specify pattern and available within the current catalog.
abstract Dataset<Function>	listFunctions()	Returns a list of functions registered in the current database (namespace).
abstract Dataset<Function>	listFunctions(String dbName)	Returns a list of functions registered in the specified database (namespace) (the name can be qualified with catalog).
abstract Dataset<Function>	listFunctions(String dbName, String pattern)	Returns a list of functions registered in the specified database (namespace) which name match the specify pattern (the name can be qualified with catalog).
abstract Dataset<Table>	listTables()	Returns a list of tables/views in the current database (namespace).
abstract Dataset<Table>	listTables(String dbName)	Returns a list of tables/views in the specified database (namespace) (the name can be qualified with catalog).
abstract Dataset<Table>	listTables(String dbName, String pattern)	Returns a list of tables/views in the specified database (namespace) which name match the specify pattern (the name can be qualified with catalog).

Methods inherited from class org.apache.spark.sql.api.Catalog

cacheTable, cacheTable, clearCache, currentCatalog, currentDatabase, databaseExists, dropGlobalTempView, dropTempView, functionExists, functionExists, getDatabase, getFunction, getFunction, getTable, getTable, isCached, recoverPartitions, refreshByPath, refreshTable, setCurrentCatalog, setCurrentDatabase, tableExists, tableExists, uncacheTable

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Catalog

public Catalog()

Method Details

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String path)

Description copied from class: Catalog

Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

path - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String path,
 String source)

Description copied from class: Catalog

Creates a table from the given path based on a data source and returns the corresponding DataFrame.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String source,
 Map<String,String> options)

Description copied from class: Catalog

Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String source,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String source,
 StructType schema,
 Map<String,String> options)

Description copied from class: Catalog

Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
 String source,
 StructType schema,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createExternalTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String path)

Description copied from class: Catalog

Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

path - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String path,
 String source)

Description copied from class: Catalog

Creates a table from the given path based on a data source and returns the corresponding DataFrame.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String source,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String source,
 String description,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

description - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String source,
 StructType schema,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public abstract Dataset<Row> createTable(String tableName,
 String source,
 StructType schema,
 String description,
 scala.collection.immutable.Map<String,String> options)

Description copied from class: Catalog

(Scala-specific) Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

description - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public Dataset<Row> createTable(String tableName,
 String source,
 Map<String,String> options)

Description copied from class: Catalog

Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public Dataset<Row> createTable(String tableName,
 String source,
 String description,
 Map<String,String> options)

Description copied from class: Catalog

Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

description - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public Dataset<Row> createTable(String tableName,
 String source,
 StructType schema,
 Map<String,String> options)

Description copied from class: Catalog

Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

createTable

public Dataset<Row> createTable(String tableName,
 String source,
 StructType schema,
 String description,
 Map<String,String> options)

Description copied from class: Catalog

Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Overrides:

createTable in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

source - (undocumented)

schema - (undocumented)

description - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listCatalogs

public abstract Dataset<CatalogMetadata> listCatalogs()

Description copied from class: Catalog

Returns a list of catalogs available in this session.

Specified by:

listCatalogs in class Catalog<Dataset>

Returns:

(undocumented)

Inheritdoc:

listCatalogs

public abstract Dataset<CatalogMetadata> listCatalogs(String pattern)

Description copied from class: Catalog

Returns a list of catalogs which name match the specify pattern and available in this session.

Specified by:

listCatalogs in class Catalog<Dataset>

Parameters:

pattern - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listColumns

public abstract Dataset<Column> listColumns(String tableName)

Description copied from class: Catalog

Returns a list of columns for the given table/view or temporary view.

Specified by:

listColumns in class Catalog<Dataset>

Parameters:

tableName - is either a qualified or unqualified name that designates a table/view. It follows the same resolution rule with SQL: search for temp views first then table/views in the current database (namespace).

Returns:

(undocumented)

Inheritdoc:

listColumns

public abstract Dataset<Column> listColumns(String dbName,
 String tableName)

Description copied from class: Catalog

Returns a list of columns for the given table/view in the specified database under the Hive Metastore.

To list columns for table/view in other catalogs, please use listColumns(tableName) with qualified table/view name instead.

Specified by:

listColumns in class Catalog<Dataset>

Parameters:

dbName - is an unqualified name that designates a database.

tableName - is an unqualified name that designates a table/view.

Returns:

(undocumented)

Inheritdoc:

listDatabases

public abstract Dataset<Database> listDatabases()

Description copied from class: Catalog

Returns a list of databases (namespaces) available within the current catalog.

Specified by:

listDatabases in class Catalog<Dataset>

Returns:

(undocumented)

Inheritdoc:

listDatabases

public abstract Dataset<Database> listDatabases(String pattern)

Description copied from class: Catalog

Returns a list of databases (namespaces) which name match the specify pattern and available within the current catalog.

Specified by:

listDatabases in class Catalog<Dataset>

Parameters:

pattern - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listFunctions

public abstract Dataset<Function> listFunctions()

Description copied from class: Catalog

Returns a list of functions registered in the current database (namespace). This includes all temporary functions.

Specified by:

listFunctions in class Catalog<Dataset>

Returns:

(undocumented)

Inheritdoc:

listFunctions

public abstract Dataset<Function> listFunctions(String dbName)

Description copied from class: Catalog

Returns a list of functions registered in the specified database (namespace) (the name can be qualified with catalog). This includes all built-in and temporary functions.

Specified by:

listFunctions in class Catalog<Dataset>

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listFunctions

public abstract Dataset<Function> listFunctions(String dbName,
 String pattern)

Description copied from class: Catalog

Returns a list of functions registered in the specified database (namespace) which name match the specify pattern (the name can be qualified with catalog). This includes all built-in and temporary functions.

Specified by:

listFunctions in class Catalog<Dataset>

Parameters:

dbName - (undocumented)

pattern - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listTables

public abstract Dataset<Table> listTables()

Description copied from class: Catalog

Returns a list of tables/views in the current database (namespace). This includes all temporary views.

Specified by:

listTables in class Catalog<Dataset>

Returns:

(undocumented)

Inheritdoc:

listTables

public abstract Dataset<Table> listTables(String dbName)

Description copied from class: Catalog

Returns a list of tables/views in the specified database (namespace) (the name can be qualified with catalog). This includes all temporary views.

Specified by:

listTables in class Catalog<Dataset>

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Inheritdoc:

listTables

public abstract Dataset<Table> listTables(String dbName,
 String pattern)

Description copied from class: Catalog

Returns a list of tables/views in the specified database (namespace) which name match the specify pattern (the name can be qualified with catalog). This includes all temporary views.

Specified by:

listTables in class Catalog<Dataset>

Parameters:

dbName - (undocumented)

pattern - (undocumented)

Returns:

(undocumented)

Inheritdoc: