org.apache.spark.sql.connector.catalog

Class DelegatingCatalogExtension

Object

org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension

All Implemented Interfaces:

CatalogExtension, CatalogPlugin, SupportsNamespaces, TableCatalog

@Experimental
public abstract class DelegatingCatalogExtension
extends Object
implements CatalogExtension

A simple implementation of CatalogExtension, which implements all the catalog functions by calling the built-in session catalog directly. This is created for convenience, so that users only need to override some methods where they want to apply custom logic. For example, they can override createTable, do something else before calling super.createTable.

Field Summary

Fields inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

PROP_COMMENT, PROP_LOCATION, PROP_PROVIDER, RESERVED_PROPERTIES

Fields inherited from interface org.apache.spark.sql.connector.catalog.SupportsNamespaces

PROP_COMMENT, PROP_LOCATION, PROP_OWNER_NAME, PROP_OWNER_TYPE, RESERVED_PROPERTIES

Constructor Summary

Constructors

Constructor and Description
DelegatingCatalogExtension()

Method Summary

Modifier and Type	Method and Description
void	alterNamespace(String[] namespace, NamespaceChange... changes) Apply a set of metadata changes to a namespace in the catalog.
Table	alterTable(Identifier ident, TableChange... changes) Apply a set of changes to a table in the catalog.
void	createNamespace(String[] namespace, java.util.Map<String,String> metadata) Create a namespace in the catalog.
Table	createTable(Identifier ident, StructType schema, Transform[] partitions, java.util.Map<String,String> properties) Create a table in the catalog.
String[]	defaultNamespace() Return a default namespace for the catalog.
boolean	dropNamespace(String[] namespace) Drop a namespace from the catalog, recursively dropping all objects within the namespace.
boolean	dropTable(Identifier ident) Drop a table in the catalog.
void	initialize(String name, CaseInsensitiveStringMap options) Called to initialize configuration.
void	invalidateTable(Identifier ident) Invalidate cached table metadata for an identifier.
String[][]	listNamespaces() List top-level namespaces from the catalog.
String[][]	listNamespaces(String[] namespace) List namespaces in a namespace.
Identifier[]	listTables(String[] namespace) List the tables in a namespace from the catalog.
java.util.Map<String,String>	loadNamespaceMetadata(String[] namespace) Load metadata properties for a namespace.
Table	loadTable(Identifier ident) Load table metadata by identifier from the catalog.
String	name() Called to get this catalog's name.
boolean	namespaceExists(String[] namespace) Test whether a namespace exists.
void	renameTable(Identifier oldIdent, Identifier newIdent) Renames a table in the catalog.
void	setDelegateCatalog(CatalogPlugin delegate) This will be called only once by Spark to pass in the Spark built-in session catalog, after CatalogPlugin.initialize(String, CaseInsensitiveStringMap) is called.
boolean	tableExists(Identifier ident) Test whether a table exists using an identifier from the catalog.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DelegatingCatalogExtension

public DelegatingCatalogExtension()

Method Detail

setDelegateCatalog

public final void setDelegateCatalog(CatalogPlugin delegate)

Description copied from interface: CatalogExtension

This will be called only once by Spark to pass in the Spark built-in session catalog, after CatalogPlugin.initialize(String, CaseInsensitiveStringMap) is called.

Specified by:

setDelegateCatalog in interface CatalogExtension

name

public String name()

Description copied from interface: CatalogPlugin

Called to get this catalog's name.

This method is only called after CatalogPlugin.initialize(String, CaseInsensitiveStringMap) is called to pass the catalog's name.

Specified by:

name in interface CatalogPlugin

initialize

public final void initialize(String name,
                             CaseInsensitiveStringMap options)

Description copied from interface: CatalogPlugin

Called to initialize configuration.

This method is called once, just after the provider is instantiated.

Specified by:

initialize in interface CatalogPlugin

Parameters:

name - the name used to identify and load this catalog

options - a case-insensitive string map of configuration

listTables

public Identifier[] listTables(String[] namespace)
                        throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: TableCatalog

List the tables in a namespace from the catalog.

If the catalog supports views, this must return identifiers for only tables and not views.

Specified by:

listTables in interface TableCatalog

Parameters:

namespace - a multi-part namespace

Returns:

an array of Identifiers for tables

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the namespace does not exist (optional).

loadTable

public Table loadTable(Identifier ident)
                throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException

Description copied from interface: TableCatalog

Load table metadata by identifier from the catalog.

If the catalog supports views and contains a view for the identifier and not a table, this must throw NoSuchTableException.

Specified by:

loadTable in interface TableCatalog

Parameters:

ident - a table identifier

Returns:

the table's metadata

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchTableException - If the table doesn't exist or is a view

invalidateTable

public void invalidateTable(Identifier ident)

Description copied from interface: TableCatalog

Invalidate cached table metadata for an identifier.

If the table is already loaded or cached, drop cached data. If the table does not exist or is not cached, do nothing. Calling this method should not query remote services.

Specified by:

invalidateTable in interface TableCatalog

Parameters:

ident - a table identifier

tableExists

public boolean tableExists(Identifier ident)

Description copied from interface: TableCatalog

Test whether a table exists using an identifier from the catalog.

If the catalog supports views and contains a view for the identifier and not a table, this must return false.

Specified by:

tableExists in interface TableCatalog

Parameters:

ident - a table identifier

Returns:

true if the table exists, false otherwise

createTable

public Table createTable(Identifier ident,
                         StructType schema,
                         Transform[] partitions,
                         java.util.Map<String,String> properties)
                  throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException,
                         org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: TableCatalog

Create a table in the catalog.

Specified by:

createTable in interface TableCatalog

Parameters:

ident - a table identifier

schema - the schema of the new table, as a struct type

partitions - transforms to use for partitioning data in the table

properties - a string map of table properties

Returns:

metadata for the new table

Throws:

org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException - If a table or view already exists for the identifier

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the identifier namespace does not exist (optional)

alterTable

public Table alterTable(Identifier ident,
                        TableChange... changes)
                 throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException

Description copied from interface: TableCatalog

Apply a set of changes to a table in the catalog.

Implementations may reject the requested changes. If any change is rejected, none of the changes should be applied to the table.

If the catalog supports views and contains a view for the identifier and not a table, this must throw NoSuchTableException.

Specified by:

alterTable in interface TableCatalog

Parameters:

ident - a table identifier

changes - changes to apply to the table

Returns:

updated metadata for the table

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchTableException - If the table doesn't exist or is a view

dropTable

public boolean dropTable(Identifier ident)

Description copied from interface: TableCatalog

Drop a table in the catalog.

If the catalog supports views and contains a view for the identifier and not a table, this must not drop the view and must return false.

Specified by:

dropTable in interface TableCatalog

Parameters:

ident - a table identifier

Returns:

true if a table was deleted, false if no table exists for the identifier

renameTable

public void renameTable(Identifier oldIdent,
                        Identifier newIdent)
                 throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException,
                        org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException

Description copied from interface: TableCatalog

Renames a table in the catalog.

If the catalog supports views and contains a view for the old identifier and not a table, this throws NoSuchTableException. Additionally, if the new identifier is a table or a view, this throws TableAlreadyExistsException.

If the catalog does not support table renames between namespaces, it throws UnsupportedOperationException.

Specified by:

renameTable in interface TableCatalog

Parameters:

oldIdent - the table identifier of the existing table to rename

newIdent - the new table identifier of the table

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchTableException - If the table to rename doesn't exist or is a view

org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException - If the new table name already exists or is a view

defaultNamespace

public String[] defaultNamespace()

Description copied from interface: SupportsNamespaces

Return a default namespace for the catalog.

When this catalog is set as the current catalog, the namespace returned by this method will be set as the current namespace.

The namespace returned by this method is not required to exist.

Specified by:

defaultNamespace in interface SupportsNamespaces

Returns:

a multi-part namespace

listNamespaces

public String[][] listNamespaces()
                          throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: SupportsNamespaces

List top-level namespaces from the catalog.

If an object such as a table, view, or function exists, its parent namespaces must also exist and must be returned by this discovery method. For example, if table a.b.t exists, this method must return ["a"] in the result array.

Specified by:

listNamespaces in interface SupportsNamespaces

Returns:

an array of multi-part namespace names

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

listNamespaces

public String[][] listNamespaces(String[] namespace)
                          throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: SupportsNamespaces

List namespaces in a namespace.

If an object such as a table, view, or function exists, its parent namespaces must also exist and must be returned by this discovery method. For example, if table a.b.t exists, this method invoked as listNamespaces(["a"]) must return ["a", "b"] in the result array.

Specified by:

listNamespaces in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

Returns:

an array of multi-part namespace names

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the namespace does not exist (optional)

namespaceExists

public boolean namespaceExists(String[] namespace)

Description copied from interface: SupportsNamespaces

Test whether a namespace exists.

If an object such as a table, view, or function exists, its parent namespaces must also exist. For example, if table a.b.t exists, this method invoked as namespaceExists(["a"]) or namespaceExists(["a", "b"]) must return true.

Specified by:

namespaceExists in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

Returns:

true if the namespace exists, false otherwise

loadNamespaceMetadata

public java.util.Map<String,String> loadNamespaceMetadata(String[] namespace)
                                                   throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: SupportsNamespaces

Load metadata properties for a namespace.

Specified by:

loadNamespaceMetadata in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

Returns:

a string map of properties for the given namespace

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the namespace does not exist (optional)

createNamespace

public void createNamespace(String[] namespace,
                            java.util.Map<String,String> metadata)
                     throws org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException

Description copied from interface: SupportsNamespaces

Create a namespace in the catalog.

Specified by:

createNamespace in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

metadata - a string map of properties for the given namespace

Throws:

org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException - If the namespace already exists

alterNamespace

public void alterNamespace(String[] namespace,
                           NamespaceChange... changes)
                    throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: SupportsNamespaces

Apply a set of metadata changes to a namespace in the catalog.

Specified by:

alterNamespace in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

changes - a collection of changes to apply to the namespace

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the namespace does not exist (optional)

dropNamespace

public boolean dropNamespace(String[] namespace)
                      throws org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException

Description copied from interface: SupportsNamespaces

Drop a namespace from the catalog, recursively dropping all objects within the namespace.

If the catalog implementation does not support this operation, it may throw UnsupportedOperationException.

Specified by:

dropNamespace in interface SupportsNamespaces

Parameters:

namespace - a multi-part namespace

Returns:

true if the namespace was dropped

Throws:

org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException - If the namespace does not exist (optional)