Packages

trait RelationCatalog extends TableCatalog with ViewCatalog

Catalog API for connectors that expose both tables and views in a single shared identifier namespace.

Connectors that expose both tables and views must implement RelationCatalog; implementing TableCatalog and ViewCatalog directly without RelationCatalog is rejected at catalog initialization. Connectors that expose only tables implement just TableCatalog; connectors that expose only views implement just ViewCatalog; this interface is not relevant to them.

Two principles

A RelationCatalog follows two rules that, taken together, define every cross-cutting subtlety:

  • Orthogonal interfaces. Every TableCatalog method behaves as if views did not exist, and every ViewCatalog method behaves as if tables did not exist. From the perspective of a TableCatalog caller, a view at an identifier is indistinguishable from "nothing there"; symmetrically for ViewCatalog on tables. The implementation, of course, knows about both kinds -- it just filters them apart at each method boundary.
  • Single identifier namespace. Tables and views share one keyspace within a namespace; the same Identifier cannot resolve to both at the same time. The implementation typically enforces this with a single backing keyspace plus a kind discriminator.

Per-method cross-type behavior

Active rejection (write-side methods that throw on cross-type collision):

Cross-type rejection
MethodRejects whenThrows
`[[TableCatalog#createTable]]`a view sits at ident `[[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]]`
`[[TableCatalog#renameTable]]` a view sits at newIdent `[[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]]`
`[[ViewCatalog#createView]]`a table sits at ident `[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`
`[[ViewCatalog#createOrReplaceView]]`a table sits at ident `[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`
`[[ViewCatalog#replaceView]]`a table sits at ident `[[org.apache.spark.sql.catalyst.analysis.NoSuchViewException]]`
`[[ViewCatalog#renameView]]` a table sits at newIdent `[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`

Passive filtering (read / non-collision mutation methods that behave as if the wrong kind doesn't exist):

Cross-type filtering
MethodOn wrong-kind ident
`[[TableCatalog#loadTable(Identifier)]]` throws NoSuchTableException for a view
`[[TableCatalog#loadTable(Identifier, String)]]` / `[[TableCatalog#loadTable(Identifier, long)]]` throws NoSuchTableException for a view (no perf opt-in -- time-travel does not apply to views)
`[[TableCatalog#tableExists]]`returns false for a view
`[[TableCatalog#dropTable]]` / `[[TableCatalog#purgeTable]]` returns false for a view; does not drop it
`[[TableCatalog#renameTable]]` throws NoSuchTableException when the source is a view
`[[TableCatalog#listTables]]`tables only
`[[ViewCatalog#loadView]]` throws NoSuchViewException for a table
`[[ViewCatalog#viewExists]]`returns false for a table
`[[ViewCatalog#dropView]]` returns false for a table; does not drop it
`[[ViewCatalog#renameView]]` throws NoSuchViewException when the source is a table
`[[ViewCatalog#listViews]]`views only

Single-RPC perf entry points

The orthogonal TableCatalog and ViewCatalog answer two cross-cutting questions in two round trips each. RelationCatalog adds dedicated methods so a catalog can answer both in one round trip:

  • #loadRelation(Identifier) -- the resolver's per-identifier read path. Returns a Table for a table or a View for a view; callers discriminate via instanceof. Saves the loadTable -> loadView fallback on a cold cache.
  • #listRelationSummaries(String[]) -- a unified listing of tables and views with the kind preserved on each TableSummary. Default impl performs both TableCatalog#listTableSummaries and ViewCatalog#listViews; override to fetch in one round trip.
Annotations
@Evolving()
Source
RelationCatalog.java
Since

4.2.0

Linear Supertypes
ViewCatalog, TableCatalog, CatalogPlugin, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RelationCatalog
  2. ViewCatalog
  3. TableCatalog
  4. CatalogPlugin
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def alterTable(ident: Identifier, changes: <repeated...>[TableChange]): Table

    Apply a set of changes to a table in the catalog.

    Apply a set of changes to a table in the catalog.

    Implementations may reject the requested changes. If any change is rejected, none of the changes should be applied to the table.

    The requested changes must be applied in the order given.

    ident

    a table identifier

    changes

    changes to apply to the table

    returns

    updated metadata for the table. This can be null if getting the metadata for the updated table is expensive. Spark always discard the returned table here.

    Definition Classes
    TableCatalog
    Exceptions thrown

    IllegalArgumentException If any change is rejected by the implementation.

    NoSuchTableException If the table doesn't exist

  2. abstract def createView(ident: Identifier, info: View): View

    Create a view.

    Create a view.

    ident

    the view identifier

    info

    the view metadata

    returns

    the metadata of the newly created view; may equal info

    Definition Classes
    ViewCatalog
    Exceptions thrown

    NoSuchNamespaceException if the identifier's namespace does not exist (optional)

    ViewAlreadyExistsException if a view already exists at ident

  3. abstract def dropTable(ident: Identifier): Boolean

    Drop a table in the catalog.

    Drop a table in the catalog.

    ident

    a table identifier

    returns

    true if a table was deleted, false if no table exists for the identifier

    Definition Classes
    TableCatalog
  4. abstract def dropView(ident: Identifier): Boolean

    Drop a view.

    Drop a view.

    ident

    a view identifier

    returns

    true if a view was dropped, false otherwise

    Definition Classes
    ViewCatalog
  5. abstract def initialize(name: String, options: CaseInsensitiveStringMap): Unit

    Called to initialize configuration.

    Called to initialize configuration.

    This method is called once, just after the provider is instantiated.

    name

    the name used to identify and load this catalog

    options

    a case-insensitive string map of configuration

    Definition Classes
    CatalogPlugin
  6. abstract def listTables(namespace: Array[String]): Array[Identifier]

    List the tables in a namespace from the catalog.

    List the tables in a namespace from the catalog.

    namespace

    a multi-part namespace

    returns

    an array of Identifiers for tables

    Definition Classes
    TableCatalog
    Exceptions thrown

    NoSuchNamespaceException If the namespace does not exist (optional).

  7. abstract def listViews(namespace: Array[String]): Array[Identifier]

    List the views in a namespace from the catalog.

    List the views in a namespace from the catalog.

    namespace

    a multi-part namespace

    returns

    an array of identifiers for views

    Definition Classes
    ViewCatalog
    Exceptions thrown

    NoSuchNamespaceException if the namespace does not exist (optional)

  8. abstract def loadRelation(ident: Identifier): Relation

    Load the relation for an identifier that may resolve to either a table or a view.

    Load the relation for an identifier that may resolve to either a table or a view.

    Returns a Table for a table or a View for a view; callers discriminate via instanceof Table / instanceof View. This lets the resolver answer in a single RPC instead of falling back from TableCatalog#loadTable to ViewCatalog#loadView.

    ident

    the identifier

    returns

    a Table for tables, or a View for views

    Exceptions thrown

    NoSuchTableException if neither a table nor a view exists at ident

  9. abstract def name(): String

    Called to get this catalog's name.

    Called to get this catalog's name.

    This method is only called after #initialize(String,CaseInsensitiveStringMap) is called to pass the catalog's name.

    Definition Classes
    CatalogPlugin
  10. abstract def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit

    Renames a table in the catalog.

    Renames a table in the catalog.

    If the catalog does not support table renames between namespaces, it throws UnsupportedOperationException.

    oldIdent

    the table identifier of the existing table to rename

    newIdent

    the new table identifier of the table

    Definition Classes
    TableCatalog
    Exceptions thrown

    NoSuchTableException If the table to rename doesn't exist

    TableAlreadyExistsException If the new table name already exists

    UnsupportedOperationException If the namespaces of old and new identifiers do not match (optional)

  11. abstract def renameView(oldIdent: Identifier, newIdent: Identifier): Unit

    Rename a view.

    Rename a view.

    If the catalog supports tables and contains a table at the new identifier, this must throw ViewAlreadyExistsException. If the source identifier resolves to a table rather than a view, this must throw NoSuchViewException. The cross-type contract for catalogs that expose both tables and views lives on RelationCatalog.

    oldIdent

    the view identifier of the existing view to rename

    newIdent

    the new view identifier

    Definition Classes
    ViewCatalog
    Exceptions thrown

    NoSuchViewException if no view exists at oldIdent

    ViewAlreadyExistsException if a view (or, in a RelationCatalog, a table) already exists at newIdent

  12. abstract def replaceView(ident: Identifier, info: View): View

    Atomically replace an existing view's metadata.

    Atomically replace an existing view's metadata.

    Used by ALTER VIEW ... AS. Implementations should commit the new metadata atomically; views carry no data, so a single transactional metastore call (or equivalent) is sufficient -- there is no separate staging API.

    ident

    the view identifier

    info

    the new view metadata

    returns

    the metadata of the replaced view; may equal info

    Definition Classes
    ViewCatalog
    Exceptions thrown

    NoSuchViewException if no view exists at ident

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def capabilities(): Set[TableCatalogCapability]

    returns

    the set of capabilities for this TableCatalog

    Definition Classes
    TableCatalog
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. def createOrReplaceView(ident: Identifier, info: View): View

    Create a view if one does not exist at ident, or atomically replace it if one does.

    Create a view if one does not exist at ident, or atomically replace it if one does.

    Used by CREATE OR REPLACE VIEW. The default implementation calls #replaceView, falling back to #createView on NoSuchViewException. The fallback is non-atomic across the two calls (a concurrent drop or create can race), so catalogs that can answer the upsert in a single transactional call should override this method to collapse to one RPC and to make the swap atomic.

    ident

    the view identifier

    info

    the view metadata

    returns

    the metadata of the created or replaced view; may equal info

    Definition Classes
    ViewCatalog
    Exceptions thrown

    NoSuchNamespaceException if the identifier's namespace does not exist (optional)

    ViewAlreadyExistsException if ident cannot host this view -- either a concurrent CREATE VIEW won the race in the default impl's gap between #replaceView and the fallback #createView, or, in a RelationCatalog, a table sits at ident

  8. def createTable(ident: Identifier, tableInfo: TableInfo): Table

    Create a table in the catalog.

    Create a table in the catalog.

    ident

    a table identifier

    tableInfo

    information about the table

    returns

    metadata for the new table. This can be null if getting the metadata for the new table is expensive. Spark will call #loadTable(Identifier) if needed (e.g. CTAS).

    Definition Classes
    TableCatalog
    Since

    4.1.0

    Exceptions thrown

    NoSuchNamespaceException If the identifier namespace does not exist (optional)

    TableAlreadyExistsException If a table already exists for the identifier

    UnsupportedOperationException If a requested partition transform is not supported

  9. def createTableLike(ident: Identifier, tableInfo: TableInfo, sourceTable: Table): Table

    Create a table in the catalog by copying metadata from an existing source table.

    Create a table in the catalog by copying metadata from an existing source table.

    This method is called for CREATE TABLE ... LIKE ... statements targeting this catalog. The tableInfo parameter contains all the explicit information for the new table: columns and partitioning copied from the source, any constraints copied from the source, user-specified TBLPROPERTIES / LOCATION / USING provider (if given), and #PROP_OWNER set to the current user. Source table properties are intentionally excluded from tableInfo; connectors may read sourceTable.properties() to clone additional format-specific or custom state as appropriate for their implementation.

    The default implementation throws UnsupportedOperationException. Connectors that support CREATE TABLE ... LIKE ... must override this method.

    ident

    a table identifier for the new table

    tableInfo

    complete description of the new table: columns, partitioning, constraints, explicit properties (user overrides + owner); source table properties are NOT included

    sourceTable

    the resolved source table; connectors may read format-specific properties or other custom state from this object to clone additional metadata

    returns

    metadata for the new table

    Definition Classes
    TableCatalog
    Since

    4.2.0

    Exceptions thrown

    NoSuchNamespaceException If the identifier namespace does not exist (optional)

    TableAlreadyExistsException If a table already exists for the identifier

    UnsupportedOperationException If the catalog does not support CREATE TABLE LIKE

  10. def defaultNamespace(): Array[String]

    Return a default namespace for the catalog.

    Return a default namespace for the catalog.

    When this catalog is set as the current catalog, the namespace returned by this method will be set as the current namespace.

    The namespace returned by this method is not required to exist.

    returns

    a multi-part namespace

    Definition Classes
    CatalogPlugin
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  15. def invalidateTable(ident: Identifier): Unit

    Invalidate cached table metadata for an identifier.

    Invalidate cached table metadata for an identifier.

    If the table is already loaded or cached, drop cached data. If the table does not exist or is not cached, do nothing. Calling this method should not query remote services.

    ident

    a table identifier

    Definition Classes
    TableCatalog
  16. def invalidateView(ident: Identifier): Unit

    Invalidate cached metadata for a view.

    Invalidate cached metadata for a view.

    If the view is currently cached, drop the cached entry; otherwise do nothing. This must not issue remote calls.

    ident

    a view identifier

    Definition Classes
    ViewCatalog
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def listRelationSummaries(namespace: Array[String]): Array[TableSummary]

    List the tables and views in a namespace, returned as TableSummary entries with the kind preserved on each summary.

    List the tables and views in a namespace, returned as TableSummary entries with the kind preserved on each summary.

    The default implementation enumerates via TableCatalog#listTableSummaries for tables and ViewCatalog#listViews for views (two round trips). Catalogs that can fetch the unified listing in a single round trip should override.

    namespace

    a multi-part namespace

    returns

    an array of summaries for both tables and views in the namespace

    Exceptions thrown

    NoSuchNamespaceException if the namespace does not exist (optional)

    NoSuchTableException if a table listed by the underlying enumeration disappears before its summary can be assembled (default impl only)

  19. def listTableSummaries(namespace: Array[String]): Array[TableSummary]

    List the table summaries in a namespace from the catalog.

    List the table summaries in a namespace from the catalog.

    Returns one summary per entry returned by #listTables. Each TableSummary carries the entry's tableType.

    The default implementation enumerates via #listTables + #loadTable. Catalogs that can fetch summaries in a single round-trip should override.

    namespace

    a multi-part namespace

    returns

    an array of summaries for tables in the namespace

    Definition Classes
    TableCatalog
    Exceptions thrown

    NoSuchNamespaceException If the namespace does not exist (optional).

    NoSuchTableException If certain table listed by listTables API does not exist.

  20. def loadChangelog(ident: Identifier, context: ChangelogContext, options: CaseInsensitiveStringMap): Changelog

    Load a Changelog for the given table, representing the row-level changes within the range specified by context.

    Load a Changelog for the given table, representing the row-level changes within the range specified by context.

    The default implementation throws an analysis exception indicating that the catalog does not support CDC. Catalogs that support CDC must override this method.

    ident

    a table identifier

    context

    the CDC query context (range, deduplication mode, etc.)

    options

    all options passed to the changelog query, including the CDC-recognized keys (range, deduplication mode, etc.) that are also parsed into context

    returns

    a Changelog instance for the requested table and range

    Definition Classes
    TableCatalog
    Since

    4.2.0

    Exceptions thrown

    NoSuchTableException If the table doesn't exist

  21. def loadTable(ident: Identifier): Table

    Load table metadata by identifier from the catalog.

    Load table metadata by identifier from the catalog.

    The default implementation derives from #loadRelation: a View is rejected as not-a-table; a Table is returned. Override only if a tables-only path is materially cheaper than the unified one.

    ident

    a table identifier

    returns

    the table's metadata

    Definition Classes
    RelationCatalog → TableCatalog
    Annotations
    @Override()
  22. def loadTable(ident: Identifier, timestamp: Long): Table

    Load table metadata at a specific time by identifier from the catalog.

    Load table metadata at a specific time by identifier from the catalog.

    ident

    a table identifier

    timestamp

    timestamp of the table, which is microseconds since 1970-01-01 00:00:00 UTC

    returns

    the table's metadata

    Definition Classes
    TableCatalog
    Exceptions thrown

    NoSuchTableException If the table doesn't exist

  23. def loadTable(ident: Identifier, version: String): Table

    Load table metadata of a specific version by identifier from the catalog.

    Load table metadata of a specific version by identifier from the catalog.

    ident

    a table identifier

    version

    version of the table

    returns

    the table's metadata

    Definition Classes
    TableCatalog
    Exceptions thrown

    NoSuchTableException If the table doesn't exist

  24. def loadTable(ident: Identifier, writePrivileges: Set[TableWritePrivilege]): Table

    Load table metadata by identifier from the catalog.

    Load table metadata by identifier from the catalog. Spark will write data into this table later.

    ident

    a table identifier

    returns

    the table's metadata

    Definition Classes
    TableCatalog
    Since

    3.5.3

    Exceptions thrown

    NoSuchTableException If the table doesn't exist

  25. def loadView(ident: Identifier): View

    Load view metadata by identifier.

    Load view metadata by identifier.

    The default implementation derives from #loadRelation: a View is returned; anything else (table or absent) is surfaced as NoSuchViewException. Override only if a views-only path is materially cheaper than the unified one.

    ident

    a view identifier

    returns

    the view metadata

    Definition Classes
    RelationCatalog → ViewCatalog
    Annotations
    @Override()
  26. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  27. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  28. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  29. def purgeTable(ident: Identifier): Boolean

    Drop a table in the catalog and completely remove its data by skipping a trash even if it is supported.

    Drop a table in the catalog and completely remove its data by skipping a trash even if it is supported.

    If the catalog supports to purge a table, this method should be overridden. The default implementation throws UnsupportedOperationException.

    ident

    a table identifier

    returns

    true if a table was deleted, false if no table exists for the identifier

    Definition Classes
    TableCatalog
    Since

    3.1.0

    Exceptions thrown

    UnsupportedOperationException If table purging is not supported

  30. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  31. def tableExists(ident: Identifier): Boolean

    Test whether a table exists using an identifier from the catalog.

    Test whether a table exists using an identifier from the catalog.

    The default implementation derives from #loadRelation: returns true only if the entry exists and is a table. Override only if a cheaper existence-check path exists.

    ident

    a table identifier

    returns

    true if a table exists at ident, false otherwise

    Definition Classes
    RelationCatalog → TableCatalog
    Annotations
    @Override()
  32. def toString(): String
    Definition Classes
    AnyRef → Any
  33. def useNullableQuerySchema(): Boolean

    If true, mark all the fields of the query schema as nullable when executing CREATE/REPLACE TABLE ...

    If true, mark all the fields of the query schema as nullable when executing CREATE/REPLACE TABLE ... AS SELECT ... and creating the table.

    Definition Classes
    TableCatalog
  34. def viewExists(ident: Identifier): Boolean

    Test whether a view exists.

    Test whether a view exists.

    The default implementation calls #loadView and catches NoSuchViewException. Catalogs that can answer existence cheaply should override.

    The default implementation derives from #loadRelation: returns true only if the entry exists and is a view. Override only if a cheaper existence-check path exists.

    ident

    a view identifier

    returns

    true if a view exists at ident, false otherwise

    Definition Classes
    RelationCatalog → ViewCatalog
    Annotations
    @Override()
  35. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  36. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  37. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def createTable(ident: Identifier, columns: Array[Column], partitions: Array[Transform], properties: Map[String, String]): Table

    Create a table in the catalog.

    Create a table in the catalog.

    Definition Classes
    TableCatalog
    Annotations
    @Deprecated
    Deprecated

    (Since version 4.1.0)

  2. def createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table

    Create a table in the catalog.

    Create a table in the catalog.

    Definition Classes
    TableCatalog
    Annotations
    @Deprecated
    Deprecated

    (Since version 3.4.0)

  3. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ViewCatalog

Inherited from TableCatalog

Inherited from CatalogPlugin

Inherited from AnyRef

Inherited from Any

Ungrouped