catalog

package catalog

Package Members

package constraints
package functions
package index
package procedures
package transactions

Type Members

trait CatalogExtension extends TableCatalog with FunctionCatalog with SupportsNamespaces
An API to extend the Spark built-in session catalog.
An API to extend the Spark built-in session catalog. Implementation can get the built-in session catalog from #setDelegateCatalog(CatalogPlugin), implement catalog functions with some custom logic and call the built-in session catalog at the end. For example, they can implement createTable, do something else before calling createTable of the built-in session catalog.
Annotations
@Evolving()
Since
3.0.0
class CatalogNotFoundException extends SparkException
Annotations
@Experimental()
trait CatalogPlugin extends AnyRef
A marker interface to provide a catalog implementation for Spark.
A marker interface to provide a catalog implementation for Spark.
Implementations can provide catalog functions by implementing additional interfaces for tables, views, and functions.
Catalog implementations must implement this marker interface to be loaded by org.apache.spark.sql.connector.catalog.Catalogs#load(String,SQLConf). The loader will instantiate catalog classes using the required public no-arg constructor. After creating an instance, it will be configured by calling #initialize(String,CaseInsensitiveStringMap).
Catalog implementations are registered to a name by adding a configuration option to Spark: spark.sql.catalog.catalog-name=com.example.YourCatalogClass. All configuration properties in the Spark configuration that share the catalog name prefix, spark.sql.catalog.catalog-name.(key)=(value) will be passed in the case insensitive string map of options in initialization with the prefix removed. name, is also passed and is the catalog's name; in this case, "catalog-name".
Annotations
@Evolving()
Since
3.0.0
trait Changelog extends AnyRef
The central connector interface for Change Data Capture (CDC).
The central connector interface for Change Data Capture (CDC).
Connectors implement this minimal interface to expose change data. Spark handles post-processing (carry-over removal, update detection, net change computation) based on the properties declared by the connector.
The columns returned by #columns() must include the following metadata columns:
- _change_type (STRING) — the kind of change: insert, delete, update_preimage, or update_postimage
- _commit_version — the commit version containing this change. Must be either LongType or StringType; all other types are rejected. The column's natural ordering (numeric for LongType, lexicographic for StringType) must match commit order, because the netChanges post-processing path sorts rows of a given row identity by this column to determine the first and last events.
- _commit_timestamp (TIMESTAMP) -- the timestamp of the commit. All rows belonging to a single _commit_version must share the same _commit_timestamp. For streaming reads with post-processing enabled, two additional requirements apply:
- All rows of a single commit must appear in the same micro-batch (i.e. micro-batch boundaries align with commit boundaries).
- Each micro-batch's rows must have _commit_timestamp strictly greater than the maximum _commit_timestamp of any prior micro-batch.
Streaming post-processing uses _commit_timestamp as event time with a zero-delay watermark, so once a micro-batch observes max event time T the global watermark advances to T. Both Spark's late-event filter and its state-eviction predicate then use eventTime <= T -- so any later row at _commit_timestamp <= T (whether from the same commit split across batches, a different commit emitted later, or simply an out-of-order commit) is silently dropped as late. Requirement 1 keeps a single commit's rows together; requirement 2 keeps distinct commits in strictly increasing event-time order across batches. Multiple distinct commits with equal _commit_timestamp are allowed within a single micro-batch -- only across batches does timestamp progression need to be strictly increasing. Atomic-commit CDC connectors (e.g. Delta versions, Iceberg snapshots) that derive _commit_timestamp from wall-clock time at commit time naturally satisfy both requirements. _commit_timestamp must be non-NULL on every row of a streaming read engaging post-processing; both the row-level Aggregate path and the netChanges transformWithState path raise CHANGELOG_CONTRACT_VIOLATION.NULL_COMMIT_TIMESTAMP on a violation
Streaming reads support carry-over removal, update detection, and net change computation. Two streaming-specific behaviors to be aware of:
- Output is buffered until the watermark advances past the commit. When a micro-batch ingests a commit, that commit's output rows are buffered in state and not emitted in the same batch. They are emitted by a later micro-batch -- whichever one advances the watermark past the commit's _commit_timestamp. The last commit's output is emitted when the source terminates.
- netChanges only merges changes that are buffered together. When each row identity appears in at most one commit within any buffered window, the streaming output is the same as computeUpdates. Cross-commit merging only happens when several commits touch the same row before the earliest one's output has been released. For full-range collapse, use a batch read.
Pushdown contract. When any post-processing pass applies (carry-over removal, update detection, or netChanges), Spark only pushes predicates that reference _commit_version, _commit_timestamp, or columns named by #rowId() to the connector's org.apache.spark.sql.connector.read.SupportsPushDownFilters / org.apache.spark.sql.connector.read.SupportsPushDownV2Filters. Predicates on _change_type, the #rowVersion() column, or non-rowId data columns are kept above the scan: pushing them would drop one half of a delete/insert pair within a row-identity group and silently break post-processing. Catalyst's pushdown rules enforce this via the rewrite operators, so connectors do not need to code the restriction themselves -- but must not bypass it via connector-specific options. When no post-processing pass applies, Spark does not impose any CDC-specific predicate-pushdown restriction. org.apache.spark.sql.connector.read.SupportsPushDownRequiredColumns (column pruning) is unrestricted in either case: Spark's pruning already respects what the rewrite operators reference.
Annotations
@Evolving()
Since
4.2.0
class ChangelogContext extends AnyRef
Encapsulates the parameters of a Change Data Capture (CDC) query, passed from the parser / DataFrame API to the catalog's ChangelogContext, CaseInsensitiveStringMap) method.
Encapsulates the parameters of a Change Data Capture (CDC) query, passed from the parser / DataFrame API to the catalog's ChangelogContext, CaseInsensitiveStringMap) method.
Annotations
@Evolving()
Since
4.2.0
sealed trait ChangelogRange extends AnyRef
Represents the version or timestamp range for a Change Data Capture (CDC) query.
Represents the version or timestamp range for a Change Data Capture (CDC) query.
This sealed interface has three implementations:
- VersionRange — range defined by version identifiers
- TimestampRange — range defined by timestamps
- UnboundedRange — no boundaries (used by streaming queries)
Annotations
@Evolving()
Since
4.2.0
trait Column extends AnyRef
An interface representing a column of a Table.
An interface representing a column of a Table. It defines basic properties of a column, such as name and data type, as well as some advanced ones like default column value.
Data Sources do not need to implement it. They should consume it in APIs like Column[], Transform[], Map), and report it in Table#columns() by calling the static create functions of this interface to create it.
A column cannot have both a default value and a generation expression.
Annotations
@Evolving()
class ColumnDefaultValue extends DefaultValue
A class representing the default value of a column.
A class representing the default value of a column. It contains both the SQL string and literal value of the user-specified default value expression. The SQL string should be re-evaluated for each table writing command, which may produce different values if the default value expression is something like CURRENT_DATE(). The literal value is used to back-fill existing data if new columns with default value are added. Note: the back-fill can be lazy. The data sources can remember the column default value and let the reader fill the column value when reading existing data that do not have these new columns.
Annotations
@Evolving()
class DefaultValue extends AnyRef
A class that represents default values.
A class that represents default values.
Connectors can define default values using either a SQL string (Spark SQL dialect) or an expression if the default value can be expressed as a supported connector expression. If both the SQL string and the expression are provided, Spark first attempts to convert the given expression to its internal representation. If the expression cannot be converted, and a SQL string is provided, Spark will fall back to parsing the SQL string.
Annotations
@Evolving()
Since
4.1.0
abstract class DelegatingCatalogExtension extends CatalogExtension
A simple implementation of CatalogExtension, which implements all the catalog functions by calling the built-in session catalog directly.
A simple implementation of CatalogExtension, which implements all the catalog functions by calling the built-in session catalog directly. This is created for convenience, so that users only need to override some methods where they want to apply custom logic. For example, they can override createTable, do something else before calling super.createTable.
Annotations
@Evolving()
Since
3.0.0
class DelegatingTable extends Table
A concrete Table that adapts a TableInfo -- it contains only table metadata and defers read/write to Spark, which resolves the table provider into a data source at read time.
A concrete Table that adapts a TableInfo -- it contains only table metadata and defers read/write to Spark, which resolves the table provider into a data source at read time.
Catalogs build the metadata via TableInfo.Builder and return a DelegatingTable from TableCatalog#loadTable(Identifier) (or RelationCatalog#loadRelation for a data-source table) when they want Spark to handle the underlying source. A catalog that has its own Table object returns that instead. Views are never represented as a DelegatingTable: a view is a View, which is itself a Relation.
Annotations
@Evolving()
Since
4.2.0
sealed trait Dependency extends AnyRef
Represents a dependency of a SQL object such as a view or metric view.
Represents a dependency of a SQL object such as a view or metric view.
A dependency is one of: TableDependency or FunctionDependency. The sealed declaration enforces this structurally.
Note: today the only producer in Spark itself is metric-view dependency extraction, which emits TableDependency only. FunctionDependency and the #function(String[]) factory are exposed as groundwork for future producers (e.g. SQL UDF dependency tracking); consumers iterating a DependencyList received from Spark today should expect to see only TableDependency instances.
Annotations
@Evolving()
Since
4.2.0
final class DependencyList extends Record
A list of dependencies for a SQL object such as a view or metric view.
A list of dependencies for a SQL object such as a view or metric view.
- When null, the dependency information is not provided.
- When the array is empty, dependencies are provided but the object has none.
- When the array is non-empty, each entry describes one dependency.
Records' auto-generated equals/hashCode on array fields fall through to Object#equals (reference equality), so this record overrides them to use Object[]) / Arrays#hashCode(Object[]) on dependencies; per-element equality delegates to the element's overridden equals (TableDependency / FunctionDependency both implement value semantics on their nameParts array). The defensive-copy accessor override clones on read so callers cannot mutate the record's internal array.
Annotations
@Evolving()
Since
4.2.0
trait FunctionCatalog extends CatalogPlugin
Catalog methods for working with Functions.
Catalog methods for working with Functions.
Annotations
@Evolving()
Since
3.2.0
final class FunctionDependency extends Record with Dependency
A function dependency of a SQL object.
A function dependency of a SQL object.
The dependent function is identified by its structural multi-part name. See TableDependency for the parts-form contract.
Records' auto-generated equals/hashCode on array fields fall through to Object#equals (reference equality), so this record overrides them to use Object[]) / Arrays#hashCode(Object[]) on nameParts and give value-based semantics. The defensive-copy accessor override also clones on read so callers cannot mutate the record's internal array.
Annotations
@Evolving()
Since
4.2.0
trait Identifier extends AnyRef
Identifies an object in a catalog.
Identifies an object in a catalog.
Annotations
@Evolving()
Since
3.0.0
class IdentityColumnSpec extends AnyRef
Identity column specification.
Identity column specification.
Annotations
@Evolving()
trait MetadataColumn extends AnyRef
Interface for a metadata column.
Interface for a metadata column.
A metadata column can expose additional metadata about a row. For example, rows from Kafka can use metadata columns to expose a message's topic, partition number, and offset.
A metadata column could also be the result of a transform applied to a value in the row. For example, a partition value produced by bucket(id, 16) could be exposed by a metadata column. In this case, #transform() should return a non-null Transform that produced the metadata column's values.
Annotations
@Evolving()
Since
3.1.0
trait NamespaceChange extends AnyRef
NamespaceChange subclasses represent requested changes to a namespace.
NamespaceChange subclasses represent requested changes to a namespace. These are passed to SupportsNamespaces#alterNamespace. For example,
```
  import NamespaceChange._
  val catalog = Catalogs.load(name)
  catalog.alterNamespace(ident,
      setProperty("prop", "value"),
      removeProperty("other_prop")
    )
```
Annotations
@Evolving()
Since
3.0.0
trait ProcedureCatalog extends CatalogPlugin
A catalog API for working with procedures.
A catalog API for working with procedures.
Annotations
@Evolving()
Since
4.0.0
trait Relation extends AnyRef
A relation in a catalog: either a Table or a View.
A relation in a catalog: either a Table or a View. This is the common type returned by RelationCatalog#loadRelation so a catalog that exposes both kinds can answer a single read in one round trip; callers discriminate with instanceof Table / instanceof View.
The two kinds are deliberately asymmetric, mirroring how Spark treats them: a Table is an object Spark reads from and writes to, while a View carries only metadata (Spark expands its query text at read time and never builds a view object). Modeling both as siblings of Relation -- rather than smuggling a view through the Table surface -- keeps table-only concepts (partitioning, constraints, scans, writes) off the view side.
In practice the only two kinds are Table and View. Relation is left un-sealed because Table is itself an open interface (so a closed hierarchy would add little) and because a sealed Java interface trips Scala's pattern-match analysis.
Annotations
@Evolving()
Since
4.2.0

trait RelationCatalog extends TableCatalog with ViewCatalog

Catalog API for connectors that expose both tables and views in a single shared identifier namespace.

Connectors that expose both tables and views must implement RelationCatalog; implementing TableCatalog and ViewCatalog directly without RelationCatalog is rejected at catalog initialization. Connectors that expose only tables implement just TableCatalog; connectors that expose only views implement just ViewCatalog; this interface is not relevant to them.

Two principles

A RelationCatalog follows two rules that, taken together, define every cross-cutting subtlety:

Orthogonal interfaces. Every TableCatalog method behaves as if views did not exist, and every ViewCatalog method behaves as if tables did not exist. From the perspective of a TableCatalog caller, a view at an identifier is indistinguishable from "nothing there"; symmetrically for ViewCatalog on tables. The implementation, of course, knows about both kinds -- it just filters them apart at each method boundary.
Single identifier namespace. Tables and views share one keyspace within a namespace; the same Identifier cannot resolve to both at the same time. The implementation typically enforces this with a single backing keyspace plus a kind discriminator.

Per-method cross-type behavior

Active rejection (write-side methods that throw on cross-type collision):

Cross-type rejection
Method	Rejects when	Throws
`[[TableCatalog#createTable]]`	a view sits at `ident`	`[[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]]`
`[[TableCatalog#renameTable]]`	a view sits at `newIdent`	`[[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]]`
`[[ViewCatalog#createView]]`	a table sits at `ident`	`[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`
`[[ViewCatalog#createOrReplaceView]]`	a table sits at `ident`	`[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`
`[[ViewCatalog#replaceView]]`	a table sits at `ident`	`[[org.apache.spark.sql.catalyst.analysis.NoSuchViewException]]`
`[[ViewCatalog#renameView]]`	a table sits at `newIdent`	`[[org.apache.spark.sql.catalyst.analysis.ViewAlreadyExistsException]]`

Passive filtering (read / non-collision mutation methods that behave as if the wrong kind doesn't exist):

Cross-type filtering
Method	On wrong-kind ident
`[[TableCatalog#loadTable(Identifier)]]`	throws `NoSuchTableException` for a view
`[[TableCatalog#loadTable(Identifier, String)]]` / `[[TableCatalog#loadTable(Identifier, long)]]`	throws `NoSuchTableException` for a view (no perf opt-in -- time-travel does not apply to views)
`[[TableCatalog#tableExists]]`	returns `false` for a view
`[[TableCatalog#dropTable]]` / `[[TableCatalog#purgeTable]]`	returns `false` for a view; does not drop it
`[[TableCatalog#renameTable]]`	throws `NoSuchTableException` when the source is a view
`[[TableCatalog#listTables]]`	tables only
`[[ViewCatalog#loadView]]`	throws `NoSuchViewException` for a table
`[[ViewCatalog#viewExists]]`	returns `false` for a table
`[[ViewCatalog#dropView]]`	returns `false` for a table; does not drop it
`[[ViewCatalog#renameView]]`	throws `NoSuchViewException` when the source is a table
`[[ViewCatalog#listViews]]`	views only

Single-RPC perf entry points

The orthogonal TableCatalog and ViewCatalog answer two cross-cutting questions in two round trips each. RelationCatalog adds dedicated methods so a catalog can answer both in one round trip:

#loadRelation(Identifier) -- the resolver's per-identifier read path. Returns a Table for a table or a View for a view; callers discriminate via instanceof. Saves the loadTable -> loadView fallback on a cold cache.
#listRelationSummaries(String[]) -- a unified listing of tables and views with the kind preserved on each TableSummary. Default impl performs both TableCatalog#listTableSummaries and ViewCatalog#listViews; override to fetch in one round trip.

Annotations: @Evolving()
Since: 4.2.0

trait SessionConfigSupport extends TableProvider
A mix-in interface for TableProvider.
A mix-in interface for TableProvider. Data sources can implement this interface to propagate session configs with the specified key-prefix to all data source operations in this session.
Annotations
@Evolving()
Since
3.0.0
trait StagedTable extends Table
Represents a table which is staged for being committed to the metastore.
Represents a table which is staged for being committed to the metastore.
This is used to implement atomic CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT queries. The planner will create one of these via StructType, Transform[], Map) or StructType, Transform[], Map) to prepare the table for being written to. This table should usually implement SupportsWrite. A new writer will be constructed via SupportsWrite#newWriteBuilder(LogicalWriteInfo), and the write will be committed. The job concludes with a call to #commitStagedChanges(), at which point implementations are expected to commit the table's metadata into the metastore along with the data that was written by the writes from the write builder this table created.
Annotations
@Evolving()
Since
3.0.0
trait StagingTableCatalog extends TableCatalog
An optional mix-in for implementations of TableCatalog that support staging creation of a table before committing the table's metadata along with its contents in CREATE TABLE AS SELECT or REPLACE TABLE AS SELECT operations.
An optional mix-in for implementations of TableCatalog that support staging creation of a table before committing the table's metadata along with its contents in CREATE TABLE AS SELECT or REPLACE TABLE AS SELECT operations.
It is highly recommended to implement this trait whenever possible so that CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT operations are atomic. For example, when one runs a REPLACE TABLE AS SELECT operation, if the catalog does not implement this trait, the planner will first drop the table via TableCatalog#dropTable(Identifier), then create the table via TableInfo), and then perform the write via SupportsWrite#newWriteBuilder(LogicalWriteInfo). However, if the write operation fails, the catalog will have already dropped the table, and the planner cannot roll back the dropping of the table.
If the catalog implements this plugin, the catalog can implement the methods to "stage" the creation and the replacement of a table. After the table's BatchWrite#commit(WriterCommitMessage[]) is called, StagedTable#commitStagedChanges() is called, at which point the staged table can complete both the data write and the metadata swap operation atomically.
Annotations
@Evolving()
Since
3.0.0
trait SupportsAtomicPartitionManagement extends SupportsPartitionManagement
An atomic partition interface of Table to operate multiple partitions atomically.
An atomic partition interface of Table to operate multiple partitions atomically.
These APIs are used to modify table partition or partition metadata, they will change the table data as well.
- #createPartitions: add an array of partitions and any data they contain to the table
- #dropPartitions: remove an array of partitions and any data they contain from the table
- #purgePartitions: remove an array of partitions and any data they contain from the table by skipping a trash even if it is supported
- #truncatePartitions: truncate an array of partitions by removing partitions data
Annotations
@Experimental()
Since
3.1.0
trait SupportsCatalogOptions extends TableProvider
An interface, which TableProviders can implement, to support table existence checks and creation through a catalog, without having to use table identifiers.
An interface, which TableProviders can implement, to support table existence checks and creation through a catalog, without having to use table identifiers. For example, when file based data sources use the DataFrameWriter.save(path) method, the option path can translate to a PathIdentifier. A catalog can then use this PathIdentifier to check the existence of a table, or whether a table can be created at a given directory.
Annotations
@Evolving()
Since
3.0.0
trait SupportsDelete extends SupportsDeleteV2
A mix-in interface for Table delete support.
A mix-in interface for Table delete support. Data sources can implement this interface to provide the ability to delete data from tables that matches filter expressions.
Annotations
@Evolving()
Since
3.0.0
trait SupportsDeleteV2 extends TruncatableTable
A mix-in interface for Table delete support.
A mix-in interface for Table delete support. Data sources can implement this interface to provide the ability to delete data from tables that matches filter expressions.
Annotations
@Evolving()
Since
3.4.0
trait SupportsMetadataColumns extends Table
An interface for exposing data columns for a table that are not in the table schema.
An interface for exposing data columns for a table that are not in the table schema. For example, a file source could expose a "file" column that contains the path of the file that contained each row.
The columns returned by #metadataColumns() may be passed as StructField in requested projections. Sources that implement this interface and column projection using SupportsPushDownRequiredColumns must accept metadata fields passed to SupportsPushDownRequiredColumns#pruneColumns(StructType).
If a table column and a metadata column have the same name, the conflict is resolved by either renaming or suppressing the metadata column. See canRenameConflictingMetadataColumns.
Annotations
@Evolving()
Since
3.1.0
trait SupportsNamespaces extends CatalogPlugin
Catalog methods for working with namespaces.
Catalog methods for working with namespaces.
If an object such as a table, view, or function exists, its parent namespaces must also exist and must be returned by the discovery methods #listNamespaces() and #listNamespaces(String[]).
Catalog implementations are not required to maintain the existence of namespaces independent of objects in a namespace. For example, a function catalog that loads functions using reflection and uses Java packages as namespaces is not required to support the methods to create, alter, or drop a namespace. Implementations are allowed to discover the existence of objects or namespaces without throwing NoSuchNamespaceException when no namespace is found.
Annotations
@Evolving()
Since
3.0.0
trait SupportsPartitionManagement extends Table
A partition interface of Table.
A partition interface of Table. A partition is composed of identifier and properties, and properties contains metadata information of the partition.
These APIs are used to modify table partition identifier or partition metadata. In some cases, they will change the table data as well.
- #createPartition: add a partition and any data it contains to the table
- #dropPartition: remove a partition and any data it contains from the table
- #purgePartition: remove a partition and any data it contains from the table by skipping a trash even if it is supported.
- #replacePartitionMetadata: point a partition to a new location, which will swap one location's data for the other
- #truncatePartition: remove partition data from the table
Annotations
@Experimental()
Since
3.1.0
trait SupportsRead extends Table
A mix-in interface of Table, to indicate that it's readable.
A mix-in interface of Table, to indicate that it's readable. This adds #newScanBuilder(CaseInsensitiveStringMap) that is used to create a scan for batch, micro-batch, or continuous processing.
Annotations
@Evolving()
Since
3.0.0
trait SupportsRowLevelOperations extends Table
A mix-in interface for Table row-level operations support.
A mix-in interface for Table row-level operations support. Data sources can implement this interface to indicate they support rewriting data for DELETE, UPDATE, MERGE operations.
Annotations
@Experimental()
Since
3.3.0
trait SupportsSchemaEvolution extends Table
A mix-in interface for Table schema evolution support.
A mix-in interface for Table schema evolution support. Data sources can implement this interface to indicate the schema changes they support during write operations. Tables must report capability TableCapability#AUTOMATIC_SCHEMA_EVOLUTION for schema evolution to happen.
During automatic schema evolution, Spark computes the set of column changes (e.g. adding a new column, widening a column type) needed to make the target table schema compatible with the source data schema. Each candidate change is passed to #supportsColumnChange to determine whether the data source can apply it. Changes that are not supported are skipped and Spark will attempt to resolve such changes using casts. If casting is not supported, the query will fail.
Annotations
@Experimental()
Since
4.2.0
trait SupportsV1OverwriteWithSaveAsTable extends TableProvider
A marker interface that can be mixed into a TableProvider to indicate that the data source needs to distinguish between DataFrameWriter V1 saveAsTable operations and DataFrameWriter V2 createOrReplace/replace operations.
A marker interface that can be mixed into a TableProvider to indicate that the data source needs to distinguish between DataFrameWriter V1 saveAsTable operations and DataFrameWriter V2 createOrReplace/replace operations.
Background: DataFrameWriter V1's saveAsTable with SaveMode.Overwrite creates a ReplaceTableAsSelect logical plan, which is identical to the plan created by DataFrameWriter V2's createOrReplace. However, the documented semantics can have different interpretations:
- V1 saveAsTable with Overwrite: "if data/table already exists, existing data is expected to be overwritten by the contents of the DataFrame" - does not define behavior for metadata (schema) overwriting
- V2 createOrReplace: "The output table's schema, partition layout, properties, and other configuration will be based on the contents of the data frame... If the table exists, its configuration and data will be replaced"
Data sources that migrated from V1 to V2 may have adopted different behaviors based on these documented semantics. For example, Delta Lake interprets V1 saveAsTable to not replace table schema unless the overwriteSchema option is explicitly set.
When a TableProvider implements this interface and #addV1OverwriteWithSaveAsTableOption() returns true, DataFrameWriter V1 will add an internal write option to indicate that the command originated from saveAsTable API. The option key used is defined by #OPTION_NAME and the value will be set to "true". This allows the data source to distinguish between the two APIs and apply appropriate semantics.
Annotations
@Evolving()
Since
4.1.0
trait SupportsWrite extends Table
A mix-in interface of Table, to indicate that it's writable.
A mix-in interface of Table, to indicate that it's writable. This adds #newWriteBuilder(LogicalWriteInfo) that is used to create a write for batch or streaming.
Annotations
@Evolving()
Since
3.0.0
trait Table extends Relation
An interface representing a logical structured data set of a data source.
An interface representing a logical structured data set of a data source. For example, the implementation can be a directory on the file system, a topic of Kafka, or a table in the catalog, etc.
This interface can mixin SupportsRead and SupportsWrite to provide data reading and writing ability.
The default implementation of #partitioning() returns an empty array of partitions, and the default implementation of #properties() returns an empty map. These should be overridden by implementations that support partitioning and table properties.
A Table is one kind of Relation; the other is View.
Annotations
@Evolving()
Since
3.0.0
sealed final class TableCapability extends Enum[TableCapability]
Capabilities that can be provided by a Table implementation.
Capabilities that can be provided by a Table implementation.
Tables use Table#capabilities() to return a set of capabilities. Each capability signals to Spark that the table supports a feature identified by the capability. For example, returning #BATCH_READ allows Spark to read from the table using a batch scan.
Annotations
@Evolving()
Since
3.0.0
trait TableCatalog extends CatalogPlugin
Catalog API for connectors that expose tables.
Catalog API for connectors that expose tables.
Connectors that expose only tables implement this interface. Connectors that expose both tables and views must implement RelationCatalog (which extends both this interface and ViewCatalog and adds the cross-cutting contract for the combined case); the methods on this interface remain table-only -- they do not interact with views.
TableCatalog implementations may be case-sensitive or case-insensitive. Spark will pass table identifiers without modification. Field names passed to TableChange...) will be normalized to match the case used in the table schema when updating, renaming, or dropping existing columns when catalyst analysis is case-insensitive.
Annotations
@Evolving()
Since
3.0.0
sealed final class TableCatalogCapability extends Enum[TableCatalogCapability]
Capabilities that can be provided by a TableCatalog implementation.
Capabilities that can be provided by a TableCatalog implementation.
TableCatalogs use TableCatalog#capabilities() to return a set of capabilities. Each capability signals to Spark that the catalog supports a feature identified by the capability. For example, returning #SUPPORTS_CREATE_TABLE_WITH_GENERATED_COLUMNS allows Spark to accept GENERATED ALWAYS AS expressions in CREATE TABLE statements.
Annotations
@Evolving()
Since
3.4.0
trait TableChange extends AnyRef
TableChange subclasses represent requested changes to a table.
TableChange subclasses represent requested changes to a table. These are passed to TableCatalog#alterTable. For example,
```
  import TableChange._
  val catalog = Catalogs.load(name)
  catalog.asTableCatalog.alterTable(ident,
      addColumn("x", IntegerType),
      renameColumn("a", "b"),
      deleteColumn("c")
    )
```
Annotations
@Evolving()
Since
3.0.0
final class TableDependency extends Record with Dependency
A table dependency of a SQL object.
A table dependency of a SQL object.
The dependent table is identified by its structural multi-part name. nameParts arity matches the catalog's namespace depth plus one for the table name -- for a catalog with single-level namespaces the parts are [catalog, schema, table]; for a catalog with multi-level namespaces (e.g. Iceberg with db1.db2) the parts are [catalog, db1, db2, ..., table]; for v1 sources resolved through the session catalog, producers should normalize to [spark_catalog, db, table] so consumers see a stable arity per source kind. The structural form preserves arity and is unambiguous against quoted identifiers containing a literal .; consumers that need a flat string should join the parts themselves with a quoting scheme appropriate to their wire format.
Records' auto-generated equals/hashCode on array fields fall through to Object#equals (reference equality), so this record overrides them to use Object[]) / Arrays#hashCode(Object[]) on nameParts and give value-based semantics. The defensive-copy accessor override also clones on read so callers cannot mutate the record's internal array.
Annotations
@Evolving()
Since
4.2.0
class TableInfo extends AnyRef
Metadata describing a data-source table: its columns, properties, partitioning and constraints.
Metadata describing a data-source table: its columns, properties, partitioning and constraints. Spark realizes a TableInfo into a Table via DelegatingTable; a catalog that has its own Table object returns that instead. Views are described by the sibling View, which -- unlike a table -- is itself a Relation because Spark never builds a view object.
trait TableProvider extends AnyRef
The base interface for v2 data sources which don't have a real catalog.
The base interface for v2 data sources which don't have a real catalog. Implementations must have a public, 0-arg constructor.
Note that, TableProvider can only apply data operations to existing tables, like read, append, delete, and overwrite. It does not support the operations that require metadata changes, like create/drop tables.
The major responsibility of this interface is to return a Table for read/write.
Annotations
@Evolving()
Since
3.0.0
trait TableSummary extends AnyRef
Annotations
@Evolving()
sealed final class TableWritePrivilege extends Enum[TableWritePrivilege]
The table write privileges that will be provided when loading a table.
The table write privileges that will be provided when loading a table.
Since
3.5.3
trait TransactionalCatalogPlugin extends CatalogPlugin
A CatalogPlugin that supports transactions.
A CatalogPlugin that supports transactions.
Catalogs that implement this interface opt in to transactional query execution. A catalog implementing this interface is responsible for starting transactions.
Annotations
@Evolving()
Since
4.2.0
trait TruncatableTable extends Table
Represents a table which can be atomically truncated.
Represents a table which can be atomically truncated.
Annotations
@Evolving()
Since
3.2.0
class View extends Relation
A view in a catalog -- the typed payload returned by ViewCatalog#loadView and accepted by ViewCatalog#createView / ViewCatalog#replaceView.
A view in a catalog -- the typed payload returned by ViewCatalog#loadView and accepted by ViewCatalog#createView / ViewCatalog#replaceView. A View carries the view-specific fields that cannot be represented as string table properties: the query text, captured creation-time resolution context, captured SQL configs, schema-binding mode, and query output column names. Columns and user TBLPROPERTIES are set via the typed builder.
Unlike a Table, a View is itself a Relation rather than something Spark realizes into a Table: Spark expands the view's query text at read time and never builds a view object. A RelationCatalog returns a View directly from RelationCatalog#loadRelation for a view identifier, so it never has to smuggle a view through the Table surface.
Annotations
@Evolving()
Since
4.2.0
trait ViewCatalog extends CatalogPlugin
Catalog API for connectors that expose views.
Catalog API for connectors that expose views.
Connectors that expose only views implement this interface. Connectors that expose both tables and views must implement RelationCatalog (which extends both this interface and TableCatalog and adds the cross-cutting contract for the combined case); the methods on this interface remain view-only -- they do not interact with tables.
The presence of ViewCatalog on the catalog plugin is the signal that it supports views; there is no capability flag to declare.
Annotations
@Evolving()
Since
4.2.0

Packages

catalog

package catalog

Package Members

Type Members

Two principles

Per-method cross-type behavior

Single-RPC perf entry points

Ungrouped

Packages

catalog

package catalog

Package Members

Type Members

Two principles

Per-method cross-type behavior

Single-RPC perf entry points

Ungrouped

catalog