Enum Class TableCapability

Object
Enum<TableCapability>
org.apache.spark.sql.connector.catalog.TableCapability
All Implemented Interfaces:
Serializable, Comparable<TableCapability>, Constable

@Evolving public enum TableCapability extends Enum<TableCapability>
Capabilities that can be provided by a Table implementation.

Tables use Table.capabilities() to return a set of capabilities. Each capability signals to Spark that the table supports a feature identified by the capability. For example, returning BATCH_READ allows Spark to read from the table using a batch scan.

Since:
3.0.0
  • Nested Class Summary

    Nested classes/interfaces inherited from class java.lang.Enum

    Enum.EnumDesc<E extends Enum<E>>
  • Enum Constant Summary

    Enum Constants
    Enum Constant
    Description
    Signals that the table accepts input of any schema in a write operation.
    Signals that table supports Spark altering the schema if necessary as part of an operation.
    Signals that the table supports reads in batch execution mode.
    Signals that the table supports append writes in batch execution mode.
    Signals that the table supports reads in continuous streaming execution mode.
    Signals that the table wants Spark to auto-fill generated column values and enforce generated column constraints during writes.
    Signals that the table supports reads in micro-batch streaming execution mode.
    Signals that the table can replace existing data that matches a filter with appended data in a write operation.
    Signals that the table can dynamically replace existing data partitions with appended data in a write operation.
    Signals that Spark may fuse two batch scans of this table that differ only in their projected columns and/or pushed filters into a single scan (Spark-side scan merging).
    Signals that the table supports append writes in streaming execution mode.
    Signals that the table can be truncated in a write operation.
    Signals that the table supports append writes using the V1 InsertableRelation interface.
  • Method Summary

    Modifier and Type
    Method
    Description
    Returns the enum constant of this class with the specified name.
    Returns an array containing the constants of this enum class, in the order they are declared.

    Methods inherited from class java.lang.Enum

    compareTo, describeConstable, equals, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

    Methods inherited from class java.lang.Object

    getClass, notify, notifyAll, wait, wait, wait
  • Enum Constant Details

    • BATCH_READ

      public static final TableCapability BATCH_READ
      Signals that the table supports reads in batch execution mode.
    • MICRO_BATCH_READ

      public static final TableCapability MICRO_BATCH_READ
      Signals that the table supports reads in micro-batch streaming execution mode.
    • CONTINUOUS_READ

      public static final TableCapability CONTINUOUS_READ
      Signals that the table supports reads in continuous streaming execution mode.
    • BATCH_WRITE

      public static final TableCapability BATCH_WRITE
      Signals that the table supports append writes in batch execution mode.

      Tables that return this capability must support appending data and may also support additional write modes, like TRUNCATE, OVERWRITE_BY_FILTER, and OVERWRITE_DYNAMIC.

    • STREAMING_WRITE

      public static final TableCapability STREAMING_WRITE
      Signals that the table supports append writes in streaming execution mode.

      Tables that return this capability must support appending data and may also support additional write modes, like TRUNCATE, OVERWRITE_BY_FILTER, and OVERWRITE_DYNAMIC.

    • TRUNCATE

      public static final TableCapability TRUNCATE
      Signals that the table can be truncated in a write operation.

      Truncating a table removes all existing rows.

      See SupportsTruncate.

    • OVERWRITE_BY_FILTER

      public static final TableCapability OVERWRITE_BY_FILTER
      Signals that the table can replace existing data that matches a filter with appended data in a write operation.

      See SupportsOverwriteV2.

    • OVERWRITE_DYNAMIC

      public static final TableCapability OVERWRITE_DYNAMIC
      Signals that the table can dynamically replace existing data partitions with appended data in a write operation.

      See SupportsDynamicOverwrite.

    • ACCEPT_ANY_SCHEMA

      public static final TableCapability ACCEPT_ANY_SCHEMA
      Signals that the table accepts input of any schema in a write operation.
    • AUTOMATIC_SCHEMA_EVOLUTION

      public static final TableCapability AUTOMATIC_SCHEMA_EVOLUTION
      Signals that table supports Spark altering the schema if necessary as part of an operation.
    • V1_BATCH_WRITE

      public static final TableCapability V1_BATCH_WRITE
      Signals that the table supports append writes using the V1 InsertableRelation interface.

      Tables that return this capability must create a V1Write and may also support additional write modes, like TRUNCATE, and OVERWRITE_BY_FILTER, but cannot support OVERWRITE_DYNAMIC.

    • GENERATE_COLUMN_VALUES_ON_WRITE

      public static final TableCapability GENERATE_COLUMN_VALUES_ON_WRITE
      Signals that the table wants Spark to auto-fill generated column values and enforce generated column constraints during writes.

      When this capability is present, Spark will:

      • Auto-compute missing generated column values using the generation expression for by-name writes. Ordinary by-position writes still require the input to provide a value for every table column.
      • Validate explicitly-provided generated column values against the generation expression.

      Without this capability, the connector is responsible for handling generated column values during writes.

      Since:
      4.3.0
    • SCAN_MERGING

      public static final TableCapability SCAN_MERGING
      Signals that Spark may fuse two batch scans of this table that differ only in their projected columns and/or pushed filters into a single scan (Spark-side scan merging).

      By returning this capability a table declares a determinism contract: holding the scan options constant, the rows and columns a scan reads are fully determined by the filters pushed via SupportsPushDownV2Filters and the columns pruned via SupportsPushDownRequiredColumns. Equivalently, obtaining a fresh ScanBuilder with the same options and re-applying the same pushed filters and pruned columns yields an equivalent scan.

      Given that contract, Spark builds the merged scan itself: it prunes a fresh ScanBuilder to the union of both read schemas, re-pushes the (possibly OR-widened) filters, and builds. The merged scan reads the union of the two scans' columns and a superset of their rows; each original scan's result is recovered by a projection and filter applied above it. The connector supplies no merge logic of its own.

      This capability lives on the table rather than on the scan so that a source using the V1 scan fallback (whose scan Spark wraps in an internal wrapper) can still opt in. A table need not reason about pushdowns that are not reproducible this way (a pushed aggregate, join, variant extraction, limit, offset, top-N, or table sample): Spark tracks those on its own side while building the scan and never merges a scan that carries one, whether or not the table returns this capability.

      Since:
      4.3.0
  • Method Details

    • values

      public static TableCapability[] values()
      Returns an array containing the constants of this enum class, in the order they are declared.
      Returns:
      an array containing the constants of this enum class, in the order they are declared
    • valueOf

      public static TableCapability valueOf(String name)
      Returns the enum constant of this class with the specified name. The string must match exactly an identifier used to declare an enum constant in this class. (Extraneous whitespace characters are not permitted.)
      Parameters:
      name - the name of the enum constant to be returned.
      Returns:
      the enum constant with the specified name
      Throws:
      IllegalArgumentException - if this enum class has no constant with the specified name
      NullPointerException - if the argument is null