Package org.apache.spark.sql.connector.catalog

Interface Column

@Evolving public interface Column
An interface representing a column of a Table. It defines basic properties of a column, such as name and data type, as well as some advanced ones like default column value.

Data Sources do not need to implement it. They should consume it in APIs like TableCatalog.createTable(Identifier, Column[], Transform[], Map), and report it in Table.columns() by calling the static create functions of this interface to create it.

A column cannot have both a default value and a generation expression.

  • Method Details

    • create

      static Column create(String name, DataType dataType)

    • create

      static Column create(String name, DataType dataType, boolean nullable)

    • create

      static Column create(String name, DataType dataType, boolean nullable, String comment, String metadataInJSON)

    • create

      static Column create(String name, DataType dataType, boolean nullable, String comment, ColumnDefaultValue defaultValue, String metadataInJSON)

    • create

      static Column create(String name, DataType dataType, boolean nullable, String comment, String generationExpression, String metadataInJSON)

    • create

      static Column create(String name, DataType dataType, boolean nullable, String comment, IdentityColumnSpec identityColumnSpec, String metadataInJSON)

    • builderFor

      static Column.Builder builderFor(String name, DataType dataType)
      Creates a builder for a new column with the given name and data type.
      Parameters:
      name - the name of the column
      dataType - the data type of the column
      Returns:
      a new builder
      Since:
      4.2.0

    • builderFrom

      static Column.Builder builderFrom(Column column)
      Creates a builder with pre-populated info from an existing column.
      Parameters:
      column - the source column
      Returns:
      a new builder seeded with the column's current state
      Since:
      4.2.0

    • name

      String name()
      Returns the name of this table column.

    • dataType

      DataType dataType()
      Returns the data type of this table column.

    • nullable

      boolean nullable()
      Returns true if this column may produce null values.

    • comment

      @Nullable String comment()
      Returns the comment of this table column. Null means no comment.

    • defaultValue

      @Nullable ColumnDefaultValue defaultValue()
      Returns the default value of this table column. Null means no default value.

    • generationExpression

      @Nullable String generationExpression()
      Returns the generation expression of this table column. Null means no generation expression.

      The generation expression is stored as spark SQL dialect. It is up to the data source to verify expression compatibility and reject writes as necessary.

    • identityColumnSpec

      @Nullable IdentityColumnSpec identityColumnSpec()
      Returns the identity column specification of this table column. Null means no identity column.

    • metadataInJSON

      @Nullable String metadataInJSON()
      Returns the column metadata in JSON format.

    • id

      @Nullable default String id()
      Returns the ID of this top-level column, or null. The ID is an opt-in identifier that the connector uses to track column identity beyond column name and type.

      When a non-null ID is returned, the connector commits to the following contract:

      • The ID is stable across renames (logical name changes preserve the ID).
      • The ID changes when a top-level column is dropped and re-added, even with the same name and type.
      • IDs are not reused within a table's history.

      When null is returned, Spark skips identity validation for that column. Connectors should return null when:

      • The catalog has no notion of column identity beyond name and type, OR
      • The connector chooses to treat same-name drop+re-add as the same column (lenient semantics).
      Returning null is per-column: a connector may return IDs for some columns and null for others.

      This API covers top-level columns only. Nested struct fields, array elements, and map keys/values carry their own IDs in struct field metadata. Spark validates both top-level and nested struct field IDs as part of schema compatibility checks (array elements and map/key values' validation is not supported yet). See StructField.id().