Package org.apache.spark.sql.connector.catalog

Interface MetadataColumn

@Evolving public interface MetadataColumn
Interface for a metadata column.

A metadata column can expose additional metadata about a row. For example, rows from Kafka can use metadata columns to expose a message's topic, partition number, and offset.

A metadata column could also be the result of a transform applied to a value in the row. For example, a partition value produced by bucket(id, 16) could be exposed by a metadata column. In this case, transform() should return a non-null Transform that produced the metadata column's values.

Since:
3.1.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default String
    comment()
    Documentation for this metadata column, or null.
    DataType
    dataType()
    The data type of values in this metadata column.
    default boolean
    isNullable()
     
    String
    name()
    The name of this metadata column.
    default Transform
    transform()
    The Transform used to produce this metadata column from data rows, or null.

  • Method Details

    • name

      String name()
      The name of this metadata column.
      Returns:
      a String name

    • dataType

      DataType dataType()
      The data type of values in this metadata column.
      Returns:
      a DataType

    • isNullable

      default boolean isNullable()
      Returns:
      whether values produced by this metadata column may be null

    • comment

      default String comment()
      Documentation for this metadata column, or null.
      Returns:
      a documentation String

    • transform

      default Transform transform()
      The Transform used to produce this metadata column from data rows, or null.
      Returns:
      a Transform used to produce the column's values, or null if there isn't one