Package org.apache.spark.sql.types.ops

Interface TypeApiOps

All Superinterfaces:
Serializable
All Known Implementing Classes:
TimeTypeApiOps
public interface TypeApiOps extends Serializable
Client-side (spark-api) type operations for the Types Framework.

This trait consolidates all client-side operations that a data type must implement to be usable in the Spark SQL API layer. Mandatory methods (format, toSQLValue, getEncoder) must be implemented by every type. Optional methods (Arrow, Python, Hive, Thrift) return Option and default to None - types implement them as they expand their integration coverage.

RELATIONSHIP TO TypeOps: - TypeOps (catalyst): Server-side operations - physical types, literals, conversions - TypeApiOps (spark-api): Client-side operations - formatting, encoding

The split exists because sql/api cannot depend on sql/catalyst. For TimeType, TimeTypeOps (catalyst) extends TimeTypeApiOps (sql-api) to inherit both sets of operations.

Since:
4.2.0
See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    DataType
    dataType()
    The DataType this Ops instance handles.
    String
    format(Object v)
    Formats an internal value as a display string.
    scala.Option<String>
    formatExternal(Object value)
    Formats an external-type value for Hive output.
    scala.Option<String>
    formatExternal(Object value, boolean nested)
    Formats with nesting context.
    org.apache.spark.unsafe.types.UTF8String
    formatUTF8(Object v)
    Formats an internal value as a UTF8String.
    org.apache.spark.sql.catalyst.encoders.AgnosticEncoder<?>
    getEncoder()
    Returns the AgnosticEncoder for this type.
    scala.Option<scala.Function1<Object,Object>>
    makeFromJava()
    Creates a converter function for Python/Py4J interop.
    scala.Option<Object>
    needConversionInPython()
    Returns true if values of this type need conversion when passed to/from Python.
    Object
    nullSafeConvert(Object input, scala.PartialFunction<Object,Object> f)
    Null-safe conversion helper.
    scala.Option<String>
    thriftTypeName()
    Returns the Thrift TTypeId name for this type (e.g., "STRING_TYPE").
    scala.Option<org.apache.arrow.vector.types.pojo.ArrowType>
    toArrowType(String timeZoneId)
    Converts this DataType to its Arrow representation.
    String
    toSQLValue(Object v)
    Formats an internal value as a SQL literal string.

  • Method Details

    • dataType

      DataType dataType()
      The DataType this Ops instance handles.

    • format

      String format(Object v)
      Formats an internal value as a display string.

      Used by CAST to STRING, EXPLAIN output, SHOW commands.

      Parameters:
      v - the internal value (e.g., Long nanoseconds for TimeType)
      Returns:
      formatted string (e.g., "10:30:45.123456")

    • formatUTF8

      org.apache.spark.unsafe.types.UTF8String formatUTF8(Object v)
      Formats an internal value as a UTF8String.

      Default implementation wraps format(). Override for performance if needed.

      Parameters:
      v - (undocumented)
      Returns:
      (undocumented)

    • toSQLValue

      String toSQLValue(Object v)
      Formats an internal value as a SQL literal string.

      Parameters:
      v - the internal value
      Returns:
      SQL literal string (e.g., "TIME '10:30:00'")

    • getEncoder

      org.apache.spark.sql.catalyst.encoders.AgnosticEncoder<?> getEncoder()
      Returns the AgnosticEncoder for this type.

      Used by RowEncoder for Dataset[T] operations.

      Returns:
      AgnosticEncoder instance (e.g., LocalTimeEncoder for TimeType)

    • nullSafeConvert

      Object nullSafeConvert(Object input, scala.PartialFunction<Object,Object> f)
      Null-safe conversion helper. Returns null for null input, applies the partial function for non-null input, and returns null for unmatched values.
      Parameters:
      input - (undocumented)
      f - (undocumented)
      Returns:
      (undocumented)

    • toArrowType

      scala.Option<org.apache.arrow.vector.types.pojo.ArrowType> toArrowType(String timeZoneId)
      Converts this DataType to its Arrow representation. Returns None if not supported.

    • needConversionInPython

      scala.Option<Object> needConversionInPython()
      Returns true if values of this type need conversion when passed to/from Python.

    • makeFromJava

      scala.Option<scala.Function1<Object,Object>> makeFromJava()
      Creates a converter function for Python/Py4J interop.

    • formatExternal

      scala.Option<String> formatExternal(Object value)
      Formats an external-type value for Hive output. Most types override this simple version. Types that need different formatting when nested should override the 2-param overload.
      Parameters:
      value - (undocumented)
      Returns:
      (undocumented)

    • formatExternal

      scala.Option<String> formatExternal(Object value, boolean nested)
      Formats with nesting context. Default delegates to the simple version.

    • thriftTypeName

      scala.Option<String> thriftTypeName()
      Returns the Thrift TTypeId name for this type (e.g., "STRING_TYPE").