org.apache.spark.sql.types

Class UserDefinedType<UserType>

java.lang.Object

org.apache.spark.sql.types.DataType

org.apache.spark.sql.types.UserDefinedType<UserType>

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

VectorUDT

public abstract class UserDefinedType<UserType>
extends DataType
implements scala.Serializable

::DeveloperApi:: The data type for User Defined Types (UDTs).

This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a DataFrame which has class X in the schema.

For SparkSQL to recognize UDTs, the UDT must be annotated with SQLUserDefinedType.

The conversion via serialize occurs when instantiating a DataFrame from another RDD. The conversion via deserialize occurs when reading from a DataFrame.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
UserDefinedType()

Method Summary

Methods

Modifier and Type	Method and Description
int	defaultSize() The default size of a value of the UserDefinedType is 4096 bytes.
abstract UserType	deserialize(java.lang.Object datum) Convert a SQL datum to the user type
java.lang.String	pyUDT() Paired Python UDT class, if exists.
abstract java.lang.Object	serialize(java.lang.Object obj) Convert the user type to a SQL datum
java.lang.String	serializedPyClass() Serialized Python UDT class, if exists.
abstract DataType	sqlType() Underlying storage type for this UDT
abstract java.lang.Class<UserType>	userClass() Class object for the UserType

Methods inherited from class org.apache.spark.sql.types.DataType

buildFormattedString, fromCaseClassString, fromJson, json, prettyJson, simpleString, typeName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UserDefinedType

public UserDefinedType()

Method Detail

sqlType

public abstract DataType sqlType()

Underlying storage type for this UDT

pyUDT

public java.lang.String pyUDT()

Paired Python UDT class, if exists.

serializedPyClass

public java.lang.String serializedPyClass()

Serialized Python UDT class, if exists.

serialize

public abstract java.lang.Object serialize(java.lang.Object obj)

Convert the user type to a SQL datum

TODO: Can we make this take obj: UserType? The issue is in CatalystTypeConverters.convertToCatalyst, where we need to convert Any to UserType.

Parameters:

obj - (undocumented)

Returns:

(undocumented)

deserialize

public abstract UserType deserialize(java.lang.Object datum)

Convert a SQL datum to the user type

userClass

public abstract java.lang.Class<UserType> userClass()

Class object for the UserType

Returns:

(undocumented)

defaultSize

public int defaultSize()

The default size of a value of the UserDefinedType is 4096 bytes.

Specified by:

defaultSize in class DataType

Returns:

(undocumented)