org.apache.spark.sql.types

Class DataType

Object

org.apache.spark.sql.types.DataType

Direct Known Subclasses:

ArrayType, MapType, NativeType, NullType, StructType, UserDefinedType

public abstract class DataType
extends Object

:: DeveloperApi :: The base type of all Spark SQL data types.

Constructor Summary

Constructors

Constructor and Description
DataType()

Method Summary

Methods

Modifier and Type	Method and Description
abstract DataType	asNullable() Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).
abstract int	defaultSize() The default size of a value of this data type.
static boolean	equalsIgnoreCompatibleNullability(DataType from, DataType to) Compares two types, ignoring compatible nullability of ArrayType, MapType, StructType.
static boolean	equalsIgnoreNullability(DataType left, DataType right) Compares two types, ignoring nullability of ArrayType, MapType, StructType.
static DataType	fromCaseClassString(String string)
static DataType	fromJson(String json)
boolean	isPrimitive()
String	json()
org.json4s.JsonAST.JValue	jsonValue()
String	prettyJson()
boolean	sameType(DataType other) Check if this and other are the same data type when ignoring nullability (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).
String	simpleString()
String	typeName()
boolean	unapply(org.apache.spark.sql.catalyst.expressions.Expression a) Matches any expression that evaluates to this DataType

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataType

public DataType()

Method Detail

fromJson

public static DataType fromJson(String json)

fromCaseClassString

public static DataType fromCaseClassString(String string)

equalsIgnoreNullability

public static boolean equalsIgnoreNullability(DataType left,
                              DataType right)

Compares two types, ignoring nullability of ArrayType, MapType, StructType.

equalsIgnoreCompatibleNullability

public static boolean equalsIgnoreCompatibleNullability(DataType from,
                                        DataType to)

Compares two types, ignoring compatible nullability of ArrayType, MapType, StructType.

Compatible nullability is defined as follows: - If from and to are ArrayTypes, from has a compatible nullability with to if and only if to.containsNull is true, or both of from.containsNull and to.containsNull are false. - If from and to are MapTypes, from has a compatible nullability with to if and only if to.valueContainsNull is true, or both of from.valueContainsNull and to.valueContainsNull are false. - If from and to are StructTypes, from has a compatible nullability with to if and only if for all every pair of fields, to.nullable is true, or both of fromField.nullable and toField.nullable are false.

unapply

public boolean unapply(org.apache.spark.sql.catalyst.expressions.Expression a)

Matches any expression that evaluates to this DataType

defaultSize

public abstract int defaultSize()

The default size of a value of this data type.

isPrimitive

public boolean isPrimitive()

typeName

public String typeName()

jsonValue

public org.json4s.JsonAST.JValue jsonValue()

json

public String json()

prettyJson

public String prettyJson()

simpleString

public String simpleString()

sameType

public boolean sameType(DataType other)

Check if this and other are the same data type when ignoring nullability (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

asNullable

public abstract DataType asNullable()

Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).