org.apache.spark.sql.sources.v2.reader

Interface SupportsScanUnsafeRow

All Superinterfaces:

DataSourceReader

@InterfaceStability.Unstable
public interface SupportsScanUnsafeRow
extends DataSourceReader

A mix-in interface for DataSourceReader. Data source readers can implement this interface to output UnsafeRow directly and avoid the row copy at Spark side. This is an experimental and unstable interface, as UnsafeRow is not public and may get changed in the future Spark versions.

Method Summary

Modifier and Type	Method and Description
default java.util.List<DataReaderFactory<Row>>	createDataReaderFactories() Returns a list of reader factories.
java.util.List<DataReaderFactory<org.apache.spark.sql.catalyst.expressions.UnsafeRow>>	createUnsafeRowReaderFactories() Similar to DataSourceReader.createDataReaderFactories(), but returns data in unsafe row format.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceReader

readSchema

Method Detail

createDataReaderFactories

default java.util.List<DataReaderFactory<Row>> createDataReaderFactories()

Description copied from interface: DataSourceReader

Returns a list of reader factories. Each factory is responsible for creating a data reader to output data for one RDD partition. That means the number of factories returned here is same as the number of RDD partitions this scan outputs. Note that, this may not be a full scan if the data source reader mixes in other optimization interfaces like column pruning, filter push-down, etc. These optimizations are applied before Spark issues the scan request. If this method fails (by throwing an exception), the action would fail and no Spark job was submitted.

Specified by:

createDataReaderFactories in interface DataSourceReader

createUnsafeRowReaderFactories

java.util.List<DataReaderFactory<org.apache.spark.sql.catalyst.expressions.UnsafeRow>> createUnsafeRowReaderFactories()

Similar to DataSourceReader.createDataReaderFactories(), but returns data in unsafe row format.