org.apache.spark.sql.sources.v2.reader

Interface SupportsScanColumnarBatch

All Superinterfaces:

DataSourceReader

@InterfaceStability.Evolving
public interface SupportsScanColumnarBatch
extends DataSourceReader

A mix-in interface for DataSourceReader. Data source readers can implement this interface to output ColumnarBatch and make the scan faster.

Method Summary

Modifier and Type	Method and Description
java.util.List<DataReaderFactory<ColumnarBatch>>	createBatchDataReaderFactories() Similar to DataSourceReader.createDataReaderFactories(), but returns columnar data in batches.
default java.util.List<DataReaderFactory<Row>>	createDataReaderFactories() Returns a list of reader factories.
default boolean	enableBatchRead() Returns true if the concrete data source reader can read data in batch according to the scan properties like required columns, pushes filters, etc.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceReader

readSchema

Method Detail

createDataReaderFactories

default java.util.List<DataReaderFactory<Row>> createDataReaderFactories()

Description copied from interface: DataSourceReader

Returns a list of reader factories. Each factory is responsible for creating a data reader to output data for one RDD partition. That means the number of factories returned here is same as the number of RDD partitions this scan outputs. Note that, this may not be a full scan if the data source reader mixes in other optimization interfaces like column pruning, filter push-down, etc. These optimizations are applied before Spark issues the scan request. If this method fails (by throwing an exception), the action would fail and no Spark job was submitted.

Specified by:

createDataReaderFactories in interface DataSourceReader

createBatchDataReaderFactories

java.util.List<DataReaderFactory<ColumnarBatch>> createBatchDataReaderFactories()

Similar to DataSourceReader.createDataReaderFactories(), but returns columnar data in batches.

enableBatchRead

default boolean enableBatchRead()

Returns true if the concrete data source reader can read data in batch according to the scan properties like required columns, pushes filters, etc. It's possible that the implementation can only support some certain columns with certain types. Users can overwrite this method and createDataReaderFactories() to fallback to normal read path under some conditions.