org.apache.spark.sql.sources.v2.reader

Interface SupportsPushDownRequiredColumns

All Superinterfaces:

DataSourceReader

@InterfaceStability.Evolving
public interface SupportsPushDownRequiredColumns
extends DataSourceReader

A mix-in interface for DataSourceReader. Data source readers can implement this interface to push down required columns to the data source and only read these columns during scan to reduce the size of the data to be read.

Method Summary

Modifier and Type	Method and Description
void	pruneColumns(StructType requiredSchema) Applies column pruning w.r.t.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceReader

planInputPartitions, readSchema

Method Detail

pruneColumns

void pruneColumns(StructType requiredSchema)

Applies column pruning w.r.t. the given requiredSchema. Implementation should try its best to prune the unnecessary columns or nested fields, but it's also OK to do the pruning partially, e.g., a data source may not be able to prune nested fields, and only prune top-level columns. Note that, data source readers should update DataSourceReader.readSchema() after applying column pruning.