org.apache.spark.sql.sources.v2.reader

Interface SupportsPushDownFilters

All Superinterfaces:

DataSourceReader

@InterfaceStability.Evolving
public interface SupportsPushDownFilters
extends DataSourceReader

A mix-in interface for DataSourceReader. Data source readers can implement this interface to push down filters to the data source and reduce the size of the data to be read.

Method Summary

Modifier and Type	Method and Description
Filter[]	pushedFilters() Returns the filters that are pushed to the data source via pushFilters(Filter[]).
Filter[]	pushFilters(Filter[] filters) Pushes down filters, and returns filters that need to be evaluated after scanning.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceReader

planInputPartitions, readSchema

Method Detail

pushFilters

Filter[] pushFilters(Filter[] filters)

Pushes down filters, and returns filters that need to be evaluated after scanning.

pushedFilters

Filter[] pushedFilters()

Returns the filters that are pushed to the data source via pushFilters(Filter[]). There are 3 kinds of filters: 1. pushable filters which don't need to be evaluated again after scanning. 2. pushable filters which still need to be evaluated after scanning, e.g. parquet row group filter. 3. non-pushable filters. Both case 1 and 2 should be considered as pushed filters and should be returned by this method. It's possible that there is no filters in the query and pushFilters(Filter[]) is never called, empty array should be returned for this case.