Package org.apache.spark.sql.connector.read

Interface SupportsRuntimeFiltering

All Superinterfaces:

Scan, SupportsRuntimeV2Filtering

@Experimental
public interface SupportsRuntimeFiltering
extends SupportsRuntimeV2Filtering

A mix-in interface for Scan. Data sources can implement this interface if they can filter initially planned InputPartitions using predicates Spark infers at runtime.

Note that Spark will push runtime filters only if they are beneficial.

Since:

3.2.0

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.sql.connector.read.Scan

Scan.ColumnarSupportMode

Method Summary

Modifier and Type	Method	Description
default void	filter(Predicate[] predicates)	Filters this scan using runtime predicates.
void	filter(Filter[] filters)	Filters this scan using runtime filters.
NamedReference[]	filterAttributes()	Returns attributes this scan can be filtered by at runtime.

Methods inherited from interface org.apache.spark.sql.connector.read.Scan

columnarSupportMode, description, readSchema, reportDriverMetrics, supportedCustomMetrics, toBatch, toContinuousStream, toMicroBatchStream

Methods inherited from interface org.apache.spark.sql.connector.read.SupportsRuntimeV2Filtering

pushedPredicates, supportsIterativePushdown

Method Details

filterAttributes

NamedReference[] filterAttributes()

Returns attributes this scan can be filtered by at runtime.

Spark will call filter(Filter[]) if it can derive a runtime predicate for any of the filter attributes.

Specified by:

filterAttributes in interface SupportsRuntimeV2Filtering

filter

void filter(Filter[] filters)

Filters this scan using runtime filters.

The provided expressions must be interpreted as a set of filters that are ANDed together. Implementations may use the filters to prune initially planned InputPartitions.

If the scan also implements SupportsReportPartitioning, it must preserve the originally reported partitioning during runtime filtering. While applying runtime filters, the scan may detect that some InputPartitions have no matching data, in which case it can either replace the initially planned InputPartitions that have no matching data with empty InputPartitions, or report only a subset of the original partition values (omitting those with no data) via Batch.planInputPartitions(). The scan must not report new partition values that were not present in the original partitioning.

Note that Spark will call Scan.toBatch() again after filtering the scan at runtime.

Parameters:

filters - data source filters used to filter the scan at runtime

filter

default void filter(Predicate[] predicates)

Description copied from interface: SupportsRuntimeV2Filtering

Filters this scan using runtime predicates.

The provided expressions must be interpreted as a set of predicates that are ANDed together. Implementations may use the predicates to prune initially planned InputPartitions.

If the scan also implements SupportsReportPartitioning, it must preserve the originally reported partitioning during runtime filtering. While applying runtime predicates, the scan may detect that some InputPartitions have no matching data, in which case it can either replace the initially planned InputPartitions that have no matching data with empty InputPartitions, or report only a subset of the original partition values (omitting those with no data) via Batch.planInputPartitions(). The scan must not report new partition values that were not present in the original partitioning.

This method may be called multiple times with additional predicates (e.g. PartitionPredicate) when SupportsRuntimeV2Filtering.supportsIterativePushdown() returns true. The implementation must accumulate state across all calls so that SupportsRuntimeV2Filtering.pushedPredicates() can return predicates from all of them.

Note that Spark will call Scan.toBatch() again after filtering the scan at runtime.

Specified by:

filter in interface SupportsRuntimeV2Filtering

Parameters:

predicates - data source V2 predicates used to filter the scan at runtime