Interface SupportsReportStatistics
- All Superinterfaces:
Scan
Scan. Data sources can implement this interface to
report statistics to Spark.
As of Spark 3.0, statistics are reported to the optimizer after operators are pushed to the data source. Implementations may return more accurate statistics based on pushed operators which may improve query performance by providing better information to the optimizer.
- Since:
- 3.0.0
-
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.spark.sql.connector.read.Scan
Scan.ColumnarSupportMode -
Method Summary
Modifier and TypeMethodDescriptiondefault OptionalLongReturns the estimated size in bytes of this scan without computing full statistics.Returns the estimated statistics of this data source scan.default booleanReturns whether the statistics reported by this scan already reflect all filters that were fully pushed down to the data source.Methods inherited from interface org.apache.spark.sql.connector.read.Scan
columnarSupportMode, description, readSchema, reportDriverMetrics, supportedCustomMetrics, toBatch, toContinuousStream, toMicroBatchStream
-
Method Details
-
estimateStatistics
Statistics estimateStatistics()Returns the estimated statistics of this data source scan. -
estimateSizeInBytes
Returns the estimated size in bytes of this scan without computing full statistics.When cost-based optimization or plan statistics are disabled, Spark primarily needs the scan's size in bytes (for example, for broadcast-join thresholding). This method lets connectors that can produce a size estimate cheaply serve it directly and avoid computing the full statistics.
The default implementation returns
OptionalLong.empty(), signalling that the connector does not offer a cheap size estimate. In that case Spark falls back toestimateStatistics(), so a connector that only implementsestimateStatistics()keeps the same size-estimation behavior it had before this method existed. Connectors override this method only when they have a genuinely cheaper size estimate thanestimateStatistics().- Since:
- 4.3.0
-
reflectsFullyPushedDownFilters
default boolean reflectsFullyPushedDownFilters()Returns whether the statistics reported by this scan already reflect all filters that were fully pushed down to the data source.When
true(the default), the reported statistics describe exactly the data the scan will produce. Whenfalse, they do not account for the fully pushed filters (for example, they describe the whole table), so Spark may use those fully pushed filters to adjust stats. Re-applying those fully pushed filters in Spark should be redundant for query results because the data source already evaluates them.The adjustment Spark performs when this returns
falseis best-effort: Spark re-applies a fully pushed filter for stats adjustment only when every column the filter references is still present inScan.readSchema(). IfpruneColumnsremoves a pushed-filter column, Spark drops that filter from the adjustment, so the reported statistics will not reflect it.A connector that wants a fully pushed filter to participate in this stats adjustment must, when Spark requests schema pruning through
SupportsPushDownRequiredColumns, retain the columns referenced only by that fully pushed filter; otherwise those columns are pruned and the filter is dropped from the adjustment.- Since:
- 4.3.0
-