Interface SupportsReportStatistics

All Superinterfaces:
Scan

@Evolving public interface SupportsReportStatistics extends Scan
A mix in interface for Scan. Data sources can implement this interface to report statistics to Spark.

As of Spark 3.0, statistics are reported to the optimizer after operators are pushed to the data source. Implementations may return more accurate statistics based on pushed operators which may improve query performance by providing better information to the optimizer.

Since:
3.0.0
  • Method Details

    • estimateStatistics

      Statistics estimateStatistics()
      Returns the estimated statistics of this data source scan.
    • estimateSizeInBytes

      default OptionalLong estimateSizeInBytes()
      Returns the estimated size in bytes of this scan without computing full statistics.

      When cost-based optimization or plan statistics are disabled, Spark primarily needs the scan's size in bytes (for example, for broadcast-join thresholding). This method lets connectors that can produce a size estimate cheaply serve it directly and avoid computing the full statistics.

      The default implementation returns OptionalLong.empty(), signalling that the connector does not offer a cheap size estimate. In that case Spark falls back to estimateStatistics(), so a connector that only implements estimateStatistics() keeps the same size-estimation behavior it had before this method existed. Connectors override this method only when they have a genuinely cheaper size estimate than estimateStatistics().

      Since:
      4.3.0
    • reflectsFullyPushedDownFilters

      default boolean reflectsFullyPushedDownFilters()
      Returns whether the statistics reported by this scan already reflect all filters that were fully pushed down to the data source.

      When true (the default), the reported statistics describe exactly the data the scan will produce. When false, they do not account for the fully pushed filters (for example, they describe the whole table), so Spark may use those fully pushed filters to adjust stats. Re-applying those fully pushed filters in Spark should be redundant for query results because the data source already evaluates them.

      The adjustment Spark performs when this returns false is best-effort: Spark re-applies a fully pushed filter for stats adjustment only when every column the filter references is still present in Scan.readSchema(). If pruneColumns removes a pushed-filter column, Spark drops that filter from the adjustment, so the reported statistics will not reflect it.

      A connector that wants a fully pushed filter to participate in this stats adjustment must, when Spark requests schema pruning through SupportsPushDownRequiredColumns, retain the columns referenced only by that fully pushed filter; otherwise those columns are pruned and the filter is dropped from the adjustment.

      Since:
      4.3.0