Packages

t

org.apache.spark.sql.connector.read

SupportsReportStatistics

trait SupportsReportStatistics extends Scan

A mix in interface for Scan. Data sources can implement this interface to report statistics to Spark.

As of Spark 3.0, statistics are reported to the optimizer after operators are pushed to the data source. Implementations may return more accurate statistics based on pushed operators which may improve query performance by providing better information to the optimizer.

Annotations
@Evolving()
Source
SupportsReportStatistics.java
Since

3.0.0

Linear Supertypes
Scan, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SupportsReportStatistics
  2. Scan
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def estimateStatistics(): Statistics

    Returns the estimated statistics of this data source scan.

  2. abstract def readSchema(): StructType

    Returns the actual schema of this data source scan, which may be different from the physical schema of the underlying storage, as column pruning or other optimizations may happen.

    Returns the actual schema of this data source scan, which may be different from the physical schema of the underlying storage, as column pruning or other optimizations may happen.

    Definition Classes
    Scan

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def columnarSupportMode(): ColumnarSupportMode

    Subclasses can implement this method to indicate if the support for columnar data should be determined by each partition or is set as a default for the whole scan.

    Subclasses can implement this method to indicate if the support for columnar data should be determined by each partition or is set as a default for the whole scan.

    Definition Classes
    Scan
    Since

    3.5.0

  7. def description(): String

    A description string of this scan, which may includes information like: what filters are configured for this scan, what's the value of some important options like path, etc.

    A description string of this scan, which may includes information like: what filters are configured for this scan, what's the value of some important options like path, etc. The description doesn't need to include #readSchema(), as Spark already knows it.

    By default this returns the class name of the implementation. Please override it to provide a meaningful description.

    Definition Classes
    Scan
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def estimateSizeInBytes(): OptionalLong

    Returns the estimated size in bytes of this scan without computing full statistics.

    Returns the estimated size in bytes of this scan without computing full statistics.

    When cost-based optimization or plan statistics are disabled, Spark primarily needs the scan's size in bytes (for example, for broadcast-join thresholding). This method lets connectors that can produce a size estimate cheaply serve it directly and avoid computing the full statistics.

    The default implementation returns OptionalLong.empty(), signalling that the connector does not offer a cheap size estimate. In that case Spark falls back to #estimateStatistics(), so a connector that only implements #estimateStatistics() keeps the same size-estimation behavior it had before this method existed. Connectors override this method only when they have a genuinely cheaper size estimate than #estimateStatistics().

    Since

    4.3.0

  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  17. def reflectsFullyPushedDownFilters(): Boolean

    Returns whether the statistics reported by this scan already reflect all filters that were fully pushed down to the data source.

    Returns whether the statistics reported by this scan already reflect all filters that were fully pushed down to the data source.

    When true (the default), the reported statistics describe exactly the data the scan will produce. When false, they do not account for the fully pushed filters (for example, they describe the whole table), so Spark may use those fully pushed filters to adjust stats. Re-applying those fully pushed filters in Spark should be redundant for query results because the data source already evaluates them.

    The adjustment Spark performs when this returns false is best-effort: Spark re-applies a fully pushed filter for stats adjustment only when every column the filter references is still present in Scan#readSchema(). If pruneColumns removes a pushed-filter column, Spark drops that filter from the adjustment, so the reported statistics will not reflect it.

    A connector that wants a fully pushed filter to participate in this stats adjustment must, when Spark requests schema pruning through SupportsPushDownRequiredColumns, retain the columns referenced only by that fully pushed filter; otherwise those columns are pruned and the filter is dropped from the adjustment.

    Since

    4.3.0

  18. def reportDriverMetrics(): Array[CustomTaskMetric]

    Returns an array of custom metrics which are collected with values at the driver side only.

    Returns an array of custom metrics which are collected with values at the driver side only. Note that these metrics must be included in the supported custom metrics reported by supportedCustomMetrics.

    Definition Classes
    Scan
    Since

    3.4.0

  19. def supportedCustomMetrics(): Array[CustomMetric]

    Returns an array of supported custom metrics with name and description.

    Returns an array of supported custom metrics with name and description. By default it returns empty array.

    Definition Classes
    Scan
  20. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  21. def toBatch(): Batch

    Returns the physical representation of this scan for batch query.

    Returns the physical representation of this scan for batch query. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#BATCH_READ support in its Table#capabilities().

    If the scan supports runtime filtering and implements SupportsRuntimeFiltering, this method may be called multiple times. Therefore, implementations can cache some state to avoid planning the job twice.

    Definition Classes
    Scan
    Exceptions thrown
  22. def toContinuousStream(checkpointLocation: String): ContinuousStream

    Returns the physical representation of this scan for streaming query with continuous mode.

    Returns the physical representation of this scan for streaming query with continuous mode. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#CONTINUOUS_READ support in its Table#capabilities().

    checkpointLocation

    a path to Hadoop FS scratch space that can be used for failure recovery. Data streams for the same logical source in the same query will be given the same checkpointLocation.

    Definition Classes
    Scan
    Exceptions thrown
  23. def toMicroBatchStream(checkpointLocation: String): MicroBatchStream

    Returns the physical representation of this scan for streaming query with micro-batch mode.

    Returns the physical representation of this scan for streaming query with micro-batch mode. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#MICRO_BATCH_READ support in its Table#capabilities().

    checkpointLocation

    a path to Hadoop FS scratch space that can be used for failure recovery. Data streams for the same logical source in the same query will be given the same checkpointLocation.

    Definition Classes
    Scan
    Exceptions thrown
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Scan

Inherited from AnyRef

Inherited from Any

Ungrouped