Packages

trait SupportsRealTimeRead[T] extends PartitionReader[T]

A variation on PartitionReader for use with low latency streaming processing.

Which method to implement. A source reader must provide a nextWithTimeout that proceeds to the next record, blocking until one is available or a timeout elapses. There are two overloads, and a source should override exactly one:

  • #nextWithTimeout(Long) -- implement this in the common case. It receives only the timeout and is all a third-party source needs.
  • Long) -- the overload the execution engine actually invokes. Its default implementation ignores the extra parameter and delegates to #nextWithTimeout(Long). Override it only for engine-internal sources that must observe the engine's reference start time (used together with the engine-internal low latency clock); that clock is not part of the third-party contract, so most sources should not override this overload.

Because both overloads are default methods, overriding neither is not caught at compile time -- it fails at runtime when #nextWithTimeout(Long) throws.

Annotations
@Evolving()
Source
SupportsRealTimeRead.java
Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SupportsRealTimeRead
  2. PartitionReader
  3. Closeable
  4. AutoCloseable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def close(): Unit
    Definition Classes
    Closeable → AutoCloseable
    Annotations
    @throws(classOf[java.io.IOException])
  2. abstract def get(): T

    Return the current record.

    Return the current record. This method should return same value until next is called.

    Definition Classes
    PartitionReader
  3. abstract def getOffset(): PartitionOffset

    Get the offset of the next record, or the start offset if no records have been read.

    Get the offset of the next record, or the start offset if no records have been read.

    The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.

  4. abstract def next(): Boolean

    Proceed to next record, returns false if there is no more records.

    Proceed to next record, returns false if there is no more records.

    Definition Classes
    PartitionReader
    Exceptions thrown

    IOException if failure happens during disk/network IO like reading files.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def currentMetricsValues(): Array[CustomTaskMetric]

    Returns an array of custom task metrics.

    Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.

    Definition Classes
    PartitionReader
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def nextWithTimeout(startTimeMs: Long, timeoutMs: Long): RecordStatus

    The overload of #nextWithTimeout(Long) that the execution engine actually invokes.

    The overload of #nextWithTimeout(Long) that the execution engine actually invokes. In addition to the timeout it receives startTimeMs, the reference time the engine used to compute that timeout, so the engine and the source agree on when the wait started (this matters when the engine runs against its internal low latency clock, e.g. a manual clock in tests). The default implementation ignores startTimeMs and delegates to #nextWithTimeout(Long).

    This is intended for engine-internal sources. The reference clock is not part of the third-party contract, so third-party sources should implement #nextWithTimeout(Long) and leave this overload to its default.

    startTimeMs

    the base time (milliseconds) that was used to calculate the timeout

    timeoutMs

    if no result is available after this timeout (milliseconds), return

    returns

    RecordStatus describing whether a record is available and its arrival time

    Exceptions thrown
  14. def nextWithTimeout(timeoutMs: Long): RecordStatus

    Alternative function to be called than next(), that proceed to the next record.

    Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.

    This is the recommended method to implement for a source. It is enough for any source that does not need the engine's reference start time (see Long)). The engine always invokes Long), whose default implementation delegates here, so overriding only this method is sufficient. If a source overrides neither overload, this default implementation throws to surface the mistake.

    timeoutMs

    if no result is available after this timeout (milliseconds), return

    returns

    RecordStatus describing whether a record is available and its arrival time

    Exceptions thrown
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

  2. def initMetricsValues(metrics: Array[CustomTaskMetric]): Unit

    Sets the initial value of metrics before fetching any data from the reader.

    Sets the initial value of metrics before fetching any data from the reader. This is called when multiple PartitionReaders are grouped into one partition in case of org.apache.spark.sql.connector.read.partitioning.KeyGroupedPartitioning and the reader is initialized with the metrics returned by the previous reader that belongs to the same partition. By default, this method does nothing.

    Definition Classes
    PartitionReader
    Annotations
    @Deprecated
    Deprecated

    (Since version 4.2.0)

Inherited from PartitionReader[T]

Inherited from Closeable

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped