trait SupportsRealTimeRead[T] extends PartitionReader[T]
A variation on PartitionReader for use with low latency streaming processing.
Which method to implement. A source reader must provide a nextWithTimeout that
proceeds to the next record, blocking until one is available or a timeout elapses. There are two
overloads, and a source should override exactly one:
#nextWithTimeout(Long)-- implement this in the common case. It receives only the timeout and is all a third-party source needs.Long)-- the overload the execution engine actually invokes. Its default implementation ignores the extra parameter and delegates to#nextWithTimeout(Long). Override it only for engine-internal sources that must observe the engine's reference start time (used together with the engine-internal low latency clock); that clock is not part of the third-party contract, so most sources should not override this overload.
Because both overloads are default methods, overriding neither is not caught at compile
time -- it fails at runtime when #nextWithTimeout(Long) throws.
- Annotations
- @Evolving()
- Source
- SupportsRealTimeRead.java
- Alphabetic
- By Inheritance
- SupportsRealTimeRead
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def close(): Unit
- Definition Classes
- Closeable → AutoCloseable
- Annotations
- @throws(classOf[java.io.IOException])
- abstract def get(): T
Return the current record.
Return the current record. This method should return same value until
nextis called.- Definition Classes
- PartitionReader
- abstract def getOffset(): PartitionOffset
Get the offset of the next record, or the start offset if no records have been read.
Get the offset of the next record, or the start offset if no records have been read.
The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.
- abstract def next(): Boolean
Proceed to next record, returns false if there is no more records.
Proceed to next record, returns false if there is no more records.
- Definition Classes
- PartitionReader
- Exceptions thrown
IOExceptionif failure happens during disk/network IO like reading files.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def currentMetricsValues(): Array[CustomTaskMetric]
Returns an array of custom task metrics.
Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.
- Definition Classes
- PartitionReader
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def nextWithTimeout(startTimeMs: Long, timeoutMs: Long): RecordStatus
The overload of
#nextWithTimeout(Long)that the execution engine actually invokes.The overload of
#nextWithTimeout(Long)that the execution engine actually invokes. In addition to the timeout it receivesstartTimeMs, the reference time the engine used to compute that timeout, so the engine and the source agree on when the wait started (this matters when the engine runs against its internal low latency clock, e.g. a manual clock in tests). The default implementation ignoresstartTimeMsand delegates to#nextWithTimeout(Long).This is intended for engine-internal sources. The reference clock is not part of the third-party contract, so third-party sources should implement
#nextWithTimeout(Long)and leave this overload to its default.- startTimeMs
the base time (milliseconds) that was used to calculate the timeout
- timeoutMs
if no result is available after this timeout (milliseconds), return
- returns
RecordStatusdescribing whether a record is available and its arrival time
- Exceptions thrown
- def nextWithTimeout(timeoutMs: Long): RecordStatus
Alternative function to be called than next(), that proceed to the next record.
Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.
This is the recommended method to implement for a source. It is enough for any source that does not need the engine's reference start time (see
Long)). The engine always invokesLong), whose default implementation delegates here, so overriding only this method is sufficient. If a source overrides neither overload, this default implementation throws to surface the mistake.- timeoutMs
if no result is available after this timeout (milliseconds), return
- returns
RecordStatusdescribing whether a record is available and its arrival time
- Exceptions thrown
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)
- def initMetricsValues(metrics: Array[CustomTaskMetric]): Unit
Sets the initial value of metrics before fetching any data from the reader.
Sets the initial value of metrics before fetching any data from the reader. This is called when multiple
PartitionReaders are grouped into one partition in case oforg.apache.spark.sql.connector.read.partitioning.KeyGroupedPartitioningand the reader is initialized with the metrics returned by the previous reader that belongs to the same partition. By default, this method does nothing.- Definition Classes
- PartitionReader
- Annotations
- @Deprecated
- Deprecated
(Since version 4.2.0)