trait SupportsRealTimeRead[T] extends PartitionReader[T]
A variation on PartitionReader for use with low latency streaming processing.
- Annotations
- @Evolving()
- Source
- SupportsRealTimeRead.java
- Alphabetic
- By Inheritance
- SupportsRealTimeRead
- PartitionReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def close(): Unit
- Definition Classes
- Closeable → AutoCloseable
- Annotations
- @throws(classOf[java.io.IOException])
- abstract def get(): T
Return the current record.
Return the current record. This method should return same value until
nextis called.- Definition Classes
- PartitionReader
- abstract def getOffset(): PartitionOffset
Get the offset of the next record, or the start offset if no records have been read.
Get the offset of the next record, or the start offset if no records have been read.
The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.
- abstract def next(): Boolean
Proceed to next record, returns false if there is no more records.
Proceed to next record, returns false if there is no more records.
- Definition Classes
- PartitionReader
- Exceptions thrown
IOExceptionif failure happens during disk/network IO like reading files.
- abstract def nextWithTimeout(startTimeMs: Long, timeoutMs: Long): RecordStatus
Alternative function to be called than next(), that proceed to the next record.
Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.
- startTimeMs
the base time (milliseconds) the was used to calculate the timeout. Sources should use it as the reference time to start waiting for the next record instead of getting the latest time from LowLatencyClock.
- timeoutMs
if no result is available after this timeout (milliseconds), return
- returns
RecordStatusdescribing whether a record is available and its arrival time
- Exceptions thrown
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def currentMetricsValues(): Array[CustomTaskMetric]
Returns an array of custom task metrics.
Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.
- Definition Classes
- PartitionReader
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def initMetricsValues(metrics: Array[CustomTaskMetric]): Unit
Sets the initial value of metrics before fetching any data from the reader.
Sets the initial value of metrics before fetching any data from the reader. This is called when multiple
PartitionReaders are grouped into one partition in case oforg.apache.spark.sql.connector.read.partitioning.KeyGroupedPartitioningand the reader is initialized with the metrics returned by the previous reader that belongs to the same partition. By default, this method does nothing.- Definition Classes
- PartitionReader
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)