Package org.apache.spark.sql.connector.read.streaming

Interface SupportsRealTimeRead<T>

All Superinterfaces:

AutoCloseable, Closeable, PartitionReader<T>

@Evolving
public interface SupportsRealTimeRead<T>
extends PartitionReader<T>

A variation on PartitionReader for use with low latency streaming processing.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	SupportsRealTimeRead.RecordStatus	A class to represent the status of a record to be read as the return type of nextWithTimeout.

Method Summary

Modifier and Type	Method	Description
PartitionOffset	getOffset()	Get the offset of the next record, or the start offset if no records have been read.
SupportsRealTimeRead.RecordStatus	nextWithTimeout(Long startTimeMs, Long timeoutMs)	Alternative function to be called than next(), that proceed to the next record.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.apache.spark.sql.connector.read.PartitionReader

currentMetricsValues, get, initMetricsValues, next

Method Details

getOffset

PartitionOffset getOffset()

Get the offset of the next record, or the start offset if no records have been read.

The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.

nextWithTimeout

SupportsRealTimeRead.RecordStatus nextWithTimeout(Long startTimeMs,
 Long timeoutMs)
                                           throws IOException

Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.

Parameters:

startTimeMs - the base time (milliseconds) the was used to calculate the timeout. Sources should use it as the reference time to start waiting for the next record instead of getting the latest time from LowLatencyClock.

timeoutMs - if no result is available after this timeout (milliseconds), return

Returns:

SupportsRealTimeRead.RecordStatus describing whether a record is available and its arrival time

Throws:

IOException