Interface SupportsRealTimeRead<T>
- All Superinterfaces:
AutoCloseable,Closeable,PartitionReader<T>
PartitionReader for use with low latency streaming processing.
Which method to implement. A source reader must provide a nextWithTimeout that
proceeds to the next record, blocking until one is available or a timeout elapses. There are two
overloads, and a source should override exactly one:
nextWithTimeout(Long)-- implement this in the common case. It receives only the timeout and is all a third-party source needs.nextWithTimeout(Long, Long)-- the overload the execution engine actually invokes. Its default implementation ignores the extra parameter and delegates tonextWithTimeout(Long). Override it only for engine-internal sources that must observe the engine's reference start time (used together with the engine-internal low latency clock); that clock is not part of the third-party contract, so most sources should not override this overload.
default methods, overriding neither is not caught at compile
time -- it fails at runtime when nextWithTimeout(Long) throws.-
Nested Class Summary
Nested ClassesModifier and TypeInterfaceDescriptionstatic classA class to represent the status of a record to be read as the return type of nextWithTimeout. -
Method Summary
Modifier and TypeMethodDescriptionGet the offset of the next record, or the start offset if no records have been read.nextWithTimeout(Long timeoutMs) Alternative function to be called than next(), that proceed to the next record.nextWithTimeout(Long startTimeMs, Long timeoutMs) The overload ofnextWithTimeout(Long)that the execution engine actually invokes.Methods inherited from interface org.apache.spark.sql.connector.read.PartitionReader
currentMetricsValues, get, initMetricsValues, next
-
Method Details
-
getOffset
PartitionOffset getOffset()Get the offset of the next record, or the start offset if no records have been read.The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.
-
nextWithTimeout
Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.This is the recommended method to implement for a source. It is enough for any source that does not need the engine's reference start time (see
nextWithTimeout(Long, Long)). The engine always invokesnextWithTimeout(Long, Long), whose default implementation delegates here, so overriding only this method is sufficient. If a source overrides neither overload, this default implementation throws to surface the mistake.- Parameters:
timeoutMs- if no result is available after this timeout (milliseconds), return- Returns:
SupportsRealTimeRead.RecordStatusdescribing whether a record is available and its arrival time- Throws:
IOException
-
nextWithTimeout
default SupportsRealTimeRead.RecordStatus nextWithTimeout(Long startTimeMs, Long timeoutMs) throws IOException The overload ofnextWithTimeout(Long)that the execution engine actually invokes. In addition to the timeout it receivesstartTimeMs, the reference time the engine used to compute that timeout, so the engine and the source agree on when the wait started (this matters when the engine runs against its internal low latency clock, e.g. a manual clock in tests). The default implementation ignoresstartTimeMsand delegates tonextWithTimeout(Long).This is intended for engine-internal sources. The reference clock is not part of the third-party contract, so third-party sources should implement
nextWithTimeout(Long)and leave this overload to its default.- Parameters:
startTimeMs- the base time (milliseconds) that was used to calculate the timeouttimeoutMs- if no result is available after this timeout (milliseconds), return- Returns:
SupportsRealTimeRead.RecordStatusdescribing whether a record is available and its arrival time- Throws:
IOException
-