Interface SupportsRealTimeRead<T>

All Superinterfaces:
AutoCloseable, Closeable, PartitionReader<T>

@Evolving public interface SupportsRealTimeRead<T> extends PartitionReader<T>
A variation on PartitionReader for use with low latency streaming processing.

Which method to implement. A source reader must provide a nextWithTimeout that proceeds to the next record, blocking until one is available or a timeout elapses. There are two overloads, and a source should override exactly one:

  • nextWithTimeout(Long) -- implement this in the common case. It receives only the timeout and is all a third-party source needs.
  • nextWithTimeout(Long, Long) -- the overload the execution engine actually invokes. Its default implementation ignores the extra parameter and delegates to nextWithTimeout(Long). Override it only for engine-internal sources that must observe the engine's reference start time (used together with the engine-internal low latency clock); that clock is not part of the third-party contract, so most sources should not override this overload.
Because both overloads are default methods, overriding neither is not caught at compile time -- it fails at runtime when nextWithTimeout(Long) throws.
  • Method Details

    • getOffset

      PartitionOffset getOffset()
      Get the offset of the next record, or the start offset if no records have been read.

      The execution engine will call this method along with get() to keep track of the current offset. When a task ends, the offset in each partition will be passed back to the driver. They will be used as the start offsets of the next batch.

    • nextWithTimeout

      default SupportsRealTimeRead.RecordStatus nextWithTimeout(Long timeoutMs) throws IOException
      Alternative function to be called than next(), that proceed to the next record. The different from next() is that, if there is no more records, the call needs to keep waiting until the timeout.

      This is the recommended method to implement for a source. It is enough for any source that does not need the engine's reference start time (see nextWithTimeout(Long, Long)). The engine always invokes nextWithTimeout(Long, Long), whose default implementation delegates here, so overriding only this method is sufficient. If a source overrides neither overload, this default implementation throws to surface the mistake.

      Parameters:
      timeoutMs - if no result is available after this timeout (milliseconds), return
      Returns:
      SupportsRealTimeRead.RecordStatus describing whether a record is available and its arrival time
      Throws:
      IOException
    • nextWithTimeout

      default SupportsRealTimeRead.RecordStatus nextWithTimeout(Long startTimeMs, Long timeoutMs) throws IOException
      The overload of nextWithTimeout(Long) that the execution engine actually invokes. In addition to the timeout it receives startTimeMs, the reference time the engine used to compute that timeout, so the engine and the source agree on when the wait started (this matters when the engine runs against its internal low latency clock, e.g. a manual clock in tests). The default implementation ignores startTimeMs and delegates to nextWithTimeout(Long).

      This is intended for engine-internal sources. The reference clock is not part of the third-party contract, so third-party sources should implement nextWithTimeout(Long) and leave this overload to its default.

      Parameters:
      startTimeMs - the base time (milliseconds) that was used to calculate the timeout
      timeoutMs - if no result is available after this timeout (milliseconds), return
      Returns:
      SupportsRealTimeRead.RecordStatus describing whether a record is available and its arrival time
      Throws:
      IOException