org.apache.spark.sql.connector.read.streaming

Interface SupportsTriggerAvailableNow

All Superinterfaces:

SparkDataStream, SupportsAdmissionControl

@Evolving
public interface SupportsTriggerAvailableNow
extends SupportsAdmissionControl

An interface for streaming sources that supports running in Trigger.AvailableNow mode, which will process all the available data at the beginning of the query in (possibly) multiple batches. This mode will have better scalability comparing to Trigger.Once mode.

Since:

3.3.0

Method Summary

Modifier and Type	Method and Description
void	prepareForTriggerAvailableNow() This will be called at the beginning of streaming queries with Trigger.AvailableNow, to let the source record the offset for the current latest data at the time (a.k.a the target offset for the query).

Methods inherited from interface org.apache.spark.sql.connector.read.streaming.SupportsAdmissionControl

getDefaultReadLimit, latestOffset, reportLatestOffset

Methods inherited from interface org.apache.spark.sql.connector.read.streaming.SparkDataStream

commit, deserializeOffset, initialOffset, stop

Method Detail

prepareForTriggerAvailableNow

void prepareForTriggerAvailableNow()

This will be called at the beginning of streaming queries with Trigger.AvailableNow, to let the source record the offset for the current latest data at the time (a.k.a the target offset for the query). The source will behave as if there is no new data coming in after the target offset, i.e., the source will not return an offset higher than the target offset when latestOffset is called.