org.apache.spark.sql.sources.v2.reader.streaming

Interface MicroBatchReader

All Superinterfaces:

org.apache.spark.sql.execution.streaming.BaseStreamingSource, DataSourceReader

@InterfaceStability.Evolving
public interface MicroBatchReader
extends DataSourceReader, org.apache.spark.sql.execution.streaming.BaseStreamingSource

A mix-in interface for DataSourceReader. Data source readers can implement this interface to indicate they allow micro-batch streaming reads. Note: This class currently extends BaseStreamingSource to maintain compatibility with DataSource V1 APIs. This extension will be removed once we get rid of V1 completely.

Method Summary

Modifier and Type	Method and Description
void	commit(Offset end) Informs the source that Spark has completed processing all data for offsets less than or equal to `end` and will only request offsets greater than `end` in the future.
Offset	deserializeOffset(String json) Deserialize a JSON string into an Offset of the implementation-defined offset type.
Offset	getEndOffset() Return the specified (if explicitly set through setOffsetRange) or inferred end offset for this reader.
Offset	getStartOffset() Returns the specified (if explicitly set through setOffsetRange) or inferred start offset for this reader.
void	setOffsetRange(java.util.Optional<Offset> start, java.util.Optional<Offset> end) Set the desired offset range for input partitions created from this reader.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceReader

planInputPartitions, readSchema

Methods inherited from interface org.apache.spark.sql.execution.streaming.BaseStreamingSource

stop

Method Detail

setOffsetRange

void setOffsetRange(java.util.Optional<Offset> start,
                    java.util.Optional<Offset> end)

Set the desired offset range for input partitions created from this reader. Partition readers will generate only data within (`start`, `end`]; that is, from the first record after `start` to the record with offset `end`.

Parameters:

start - The initial offset to scan from. If not specified, scan from an implementation-specified start point, such as the earliest available record.

end - The last offset to include in the scan. If not specified, scan up to an implementation-defined endpoint, such as the last available offset or the start offset plus a target batch size.

getStartOffset

Offset getStartOffset()

Returns the specified (if explicitly set through setOffsetRange) or inferred start offset for this reader.

Throws:

IllegalStateException - if setOffsetRange has not been called

getEndOffset

Offset getEndOffset()

Return the specified (if explicitly set through setOffsetRange) or inferred end offset for this reader.

Throws:

IllegalStateException - if setOffsetRange has not been called

deserializeOffset

Offset deserializeOffset(String json)

Deserialize a JSON string into an Offset of the implementation-defined offset type.

Throws:

IllegalArgumentException - if the JSON does not encode a valid offset for this reader

commit

void commit(Offset end)

Informs the source that Spark has completed processing all data for offsets less than or equal to `end` and will only request offsets greater than `end` in the future.