org.apache.spark.sql.sources.v2.reader

Interface DataReaderFactory<T>

All Superinterfaces:

java.io.Serializable

@InterfaceStability.Evolving
public interface DataReaderFactory<T>
extends java.io.Serializable

A reader factory returned by DataSourceReader.createDataReaderFactories() and is responsible for creating the actual data reader. The relationship between DataReaderFactory and DataReader is similar to the relationship between Iterable and Iterator. Note that, the reader factory will be serialized and sent to executors, then the data reader will be created on executors and do the actual reading. So DataReaderFactory must be serializable and DataReader doesn't need to be.

Method Summary

Modifier and Type	Method and Description
DataReader<T>	createDataReader() Returns a data reader to do the actual reading work.
default String[]	preferredLocations() The preferred locations where the data reader returned by this reader factory can run faster, but Spark does not guarantee to run the data reader on these locations.

Method Detail

preferredLocations

default String[] preferredLocations()

The preferred locations where the data reader returned by this reader factory can run faster, but Spark does not guarantee to run the data reader on these locations. The implementations should make sure that it can be run on any location. The location is a string representing the host name. Note that if a host name cannot be recognized by Spark, it will be ignored as it was not in the returned locations. By default this method returns empty string array, which means this task has no location preference. If this method fails (by throwing an exception), the action would fail and no Spark job was submitted.

createDataReader

DataReader<T> createDataReader()

Returns a data reader to do the actual reading work. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.