Type Members

1.  trait WorkerDispatcher extends AutoCloseable

   :: Experimental :: Manages workers for a single WorkerSpecification and hides worker details from Spark.

   :: Experimental :: Manages workers for a single WorkerSpecification and hides worker details from Spark.

   A WorkerDispatcher is created from a worker specification (plus context such as security scope). It owns the underlying worker processes and connections, handling pooling, reuse, and lifecycle behind the scenes. Spark interacts with workers exclusively through the WorkerSessions returned by createSession.

   Annotations

   @Experimental()

2.  abstract class WorkerSecurityScope extends AnyRef

   :: Experimental :: Identifies a security boundary for worker connection pooling.

   :: Experimental :: Identifies a security boundary for worker connection pooling.

   Workers are only reused within the same security scope. Dispatcher implementations compare scopes using equals to decide whether an existing worker can be shared. Subclasses must override equals and hashCode so that structurally equivalent scopes match; otherwise, worker reuse will silently fail.

   Annotations

   @Experimental()

3.  abstract class WorkerSession extends AutoCloseable

   :: Experimental :: Represents one single UDF execution.

   :: Experimental :: Represents one single UDF execution.

   A WorkerSession is obtained from WorkerDispatcher#createSession and can carry per-execution state for that UDF invocation. Implementations may add concrete data-processing methods and lifecycle hooks as needed.

   A WorkerSession is not 1-to-1 mapped to an actual worker process. Multiple WorkerSessions may be backed by the same worker when the worker is reused. Worker reuse and pooling are managed by each WorkerDispatcher implementation based on the WorkerSpecification.

   Annotations

   @Experimental()