TaskSetManager (Spark 1.3.1 JavaDoc)

Object
- org.apache.spark.scheduler.TaskSetManager

All Implemented Interfaces:

Logging, Schedulable
```
public class TaskSetManager
extends Object
implements Schedulable, Logging
```
Schedules the tasks within a single TaskSet in the TaskSchedulerImpl. This class keeps track of each task, retries tasks if they fail (up to a limited number of times), and handles locality-aware scheduling for this TaskSet via delay scheduling. The main interfaces to it are resourceOffer, which asks the TaskSet whether it wants to run a task on one node, and statusUpdate, which tells it that one of its tasks changed state (e.g. finished).
THREADING: This class is designed to only be called from code with a lock on the TaskScheduler (e.g. its event handlers). It should not be called from other threads.

Constructor Summary

Constructors
Constructor and Description

TaskSetManager(TaskSchedulerImpl sched, TaskSet taskSet, int maxTaskFailures, Clock clock)

Constructors
Constructor and Description
`TaskSetManager(TaskSchedulerImpl sched, TaskSet taskSet, int maxTaskFailures, Clock clock)`

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`abort(String message)`
`void`	`addRunningTask(long tid)` If the given task ID is not in the set of running tasks, adds it.
`void`	`addSchedulable(Schedulable schedulable)`
`scala.collection.mutable.ArrayBuffer<Object>`	`allPendingTasks()`
`int`	`calculatedTasks()`
`boolean`	`canFetchMoreResults(long size)` Check whether has enough quota to fetch the result with `size` bytes
`boolean`	`checkSpeculatableTasks()` Check for tasks to be speculated and return true if there are any.
`SparkConf`	`conf()`
`int[]`	`copiesRunning()`
`int`	`currentLocalityIndex()`
`boolean`	`emittedTaskSizeWarning()`
`SparkEnv`	`env()`
`long`	`epoch()`
`long`	`EXCEPTION_PRINT_INTERVAL()`
`void`	`executorAdded()`
`void`	`executorLost(String execId, String host)` Called by TaskScheduler when an executor is lost so we can re-enqueue our tasks
`int`	`getLocalityIndex(scala.Enumeration.Value locality)` Find the index in myLocalityLevels for a given locality.
`Schedulable`	`getSchedulableByName(String name)`
`scala.collection.mutable.ArrayBuffer<TaskSetManager>`	`getSortedTaskSetQueue()`
`void`	`handleFailedTask(long tid, scala.Enumeration.Value state, TaskEndReason reason)` Marks the task as failed, re-adds it to the list of pending tasks, and notifies the DAG Scheduler.
`void`	`handleSuccessfulTask(long tid, DirectTaskResult<?> result)` Marks the task as successful and notifies the DAGScheduler that a task has ended.
`void`	`handleTaskGettingResult(long tid)` Marks the task as getting result and notifies the DAG Scheduler
`boolean`	`isZombie()`
`long`	`lastLaunchTime()`
`long[]`	`localityWaits()`
`long`	`maxResultSize()`
`int`	`maxTaskFailures()`
`int`	`minShare()`
`scala.Enumeration.Value[]`	`myLocalityLevels()`
`String`	`name()`
`int`	`numTasks()`
`Pool`	`parent()`
`scala.collection.mutable.ArrayBuffer<Object>`	`pendingTasksWithNoPrefs()`
`int`	`priority()`
`scala.collection.mutable.HashMap<String,scala.Tuple2<Object,Object>>`	`recentExceptions()`
`void`	`recomputeLocality()`
`void`	`removeRunningTask(long tid)` If the given task ID is in the set of running tasks, removes it.
`void`	`removeSchedulable(Schedulable schedulable)`
`scala.Option<TaskDescription>`	`resourceOffer(String execId, String host, scala.Enumeration.Value maxLocality)` Respond to an offer of a single executor from the scheduler by finding a task
`int`	`runningTasks()`
`scala.collection.mutable.HashSet<Object>`	`runningTasksSet()`
`scala.Null`	`schedulableQueue()`
`scala.Enumeration.Value`	`schedulingMode()`
`SerializerInstance`	`ser()`
`scala.collection.mutable.HashSet<Object>`	`speculatableTasks()`
`double`	`SPECULATION_MULTIPLIER()`
`double`	`SPECULATION_QUANTILE()`
`int`	`stageId()`
`boolean[]`	`successful()`
`static int`	`TASK_SIZE_TO_WARN_KB()`
`scala.collection.immutable.List<TaskInfo>[]`	`taskAttempts()`
`scala.collection.mutable.HashMap<Object,TaskInfo>`	`taskInfos()`
`Task<?>[]`	`tasks()`
`TaskSet`	`taskSet()`
`int`	`tasksSuccessful()`
`long`	`totalResultSize()`
`int`	`weight()`

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Constructor Detail
  - TaskSetManager
```
public TaskSetManager(TaskSchedulerImpl sched,
              TaskSet taskSet,
              int maxTaskFailures,
              Clock clock)
```
- Method Detail
  - TASK_SIZE_TO_WARN_KB
```
public static int TASK_SIZE_TO_WARN_KB()
```
  - taskSet
```
public TaskSet taskSet()
```
  - maxTaskFailures
```
public int maxTaskFailures()
```
  - conf
```
public SparkConf conf()
```
  - SPECULATION_QUANTILE
```
public double SPECULATION_QUANTILE()
```
  - SPECULATION_MULTIPLIER
```
public double SPECULATION_MULTIPLIER()
```
  - maxResultSize
```
public long maxResultSize()
```
  - env
```
public SparkEnv env()
```
  - ser
```
public SerializerInstance ser()
```
  - tasks
```
public Task<?>[] tasks()
```
  - numTasks
```
public int numTasks()
```
  - copiesRunning
```
public int[] copiesRunning()
```
  - successful
```
public boolean[] successful()
```
  - taskAttempts
```
public scala.collection.immutable.List<TaskInfo>[] taskAttempts()
```
  - tasksSuccessful
```
public int tasksSuccessful()
```
  - weight
```
public int weight()
```
    Specified by:
    
    weight in interface Schedulable
  - minShare
```
public int minShare()
```
    Specified by:
    
    minShare in interface Schedulable
  - priority
```
public int priority()
```
    Specified by:
    
    priority in interface Schedulable
  - stageId
```
public int stageId()
```
    Specified by:
    
    stageId in interface Schedulable
  - name
```
public String name()
```
    Specified by:
    
    name in interface Schedulable
  - parent
```
public Pool parent()
```
    Specified by:
    
    parent in interface Schedulable
  - totalResultSize
```
public long totalResultSize()
```
  - calculatedTasks
```
public int calculatedTasks()
```
  - runningTasksSet
```
public scala.collection.mutable.HashSet<Object> runningTasksSet()
```
  - runningTasks
```
public int runningTasks()
```
    Specified by:
    
    runningTasks in interface Schedulable
  - isZombie
```
public boolean isZombie()
```
  - pendingTasksWithNoPrefs
```
public scala.collection.mutable.ArrayBuffer<Object> pendingTasksWithNoPrefs()
```
  - allPendingTasks
```
public scala.collection.mutable.ArrayBuffer<Object> allPendingTasks()
```
  - speculatableTasks
```
public scala.collection.mutable.HashSet<Object> speculatableTasks()
```
  - taskInfos
```
public scala.collection.mutable.HashMap<Object,TaskInfo> taskInfos()
```
  - EXCEPTION_PRINT_INTERVAL
```
public long EXCEPTION_PRINT_INTERVAL()
```
  - recentExceptions
```
public scala.collection.mutable.HashMap<String,scala.Tuple2<Object,Object>> recentExceptions()
```
  - epoch
```
public long epoch()
```
  - myLocalityLevels
```
public scala.Enumeration.Value[] myLocalityLevels()
```
  - localityWaits
```
public long[] localityWaits()
```
  - currentLocalityIndex
```
public int currentLocalityIndex()
```
  - lastLaunchTime
```
public long lastLaunchTime()
```
  - schedulableQueue
```
public scala.Null schedulableQueue()
```
    Specified by:
    
    schedulableQueue in interface Schedulable
  - schedulingMode
```
public scala.Enumeration.Value schedulingMode()
```
    Specified by:
    
    schedulingMode in interface Schedulable
  - emittedTaskSizeWarning
```
public boolean emittedTaskSizeWarning()
```
  - resourceOffer
```
public scala.Option<TaskDescription> resourceOffer(String execId,
                                          String host,
                                          scala.Enumeration.Value maxLocality)
```
    Respond to an offer of a single executor from the scheduler by finding a task
    NOTE: this function is either called with a maxLocality which would be adjusted by delay scheduling algorithm or it will be with a special NO_PREF locality which will be not modified
    
    Parameters:
    execId - the executor Id of the offered resource
    host - the host Id of the offered resource
    maxLocality - the maximum locality we want to schedule the tasks at
  - getLocalityIndex
```
public int getLocalityIndex(scala.Enumeration.Value locality)
```
    Find the index in myLocalityLevels for a given locality. This is also designed to work with localities that are not in myLocalityLevels (in case we somehow get those) by returning the next-biggest level we have. Uses the fact that the last value in myLocalityLevels is ANY.
  - handleTaskGettingResult
```
public void handleTaskGettingResult(long tid)
```
    Marks the task as getting result and notifies the DAG Scheduler
  - canFetchMoreResults
```
public boolean canFetchMoreResults(long size)
```
    Check whether has enough quota to fetch the result with size bytes
  - handleSuccessfulTask
```
public void handleSuccessfulTask(long tid,
                        DirectTaskResult<?> result)
```
    Marks the task as successful and notifies the DAGScheduler that a task has ended.
  - handleFailedTask
```
public void handleFailedTask(long tid,
                    scala.Enumeration.Value state,
                    TaskEndReason reason)
```
    Marks the task as failed, re-adds it to the list of pending tasks, and notifies the DAG Scheduler.
  - abort
```
public void abort(String message)
```
  - addRunningTask
```
public void addRunningTask(long tid)
```
    If the given task ID is not in the set of running tasks, adds it.
    Used to keep track of the number of running tasks, for enforcing scheduling policies.
  - removeRunningTask
```
public void removeRunningTask(long tid)
```
    If the given task ID is in the set of running tasks, removes it.
  - getSchedulableByName
```
public Schedulable getSchedulableByName(String name)
```
    Specified by:
    
    getSchedulableByName in interface Schedulable
  - addSchedulable
```
public void addSchedulable(Schedulable schedulable)
```
    Specified by:
    
    addSchedulable in interface Schedulable
  - removeSchedulable
```
public void removeSchedulable(Schedulable schedulable)
```
    Specified by:
    
    removeSchedulable in interface Schedulable
  - getSortedTaskSetQueue
```
public scala.collection.mutable.ArrayBuffer<TaskSetManager> getSortedTaskSetQueue()
```
    Specified by:
    
    getSortedTaskSetQueue in interface Schedulable
  - executorLost
```
public void executorLost(String execId,
                String host)
```
    Called by TaskScheduler when an executor is lost so we can re-enqueue our tasks
    
    Specified by:
    
    executorLost in interface Schedulable
  - checkSpeculatableTasks
```
public boolean checkSpeculatableTasks()
```
    Check for tasks to be speculated and return true if there are any. This is called periodically by the TaskScheduler.
    TODO: To make this scale to large jobs, we need to maintain a list of running tasks, so that we don't scan the whole task set. It might also help to make this sorted by launch time.
    
    Specified by:
    
    checkSpeculatableTasks in interface Schedulable
  - recomputeLocality
```
public void recomputeLocality()
```
  - executorAdded
```
public void executorAdded()
```

Class TaskSetManager

Constructor Summary

Method Summary

Methods inherited from class Object

Methods inherited from interface org.apache.spark.Logging

Constructor Detail

TaskSetManager

Method Detail

TASK_SIZE_TO_WARN_KB

taskSet

maxTaskFailures

conf

SPECULATION_QUANTILE

SPECULATION_MULTIPLIER

maxResultSize

env

ser

tasks

numTasks

copiesRunning

successful

taskAttempts

tasksSuccessful

weight

minShare

priority

stageId

name

parent

totalResultSize

calculatedTasks

runningTasksSet

runningTasks

isZombie

pendingTasksWithNoPrefs

allPendingTasks

speculatableTasks

taskInfos

EXCEPTION_PRINT_INTERVAL

recentExceptions

epoch

myLocalityLevels

localityWaits

currentLocalityIndex

lastLaunchTime

schedulableQueue

schedulingMode

emittedTaskSizeWarning

resourceOffer

getLocalityIndex

handleTaskGettingResult

canFetchMoreResults

handleSuccessfulTask

handleFailedTask

abort

addRunningTask

removeRunningTask

getSchedulableByName

addSchedulable

removeSchedulable

getSortedTaskSetQueue

executorLost

checkSpeculatableTasks

recomputeLocality

executorAdded