org.apache.spark.ui.jobs

Class JobProgressListener

Object

org.apache.spark.ui.jobs.JobProgressListener

All Implemented Interfaces:

SparkListener

public class JobProgressListener
extends Object
implements SparkListener

:: DeveloperApi :: Tracks task-level information to be displayed in the UI.

All access to the data structures in this class must be synchronized on the class, since the UI thread and the EventBus loop may otherwise be reading and updating the internal data structures concurrently.

Constructor Summary

Constructors

Constructor and Description
JobProgressListener(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.mutable.HashMap<Object,StageInfo>	activeStages()
scala.collection.Seq<BlockManagerId>	blockManagerIds()
scala.collection.mutable.ListBuffer<StageInfo>	completedStages()
static String	DEFAULT_POOL_NAME()
static int	DEFAULT_RETAINED_STAGES()
scala.collection.mutable.HashMap<String,BlockManagerId>	executorIdToBlockManagerId()
scala.collection.mutable.ListBuffer<StageInfo>	failedStages()
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updated
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart taskStart) Called when a task starts
scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>>	poolToActiveStages()
int	retainedStages()
scala.Option<scala.Enumeration.Value>	schedulingMode()
scala.collection.mutable.HashMap<Object,String>	stageIdToDescription()
scala.collection.mutable.HashMap<Object,Object>	stageIdToDiskBytesSpilled()
scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<String,ExecutorSummary>>	stageIdToExecutorSummaries()
scala.collection.mutable.HashMap<Object,Object>	stageIdToMemoryBytesSpilled()
scala.collection.mutable.HashMap<Object,String>	stageIdToPool()
scala.collection.mutable.HashMap<Object,Object>	stageIdToShuffleRead()
scala.collection.mutable.HashMap<Object,Object>	stageIdToShuffleWrite()
scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<Object,TaskUIData>>	stageIdToTaskData()
scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<Object,TaskInfo>>	stageIdToTasksActive()
scala.collection.mutable.HashMap<Object,Object>	stageIdToTasksComplete()
scala.collection.mutable.HashMap<Object,Object>	stageIdToTasksFailed()
scala.collection.mutable.HashMap<Object,Object>	stageIdToTime()
long	totalShuffleRead()
long	totalShuffleWrite()
long	totalTime()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onJobEnd, onJobStart, onUnpersistRDD

Constructor Detail

JobProgressListener

public JobProgressListener(SparkConf conf)

Method Detail

DEFAULT_POOL_NAME

public static String DEFAULT_POOL_NAME()

DEFAULT_RETAINED_STAGES

public static int DEFAULT_RETAINED_STAGES()

retainedStages

public int retainedStages()

activeStages

public scala.collection.mutable.HashMap<Object,StageInfo> activeStages()

completedStages

public scala.collection.mutable.ListBuffer<StageInfo> completedStages()

failedStages

public scala.collection.mutable.ListBuffer<StageInfo> failedStages()

totalTime

public long totalTime()

totalShuffleRead

public long totalShuffleRead()

totalShuffleWrite

public long totalShuffleWrite()

stageIdToTime

public scala.collection.mutable.HashMap<Object,Object> stageIdToTime()

stageIdToShuffleRead

public scala.collection.mutable.HashMap<Object,Object> stageIdToShuffleRead()

stageIdToShuffleWrite

public scala.collection.mutable.HashMap<Object,Object> stageIdToShuffleWrite()

stageIdToMemoryBytesSpilled

public scala.collection.mutable.HashMap<Object,Object> stageIdToMemoryBytesSpilled()

stageIdToDiskBytesSpilled

public scala.collection.mutable.HashMap<Object,Object> stageIdToDiskBytesSpilled()

stageIdToTasksActive

public scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<Object,TaskInfo>> stageIdToTasksActive()

stageIdToTasksComplete

public scala.collection.mutable.HashMap<Object,Object> stageIdToTasksComplete()

stageIdToTasksFailed

public scala.collection.mutable.HashMap<Object,Object> stageIdToTasksFailed()

stageIdToTaskData

public scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<Object,TaskUIData>> stageIdToTaskData()

stageIdToExecutorSummaries

public scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashMap<String,ExecutorSummary>> stageIdToExecutorSummaries()

stageIdToPool

public scala.collection.mutable.HashMap<Object,String> stageIdToPool()

stageIdToDescription

public scala.collection.mutable.HashMap<Object,String> stageIdToDescription()

poolToActiveStages

public scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>> poolToActiveStages()

executorIdToBlockManagerId

public scala.collection.mutable.HashMap<String,BlockManagerId> executorIdToBlockManagerId()

schedulingMode

public scala.Option<scala.Enumeration.Value> schedulingMode()

blockManagerIds

public scala.collection.Seq<BlockManagerId> blockManagerIds()

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Description copied from interface: SparkListener

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListener

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless

Specified by:

onStageSubmitted in interface SparkListener

onTaskStart

public void onTaskStart(SparkListenerTaskStart taskStart)

Description copied from interface: SparkListener

Called when a task starts

Specified by:

onTaskStart in interface SparkListener

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)

Description copied from interface: SparkListener

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListener

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)

Description copied from interface: SparkListener

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListener

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener