org.apache.spark.ui.jobs

Class JobProgressListener

java.lang.Object

org.apache.spark.scheduler.SparkListener

org.apache.spark.ui.jobs.JobProgressListener

public class JobProgressListener
extends SparkListener

:: DeveloperApi :: Tracks task-level information to be displayed in the UI.

All access to the data structures in this class must be synchronized on the class, since the UI thread and the EventBus loop may otherwise be reading and updating the internal data structures concurrently.

Constructor Summary

Constructors

Constructor and Description
JobProgressListener(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.mutable.HashMap<java.lang.Object,UIData.JobUIData>	activeJobs()
scala.collection.mutable.HashMap<java.lang.Object,StageInfo>	activeStages()
scala.collection.Seq<BlockManagerId>	blockManagerIds()
scala.collection.mutable.ListBuffer<UIData.JobUIData>	completedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	completedStages()
long	endTime()
scala.collection.mutable.HashMap<java.lang.String,BlockManagerId>	executorIdToBlockManagerId()
scala.collection.mutable.ListBuffer<UIData.JobUIData>	failedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	failedStages()
scala.collection.mutable.HashMap<java.lang.String,scala.collection.mutable.HashSet<java.lang.Object>>	jobGroupToJobIds()
scala.collection.mutable.HashMap<java.lang.Object,UIData.JobUIData>	jobIdToData()
int	numCompletedJobs()
int	numCompletedStages()
int	numFailedJobs()
int	numFailedStages()
void	onApplicationEnd(SparkListenerApplicationEnd appEnded) Called when the application ends
void	onApplicationStart(SparkListenerApplicationStart appStarted) Called when the application starts
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updated
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.
void	onJobEnd(SparkListenerJobEnd jobEnd) Called when a job ends
void	onJobStart(SparkListenerJobStart jobStart) Called when a job starts
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart taskStart) Called when a task starts
scala.collection.mutable.HashMap<java.lang.Object,StageInfo>	pendingStages()
scala.collection.mutable.HashMap<java.lang.String,scala.collection.mutable.HashMap<java.lang.Object,StageInfo>>	poolToActiveStages()
int	retainedJobs()
int	retainedStages()
scala.Option<scala.Enumeration.Value>	schedulingMode()
scala.collection.mutable.ListBuffer<StageInfo>	skippedStages()
scala.collection.mutable.HashMap<java.lang.Object,scala.collection.mutable.HashSet<java.lang.Object>>	stageIdToActiveJobIds()
scala.collection.mutable.HashMap<scala.Tuple2<java.lang.Object,java.lang.Object>,UIData.StageUIData>	stageIdToData()
scala.collection.mutable.HashMap<java.lang.Object,StageInfo>	stageIdToInfo()
long	startTime()
void	updateAggregateMetrics(UIData.StageUIData stageData, java.lang.String execId, org.apache.spark.executor.TaskMetrics taskMetrics, scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics) Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

Methods inherited from class org.apache.spark.scheduler.SparkListener

onBlockUpdated, onExecutorAdded, onExecutorRemoved, onOtherEvent, onUnpersistRDD

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JobProgressListener

public JobProgressListener(SparkConf conf)

Method Detail

startTime

public long startTime()

endTime

public long endTime()

activeJobs

public scala.collection.mutable.HashMap<java.lang.Object,UIData.JobUIData> activeJobs()

completedJobs

public scala.collection.mutable.ListBuffer<UIData.JobUIData> completedJobs()

failedJobs

public scala.collection.mutable.ListBuffer<UIData.JobUIData> failedJobs()

jobIdToData

public scala.collection.mutable.HashMap<java.lang.Object,UIData.JobUIData> jobIdToData()

jobGroupToJobIds

public scala.collection.mutable.HashMap<java.lang.String,scala.collection.mutable.HashSet<java.lang.Object>> jobGroupToJobIds()

pendingStages

public scala.collection.mutable.HashMap<java.lang.Object,StageInfo> pendingStages()

activeStages

public scala.collection.mutable.HashMap<java.lang.Object,StageInfo> activeStages()

completedStages

public scala.collection.mutable.ListBuffer<StageInfo> completedStages()

skippedStages

public scala.collection.mutable.ListBuffer<StageInfo> skippedStages()

failedStages

public scala.collection.mutable.ListBuffer<StageInfo> failedStages()

stageIdToData

public scala.collection.mutable.HashMap<scala.Tuple2<java.lang.Object,java.lang.Object>,UIData.StageUIData> stageIdToData()

stageIdToInfo

public scala.collection.mutable.HashMap<java.lang.Object,StageInfo> stageIdToInfo()

stageIdToActiveJobIds

public scala.collection.mutable.HashMap<java.lang.Object,scala.collection.mutable.HashSet<java.lang.Object>> stageIdToActiveJobIds()

poolToActiveStages

public scala.collection.mutable.HashMap<java.lang.String,scala.collection.mutable.HashMap<java.lang.Object,StageInfo>> poolToActiveStages()

numCompletedStages

public int numCompletedStages()

numFailedStages

public int numFailedStages()

numCompletedJobs

public int numCompletedJobs()

numFailedJobs

public int numFailedJobs()

executorIdToBlockManagerId

public scala.collection.mutable.HashMap<java.lang.String,BlockManagerId> executorIdToBlockManagerId()

blockManagerIds

public scala.collection.Seq<BlockManagerId> blockManagerIds()

schedulingMode

public scala.Option<scala.Enumeration.Value> schedulingMode()

retainedStages

public int retainedStages()

retainedJobs

public int retainedJobs()

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

Called when a job starts

Overrides:

onJobStart in class SparkListener

Parameters:

jobStart - (undocumented)

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

Called when a job ends

Overrides:

onJobEnd in class SparkListener

Parameters:

jobEnd - (undocumented)

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Called when a stage completes successfully or fails, with information on the completed stage.

Overrides:

onStageCompleted in class SparkListener

Parameters:

stageCompleted - (undocumented)

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless

Overrides:

onStageSubmitted in class SparkListener

Parameters:

stageSubmitted - (undocumented)

onTaskStart

public void onTaskStart(SparkListenerTaskStart taskStart)

Called when a task starts

Overrides:

onTaskStart in class SparkListener

Parameters:

taskStart - (undocumented)

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Overrides:

onTaskGettingResult in class SparkListener

Parameters:

taskGettingResult - (undocumented)

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Called when a task ends

Overrides:

onTaskEnd in class SparkListener

Parameters:

taskEnd - (undocumented)

updateAggregateMetrics

public void updateAggregateMetrics(UIData.StageUIData stageData,
                          java.lang.String execId,
                          org.apache.spark.executor.TaskMetrics taskMetrics,
                          scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics)

Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

Parameters:

stageData - (undocumented)

execId - (undocumented)

taskMetrics - (undocumented)

oldMetrics - (undocumented)

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate)

Called when the driver receives task metrics from an executor in a heartbeat.

Overrides:

onExecutorMetricsUpdate in class SparkListener

Parameters:

executorMetricsUpdate - (undocumented)

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)

Called when environment properties have been updated

Overrides:

onEnvironmentUpdate in class SparkListener

Parameters:

environmentUpdate - (undocumented)

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Called when a new block manager has joined

Overrides:

onBlockManagerAdded in class SparkListener

Parameters:

blockManagerAdded - (undocumented)

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Called when an existing block manager has been removed

Overrides:

onBlockManagerRemoved in class SparkListener

Parameters:

blockManagerRemoved - (undocumented)

onApplicationStart

public void onApplicationStart(SparkListenerApplicationStart appStarted)

Called when the application starts

Overrides:

onApplicationStart in class SparkListener

Parameters:

appStarted - (undocumented)

onApplicationEnd

public void onApplicationEnd(SparkListenerApplicationEnd appEnded)

Called when the application ends

Overrides:

onApplicationEnd in class SparkListener

Parameters:

appEnded - (undocumented)