org.apache.spark.ui.jobs

Class JobProgressListener

Object

org.apache.spark.ui.jobs.JobProgressListener

All Implemented Interfaces:

Logging, SparkListener

public class JobProgressListener
extends Object
implements SparkListener, Logging

:: DeveloperApi :: Tracks task-level information to be displayed in the UI.

All access to the data structures in this class must be synchronized on the class, since the UI thread and the EventBus loop may otherwise be reading and updating the internal data structures concurrently.

Constructor Summary

Constructors

Constructor and Description
JobProgressListener(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.mutable.HashMap<Object,org.apache.spark.ui.jobs.UIData.JobUIData>	activeJobs()
scala.collection.mutable.HashMap<Object,StageInfo>	activeStages()
scala.collection.Seq<BlockManagerId>	blockManagerIds()
scala.collection.mutable.ListBuffer<org.apache.spark.ui.jobs.UIData.JobUIData>	completedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	completedStages()
long	endTime()
scala.collection.mutable.HashMap<String,BlockManagerId>	executorIdToBlockManagerId()
scala.collection.mutable.ListBuffer<org.apache.spark.ui.jobs.UIData.JobUIData>	failedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	failedStages()
scala.collection.mutable.HashMap<String,scala.collection.mutable.HashSet<Object>>	jobGroupToJobIds()
scala.collection.mutable.HashMap<Object,org.apache.spark.ui.jobs.UIData.JobUIData>	jobIdToData()
int	numCompletedJobs()
int	numCompletedStages()
int	numFailedJobs()
int	numFailedStages()
void	onApplicationEnd(SparkListenerApplicationEnd appEnded) Called when the application ends
void	onApplicationStart(SparkListenerApplicationStart appStarted) Called when the application starts
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updated
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.
void	onJobEnd(SparkListenerJobEnd jobEnd) Called when a job ends
void	onJobStart(SparkListenerJobStart jobStart) Called when a job starts
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart taskStart) Called when a task starts
scala.collection.mutable.HashMap<Object,StageInfo>	pendingStages()
scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>>	poolToActiveStages()
int	retainedJobs()
int	retainedStages()
scala.Option<scala.Enumeration.Value>	schedulingMode()
scala.collection.mutable.ListBuffer<StageInfo>	skippedStages()
scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashSet<Object>>	stageIdToActiveJobIds()
scala.collection.mutable.HashMap<scala.Tuple2<Object,Object>,org.apache.spark.ui.jobs.UIData.StageUIData>	stageIdToData()
scala.collection.mutable.HashMap<Object,StageInfo>	stageIdToInfo()
long	startTime()
void	updateAggregateMetrics(org.apache.spark.ui.jobs.UIData.StageUIData stageData, String execId, org.apache.spark.executor.TaskMetrics taskMetrics, scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics) Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onBlockUpdated, onExecutorAdded, onExecutorRemoved, onUnpersistRDD

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JobProgressListener

public JobProgressListener(SparkConf conf)

Method Detail

startTime

public long startTime()

endTime

public long endTime()

activeJobs

public scala.collection.mutable.HashMap<Object,org.apache.spark.ui.jobs.UIData.JobUIData> activeJobs()

completedJobs

public scala.collection.mutable.ListBuffer<org.apache.spark.ui.jobs.UIData.JobUIData> completedJobs()

failedJobs

public scala.collection.mutable.ListBuffer<org.apache.spark.ui.jobs.UIData.JobUIData> failedJobs()

jobIdToData

public scala.collection.mutable.HashMap<Object,org.apache.spark.ui.jobs.UIData.JobUIData> jobIdToData()

jobGroupToJobIds

public scala.collection.mutable.HashMap<String,scala.collection.mutable.HashSet<Object>> jobGroupToJobIds()

pendingStages

public scala.collection.mutable.HashMap<Object,StageInfo> pendingStages()

activeStages

public scala.collection.mutable.HashMap<Object,StageInfo> activeStages()

completedStages

public scala.collection.mutable.ListBuffer<StageInfo> completedStages()

skippedStages

public scala.collection.mutable.ListBuffer<StageInfo> skippedStages()

failedStages

public scala.collection.mutable.ListBuffer<StageInfo> failedStages()

stageIdToData

public scala.collection.mutable.HashMap<scala.Tuple2<Object,Object>,org.apache.spark.ui.jobs.UIData.StageUIData> stageIdToData()

stageIdToInfo

public scala.collection.mutable.HashMap<Object,StageInfo> stageIdToInfo()

stageIdToActiveJobIds

public scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashSet<Object>> stageIdToActiveJobIds()

poolToActiveStages

public scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>> poolToActiveStages()

numCompletedStages

public int numCompletedStages()

numFailedStages

public int numFailedStages()

numCompletedJobs

public int numCompletedJobs()

numFailedJobs

public int numFailedJobs()

executorIdToBlockManagerId

public scala.collection.mutable.HashMap<String,BlockManagerId> executorIdToBlockManagerId()

blockManagerIds

public scala.collection.Seq<BlockManagerId> blockManagerIds()

schedulingMode

public scala.Option<scala.Enumeration.Value> schedulingMode()

retainedStages

public int retainedStages()

retainedJobs

public int retainedJobs()

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

Description copied from interface: SparkListener

Called when a job starts

Specified by:

onJobStart in interface SparkListener

Parameters:

jobStart - (undocumented)

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

Description copied from interface: SparkListener

Called when a job ends

Specified by:

onJobEnd in interface SparkListener

Parameters:

jobEnd - (undocumented)

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Description copied from interface: SparkListener

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListener

Parameters:

stageCompleted - (undocumented)

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless

Specified by:

onStageSubmitted in interface SparkListener

Parameters:

stageSubmitted - (undocumented)

onTaskStart

public void onTaskStart(SparkListenerTaskStart taskStart)

Description copied from interface: SparkListener

Called when a task starts

Specified by:

onTaskStart in interface SparkListener

Parameters:

taskStart - (undocumented)

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)

Description copied from interface: SparkListener

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListener

Parameters:

taskGettingResult - (undocumented)

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

Parameters:

taskEnd - (undocumented)

updateAggregateMetrics

public void updateAggregateMetrics(org.apache.spark.ui.jobs.UIData.StageUIData stageData,
                          String execId,
                          org.apache.spark.executor.TaskMetrics taskMetrics,
                          scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics)

Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

Parameters:

stageData - (undocumented)

execId - (undocumented)

taskMetrics - (undocumented)

oldMetrics - (undocumented)

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate)

Description copied from interface: SparkListener

Called when the driver receives task metrics from an executor in a heartbeat.

Specified by:

onExecutorMetricsUpdate in interface SparkListener

Parameters:

executorMetricsUpdate - (undocumented)

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)

Description copied from interface: SparkListener

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListener

Parameters:

environmentUpdate - (undocumented)

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

Parameters:

blockManagerAdded - (undocumented)

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener

Parameters:

blockManagerRemoved - (undocumented)

onApplicationStart

public void onApplicationStart(SparkListenerApplicationStart appStarted)

Description copied from interface: SparkListener

Called when the application starts

Specified by:

onApplicationStart in interface SparkListener

Parameters:

appStarted - (undocumented)

onApplicationEnd

public void onApplicationEnd(SparkListenerApplicationEnd appEnded)

Description copied from interface: SparkListener

Called when the application ends

Specified by:

onApplicationEnd in interface SparkListener

Parameters:

appEnded - (undocumented)