org.apache.spark.ui.jobs

Class JobProgressListener

Object

org.apache.spark.ui.jobs.JobProgressListener

All Implemented Interfaces:

Logging, SparkListener

public class JobProgressListener
extends Object
implements SparkListener, Logging

:: DeveloperApi :: Tracks task-level information to be displayed in the UI.

All access to the data structures in this class must be synchronized on the class, since the UI thread and the EventBus loop may otherwise be reading and updating the internal data structures concurrently.

Constructor Summary

Constructors

Constructor and Description
JobProgressListener(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.mutable.HashMap<Object,UIData.JobUIData>	activeJobs()
scala.collection.mutable.HashMap<Object,StageInfo>	activeStages()
scala.collection.Seq<BlockManagerId>	blockManagerIds()
scala.collection.mutable.ListBuffer<UIData.JobUIData>	completedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	completedStages()
static String	DEFAULT_POOL_NAME()
static int	DEFAULT_RETAINED_JOBS()
static int	DEFAULT_RETAINED_STAGES()
scala.collection.mutable.HashMap<String,BlockManagerId>	executorIdToBlockManagerId()
scala.collection.mutable.ListBuffer<UIData.JobUIData>	failedJobs()
scala.collection.mutable.ListBuffer<StageInfo>	failedStages()
scala.collection.immutable.Map<String,Object>	getSizesOfActiveStateTrackingCollections()
scala.collection.immutable.Map<String,Object>	getSizesOfHardSizeLimitedCollections()
scala.collection.immutable.Map<String,Object>	getSizesOfSoftSizeLimitedCollections()
scala.collection.mutable.HashMap<Object,UIData.JobUIData>	jobIdToData()
int	numCompletedStages()
int	numFailedStages()
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updated
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.
void	onJobEnd(SparkListenerJobEnd jobEnd) Called when a job ends
void	onJobStart(SparkListenerJobStart jobStart) Called when a job starts
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart taskStart) Called when a task starts
scala.collection.mutable.HashMap<Object,StageInfo>	pendingStages()
scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>>	poolToActiveStages()
int	retainedJobs()
int	retainedStages()
scala.Option<scala.Enumeration.Value>	schedulingMode()
scala.collection.mutable.ListBuffer<StageInfo>	skippedStages()
scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashSet<Object>>	stageIdToActiveJobIds()
scala.collection.mutable.HashMap<scala.Tuple2<Object,Object>,UIData.StageUIData>	stageIdToData()
scala.collection.mutable.HashMap<Object,StageInfo>	stageIdToInfo()
void	updateAggregateMetrics(UIData.StageUIData stageData, String execId, org.apache.spark.executor.TaskMetrics taskMetrics, scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics) Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onExecutorAdded, onExecutorRemoved, onUnpersistRDD

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JobProgressListener

public JobProgressListener(SparkConf conf)

Method Detail

DEFAULT_POOL_NAME

public static String DEFAULT_POOL_NAME()

DEFAULT_RETAINED_STAGES

public static int DEFAULT_RETAINED_STAGES()

DEFAULT_RETAINED_JOBS

public static int DEFAULT_RETAINED_JOBS()

activeJobs

public scala.collection.mutable.HashMap<Object,UIData.JobUIData> activeJobs()

completedJobs

public scala.collection.mutable.ListBuffer<UIData.JobUIData> completedJobs()

failedJobs

public scala.collection.mutable.ListBuffer<UIData.JobUIData> failedJobs()

jobIdToData

public scala.collection.mutable.HashMap<Object,UIData.JobUIData> jobIdToData()

pendingStages

public scala.collection.mutable.HashMap<Object,StageInfo> pendingStages()

activeStages

public scala.collection.mutable.HashMap<Object,StageInfo> activeStages()

completedStages

public scala.collection.mutable.ListBuffer<StageInfo> completedStages()

skippedStages

public scala.collection.mutable.ListBuffer<StageInfo> skippedStages()

failedStages

public scala.collection.mutable.ListBuffer<StageInfo> failedStages()

stageIdToData

public scala.collection.mutable.HashMap<scala.Tuple2<Object,Object>,UIData.StageUIData> stageIdToData()

stageIdToInfo

public scala.collection.mutable.HashMap<Object,StageInfo> stageIdToInfo()

stageIdToActiveJobIds

public scala.collection.mutable.HashMap<Object,scala.collection.mutable.HashSet<Object>> stageIdToActiveJobIds()

poolToActiveStages

public scala.collection.mutable.HashMap<String,scala.collection.mutable.HashMap<Object,StageInfo>> poolToActiveStages()

numCompletedStages

public int numCompletedStages()

numFailedStages

public int numFailedStages()

executorIdToBlockManagerId

public scala.collection.mutable.HashMap<String,BlockManagerId> executorIdToBlockManagerId()

blockManagerIds

public scala.collection.Seq<BlockManagerId> blockManagerIds()

schedulingMode

public scala.Option<scala.Enumeration.Value> schedulingMode()

retainedStages

public int retainedStages()

retainedJobs

public int retainedJobs()

getSizesOfActiveStateTrackingCollections

public scala.collection.immutable.Map<String,Object> getSizesOfActiveStateTrackingCollections()

getSizesOfHardSizeLimitedCollections

public scala.collection.immutable.Map<String,Object> getSizesOfHardSizeLimitedCollections()

getSizesOfSoftSizeLimitedCollections

public scala.collection.immutable.Map<String,Object> getSizesOfSoftSizeLimitedCollections()

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

Description copied from interface: SparkListener

Called when a job starts

Specified by:

onJobStart in interface SparkListener

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

Description copied from interface: SparkListener

Called when a job ends

Specified by:

onJobEnd in interface SparkListener

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Description copied from interface: SparkListener

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListener

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

For FIFO, all stages are contained by "default" pool but "default" pool here is meaningless

Specified by:

onStageSubmitted in interface SparkListener

onTaskStart

public void onTaskStart(SparkListenerTaskStart taskStart)

Description copied from interface: SparkListener

Called when a task starts

Specified by:

onTaskStart in interface SparkListener

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)

Description copied from interface: SparkListener

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListener

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

updateAggregateMetrics

public void updateAggregateMetrics(UIData.StageUIData stageData,
                          String execId,
                          org.apache.spark.executor.TaskMetrics taskMetrics,
                          scala.Option<org.apache.spark.executor.TaskMetrics> oldMetrics)

Upon receiving new metrics for a task, updates the per-stage and per-executor-per-stage aggregate metrics by calculating deltas between the currently recorded metrics and the new metrics.

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate)

Description copied from interface: SparkListener

Called when the driver receives task metrics from an executor in a heartbeat.

Specified by:

onExecutorMetricsUpdate in interface SparkListener

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)

Description copied from interface: SparkListener

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListener

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener