org.apache.spark.storage

Class StorageStatusListener

Object

org.apache.spark.storage.StorageStatusListener

All Implemented Interfaces:

SparkListener

public class StorageStatusListener
extends Object
implements SparkListener

:: DeveloperApi :: A SparkListener that maintains executor storage status.

Constructor Summary

Constructors

Constructor and Description
StorageStatusListener()

Method Summary

Methods

Modifier and Type	Method and Description
String	formatExecutorId(String execId) In the local mode, there is a discrepancy between the executor ID according to the task ("localhost") and that according to SparkEnv ("").
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD) Called when an RDD is manually unpersisted by the application
scala.collection.Seq<StorageStatus>	storageStatusList()
void	updateStorageStatus(int unpersistedRDDId) Update storage status list to reflect the removal of an RDD from the cache
void	updateStorageStatus(String execId, scala.collection.Seq<scala.Tuple2<BlockId,BlockStatus>> updatedBlocks) Update storage status list to reflect updated block statuses

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onEnvironmentUpdate, onJobEnd, onJobStart, onStageCompleted, onStageSubmitted, onTaskGettingResult, onTaskStart

Constructor Detail

StorageStatusListener

public StorageStatusListener()

Method Detail

storageStatusList

public scala.collection.Seq<StorageStatus> storageStatusList()

updateStorageStatus

public void updateStorageStatus(String execId,
                       scala.collection.Seq<scala.Tuple2<BlockId,BlockStatus>> updatedBlocks)

Update storage status list to reflect updated block statuses

updateStorageStatus

public void updateStorageStatus(int unpersistedRDDId)

Update storage status list to reflect the removal of an RDD from the cache

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

onUnpersistRDD

public void onUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD)

Description copied from interface: SparkListener

Called when an RDD is manually unpersisted by the application

Specified by:

onUnpersistRDD in interface SparkListener

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener

formatExecutorId

public String formatExecutorId(String execId)

In the local mode, there is a discrepancy between the executor ID according to the task ("localhost") and that according to SparkEnv (""). In the UI, this results in duplicate rows for the same executor. Thus, in this mode, we aggregate these two rows and use the executor ID of "" to be consistent.