org.apache.spark.scheduler

Class EventLoggingListener

Object

org.apache.spark.scheduler.EventLoggingListener

All Implemented Interfaces:

Logging, SparkListener

public class EventLoggingListener
extends Object
implements SparkListener, Logging

A SparkListener that logs events to persistent storage.

Event logging is specified by the following configurable parameters: spark.eventLog.enabled - Whether event logging is enabled. spark.eventLog.compress - Whether to compress logged events spark.eventLog.overwrite - Whether to overwrite any existing files. spark.eventLog.dir - Path to the directory in which events are logged. spark.eventLog.buffer.kb - Buffer size to use when writing to output streams

Constructor Summary

Constructors

Constructor and Description
EventLoggingListener(String appId, String logBaseDir, SparkConf sparkConf)
EventLoggingListener(String appId, String logBaseDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Methods

Modifier and Type	Method and Description
static String	COMPRESSION_CODEC_KEY()
static String	DEFAULT_LOG_DIR()
static String	getLogPath(String logBaseDir, String appId, scala.Option<String> compressionCodecName) Return a file-system-safe path to the log file for the given application.
static String	IN_PROGRESS()
static void	initEventLog(java.io.OutputStream logStream) Write metadata about an event log to the given stream.
scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue>	loggedEvents()
String	logPath()
void	onApplicationEnd(SparkListenerApplicationEnd event) Called when the application ends
void	onApplicationStart(SparkListenerApplicationStart event) Called when the application starts
void	onBlockManagerAdded(SparkListenerBlockManagerAdded event) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved event) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate event) Called when environment properties have been updated
void	onExecutorAdded(SparkListenerExecutorAdded event) Called when the driver registers a new executor.
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event) Called when the driver receives task metrics from an executor in a heartbeat.
void	onExecutorRemoved(SparkListenerExecutorRemoved event) Called when the driver removes an executor.
void	onJobEnd(SparkListenerJobEnd event) Called when a job ends
void	onJobStart(SparkListenerJobStart event) Called when a job starts
void	onStageCompleted(SparkListenerStageCompleted event) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted event) Called when a stage is submitted
void	onTaskEnd(SparkListenerTaskEnd event) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult event) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart event) Called when a task starts
void	onUnpersistRDD(SparkListenerUnpersistRDD event) Called when an RDD is manually unpersisted by the application
static java.io.InputStream	openEventLog(org.apache.hadoop.fs.Path log, org.apache.hadoop.fs.FileSystem fs) Opens an event log file and returns an input stream that contains the event data.
static String	SPARK_VERSION_KEY()
void	start() Creates the log file in the configured log directory.
boolean	stop() Stop logging events.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

EventLoggingListener

public EventLoggingListener(String appId,
                    String logBaseDir,
                    SparkConf sparkConf,
                    org.apache.hadoop.conf.Configuration hadoopConf)

EventLoggingListener

public EventLoggingListener(String appId,
                    String logBaseDir,
                    SparkConf sparkConf)

Method Detail

IN_PROGRESS

public static String IN_PROGRESS()

DEFAULT_LOG_DIR

public static String DEFAULT_LOG_DIR()

SPARK_VERSION_KEY

public static String SPARK_VERSION_KEY()

COMPRESSION_CODEC_KEY

public static String COMPRESSION_CODEC_KEY()

initEventLog

public static void initEventLog(java.io.OutputStream logStream)

Write metadata about an event log to the given stream. The metadata is encoded in the first line of the event log as JSON.

Parameters:

logStream - Raw output stream to the event log file.

getLogPath

public static String getLogPath(String logBaseDir,
                String appId,
                scala.Option<String> compressionCodecName)

Return a file-system-safe path to the log file for the given application.

Note that because we currently only create a single log file for each application, we must encode all the information needed to parse this event log in the file name instead of within the file itself. Otherwise, if the file is compressed, for instance, we won't know which codec to use to decompress the metadata needed to open the file in the first place.

Parameters:

logBaseDir - Directory where the log file will be written.

appId - A unique app ID.

compressionCodecName - Name to identify the codec used to compress the contents of the log, or None if compression is not enabled.

Returns:

A path which consists of file-system-safe characters.

openEventLog

public static java.io.InputStream openEventLog(org.apache.hadoop.fs.Path log,
                               org.apache.hadoop.fs.FileSystem fs)

Opens an event log file and returns an input stream that contains the event data.

Returns:

input stream that holds one JSON record per line.

loggedEvents

public scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue> loggedEvents()

logPath

public String logPath()

start

public void start()

Creates the log file in the configured log directory.

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted event)

Description copied from interface: SparkListener

Called when a stage is submitted

Specified by:

onStageSubmitted in interface SparkListener

onTaskStart

public void onTaskStart(SparkListenerTaskStart event)

Description copied from interface: SparkListener

Called when a task starts

Specified by:

onTaskStart in interface SparkListener

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult event)

Description copied from interface: SparkListener

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListener

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd event)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate event)

Description copied from interface: SparkListener

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListener

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted event)

Description copied from interface: SparkListener

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListener

onJobStart

public void onJobStart(SparkListenerJobStart event)

Description copied from interface: SparkListener

Called when a job starts

Specified by:

onJobStart in interface SparkListener

onJobEnd

public void onJobEnd(SparkListenerJobEnd event)

Description copied from interface: SparkListener

Called when a job ends

Specified by:

onJobEnd in interface SparkListener

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded event)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved event)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener

onUnpersistRDD

public void onUnpersistRDD(SparkListenerUnpersistRDD event)

Description copied from interface: SparkListener

Called when an RDD is manually unpersisted by the application

Specified by:

onUnpersistRDD in interface SparkListener

onApplicationStart

public void onApplicationStart(SparkListenerApplicationStart event)

Description copied from interface: SparkListener

Called when the application starts

Specified by:

onApplicationStart in interface SparkListener

onApplicationEnd

public void onApplicationEnd(SparkListenerApplicationEnd event)

Description copied from interface: SparkListener

Called when the application ends

Specified by:

onApplicationEnd in interface SparkListener

onExecutorAdded

public void onExecutorAdded(SparkListenerExecutorAdded event)

Description copied from interface: SparkListener

Called when the driver registers a new executor.

Specified by:

onExecutorAdded in interface SparkListener

onExecutorRemoved

public void onExecutorRemoved(SparkListenerExecutorRemoved event)

Description copied from interface: SparkListener

Called when the driver removes an executor.

Specified by:

onExecutorRemoved in interface SparkListener

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event)

Description copied from interface: SparkListener

Called when the driver receives task metrics from an executor in a heartbeat.

Specified by:

onExecutorMetricsUpdate in interface SparkListener

stop

public boolean stop()

Stop logging events. The event log file will be renamed so that it loses the ".inprogress" suffix.