org.apache.spark.scheduler

Class EventLoggingListener

Object

org.apache.spark.scheduler.EventLoggingListener

All Implemented Interfaces:

Logging, SparkListener

public class EventLoggingListener
extends Object
implements SparkListener, Logging

A SparkListener that logs events to persistent storage.

Event logging is specified by the following configurable parameters: spark.eventLog.enabled - Whether event logging is enabled. spark.eventLog.compress - Whether to compress logged events spark.eventLog.overwrite - Whether to overwrite any existing files. spark.eventLog.dir - Path to the directory in which events are logged. spark.eventLog.buffer.kb - Buffer size to use when writing to output streams

Constructor Summary

Constructors

Constructor and Description
EventLoggingListener(String appId, String logBaseDir, SparkConf sparkConf)
EventLoggingListener(String appId, String logBaseDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Methods

Modifier and Type	Method and Description
static String	APPLICATION_COMPLETE()
static String	COMPRESSION_CODEC_PREFIX()
static String	DEFAULT_LOG_DIR()
static String	getLogDirPath(String logBaseDir, String appId) Return a file-system-safe path to the log directory for the given application.
static boolean	isApplicationCompleteFile(String fileName)
static boolean	isCompressionCodecFile(String fileName)
static boolean	isEventLogFile(String fileName)
static boolean	isSparkVersionFile(String fileName)
static org.apache.hadoop.fs.permission.FsPermission	LOG_FILE_PERMISSIONS()
static String	LOG_PREFIX()
String	logDir()
String	logDirName()
scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue>	loggedEvents()
void	onApplicationEnd(SparkListenerApplicationEnd event) Called when the application ends
void	onApplicationStart(SparkListenerApplicationStart event) Called when the application starts
void	onBlockManagerAdded(SparkListenerBlockManagerAdded event) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved event) Called when an existing block manager has been removed
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate event) Called when environment properties have been updated
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event) Called when the driver receives task metrics from an executor in a heartbeat.
void	onJobEnd(SparkListenerJobEnd event) Called when a job ends
void	onJobStart(SparkListenerJobStart event) Called when a job starts
void	onStageCompleted(SparkListenerStageCompleted event) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageSubmitted(SparkListenerStageSubmitted event) Called when a stage is submitted
void	onTaskEnd(SparkListenerTaskEnd event) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult event) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart event) Called when a task starts
void	onUnpersistRDD(SparkListenerUnpersistRDD event) Called when an RDD is manually unpersisted by the application
static String	parseCompressionCodec(String fileName)
static EventLoggingInfo	parseLoggingInfo(org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.FileSystem fileSystem) Parse the event logging information associated with the logs in the given directory.
static EventLoggingInfo	parseLoggingInfo(String logDir, org.apache.hadoop.fs.FileSystem fileSystem) Parse the event logging information associated with the logs in the given directory.
static String	parseSparkVersion(String fileName)
static String	SPARK_VERSION_PREFIX()
void	start() Begin logging events.
void	stop() Stop logging events.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

EventLoggingListener

public EventLoggingListener(String appId,
                    String logBaseDir,
                    SparkConf sparkConf,
                    org.apache.hadoop.conf.Configuration hadoopConf)

EventLoggingListener

public EventLoggingListener(String appId,
                    String logBaseDir,
                    SparkConf sparkConf)

Method Detail

DEFAULT_LOG_DIR

public static String DEFAULT_LOG_DIR()

LOG_PREFIX

public static String LOG_PREFIX()

SPARK_VERSION_PREFIX

public static String SPARK_VERSION_PREFIX()

COMPRESSION_CODEC_PREFIX

public static String COMPRESSION_CODEC_PREFIX()

APPLICATION_COMPLETE

public static String APPLICATION_COMPLETE()

LOG_FILE_PERMISSIONS

public static org.apache.hadoop.fs.permission.FsPermission LOG_FILE_PERMISSIONS()

isEventLogFile

public static boolean isEventLogFile(String fileName)

isSparkVersionFile

public static boolean isSparkVersionFile(String fileName)

isCompressionCodecFile

public static boolean isCompressionCodecFile(String fileName)

isApplicationCompleteFile

public static boolean isApplicationCompleteFile(String fileName)

parseSparkVersion

public static String parseSparkVersion(String fileName)

parseCompressionCodec

public static String parseCompressionCodec(String fileName)

getLogDirPath

public static String getLogDirPath(String logBaseDir,
                   String appId)

Return a file-system-safe path to the log directory for the given application.

Parameters:

logBaseDir - A base directory for the path to the log directory for given application.

appId - A unique app ID.

Returns:

A path which consists of file-system-safe characters.

parseLoggingInfo

public static EventLoggingInfo parseLoggingInfo(org.apache.hadoop.fs.Path logDir,
                                org.apache.hadoop.fs.FileSystem fileSystem)

Parse the event logging information associated with the logs in the given directory.

Specifically, this looks for event log files, the Spark version file, the compression codec file (if event logs are compressed), and the application completion file (if the application has run to completion).

parseLoggingInfo

public static EventLoggingInfo parseLoggingInfo(String logDir,
                                org.apache.hadoop.fs.FileSystem fileSystem)

Parse the event logging information associated with the logs in the given directory.

logDir

public String logDir()

logDirName

public String logDirName()

loggedEvents

public scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue> loggedEvents()

start

public void start()

Begin logging events. If compression is used, log a file that indicates which compression library is used.

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted event)

Description copied from interface: SparkListener

Called when a stage is submitted

Specified by:

onStageSubmitted in interface SparkListener

onTaskStart

public void onTaskStart(SparkListenerTaskStart event)

Description copied from interface: SparkListener

Called when a task starts

Specified by:

onTaskStart in interface SparkListener

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult event)

Description copied from interface: SparkListener

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListener

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd event)

Description copied from interface: SparkListener

Called when a task ends

Specified by:

onTaskEnd in interface SparkListener

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate event)

Description copied from interface: SparkListener

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListener

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted event)

Description copied from interface: SparkListener

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListener

onJobStart

public void onJobStart(SparkListenerJobStart event)

Description copied from interface: SparkListener

Called when a job starts

Specified by:

onJobStart in interface SparkListener

onJobEnd

public void onJobEnd(SparkListenerJobEnd event)

Description copied from interface: SparkListener

Called when a job ends

Specified by:

onJobEnd in interface SparkListener

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded event)

Description copied from interface: SparkListener

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListener

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved event)

Description copied from interface: SparkListener

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListener

onUnpersistRDD

public void onUnpersistRDD(SparkListenerUnpersistRDD event)

Description copied from interface: SparkListener

Called when an RDD is manually unpersisted by the application

Specified by:

onUnpersistRDD in interface SparkListener

onApplicationStart

public void onApplicationStart(SparkListenerApplicationStart event)

Description copied from interface: SparkListener

Called when the application starts

Specified by:

onApplicationStart in interface SparkListener

onApplicationEnd

public void onApplicationEnd(SparkListenerApplicationEnd event)

Description copied from interface: SparkListener

Called when the application ends

Specified by:

onApplicationEnd in interface SparkListener

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event)

Description copied from interface: SparkListener

Called when the driver receives task metrics from an executor in a heartbeat.

Specified by:

onExecutorMetricsUpdate in interface SparkListener

stop

public void stop()

Stop logging events. In addition, create an empty special file to indicate application completion.