org.apache.spark.mllib.impl

Class PeriodicGraphCheckpointer<VD,ED>

Object

org.apache.spark.mllib.impl.PeriodicGraphCheckpointer<VD,ED>

All Implemented Interfaces:

Logging

public class PeriodicGraphCheckpointer<VD,ED>
extends Object
implements Logging

This class helps with persisting and checkpointing Graphs. Specifically, it automatically handles persisting and (optionally) checkpointing, as well as unpersisting and removing checkpoint files.

Users should call PeriodicGraphCheckpointer.updateGraph() when a new graph has been created, before the graph has been materialized. After updating PeriodicGraphCheckpointer, users are responsible for materializing the graph to ensure that persisting and checkpointing actually occur.

When PeriodicGraphCheckpointer.updateGraph() is called, this does the following: - Persist new graph (if not yet persisted), and put in queue of persisted graphs. - Unpersist graphs from queue until there are at most 3 persisted graphs. - If using checkpointing and the checkpoint interval has been reached, - Checkpoint the new graph, and put in a queue of checkpointed graphs. - Remove older checkpoints.

WARNINGS: - This class should NOT be copied (since copies may conflict on which Graphs should be checkpointed). - This class removes checkpoint files once later graphs have been checkpointed. However, references to the older graphs will still return isCheckpointed = true.

Example usage:

  val (graph1, graph2, graph3, ...) = ...
  val cp = new PeriodicGraphCheckpointer(graph1, dir, 2)
  graph1.vertices.count(); graph1.edges.count()
  // persisted: graph1
  cp.updateGraph(graph2)
  graph2.vertices.count(); graph2.edges.count()
  // persisted: graph1, graph2
  // checkpointed: graph2
  cp.updateGraph(graph3)
  graph3.vertices.count(); graph3.edges.count()
  // persisted: graph1, graph2, graph3
  // checkpointed: graph2
  cp.updateGraph(graph4)
  graph4.vertices.count(); graph4.edges.count()
  // persisted: graph2, graph3, graph4
  // checkpointed: graph4
  cp.updateGraph(graph5)
  graph5.vertices.count(); graph5.edges.count()
  // persisted: graph3, graph4, graph5
  // checkpointed: graph4
 

Constructor Summary

Constructors

Constructor and Description
PeriodicGraphCheckpointer(Graph<VD,ED> currentGraph, int checkpointInterval)

Method Summary

Methods

Modifier and Type	Method and Description
int	checkpointInterval()
Graph<VD,ED>	currentGraph()
void	deleteAllCheckpoints() Call this at the end to delete any remaining checkpoint files.
void	updateGraph(Graph<VD,ED> newGraph) Update currentGraph with a new graph.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

PeriodicGraphCheckpointer

public PeriodicGraphCheckpointer(Graph<VD,ED> currentGraph,
                         int checkpointInterval)

Method Detail

currentGraph

public Graph<VD,ED> currentGraph()

checkpointInterval

public int checkpointInterval()

updateGraph

public void updateGraph(Graph<VD,ED> newGraph)

Update currentGraph with a new graph. Handle persistence and checkpointing as needed. Since this handles persistence and checkpointing, this should be called before the graph has been materialized.

Parameters:

newGraph - New graph created from previous graphs in the lineage.

deleteAllCheckpoints

public void deleteAllCheckpoints()

Call this at the end to delete any remaining checkpoint files.