org.apache.spark.mllib.tree.impl

Class NodeIdCache

Object

org.apache.spark.mllib.tree.impl.NodeIdCache

public class NodeIdCache
extends Object

:: DeveloperApi :: A given TreePoint would belong to a particular node per tree. Each row in the nodeIdsForInstances RDD is an array over trees of the node index in each tree. Initially, values should all be 1 for root node. The nodeIdsForInstances RDD needs to be updated at each iteration.

Constructor Summary

Constructors

Constructor and Description
NodeIdCache(RDD<int[]> nodeIdsForInstances, scala.Option<String> checkpointDir, int checkpointInterval)

Method Summary

Methods

Modifier and Type	Method and Description
scala.Option<String>	checkpointDir()
int	checkpointInterval()
void	deleteAllCheckpoints() Call this after training is finished to delete any remaining checkpoints.
static NodeIdCache	init(RDD<BaggedPoint<TreePoint>> data, int numTrees, scala.Option<String> checkpointDir, int checkpointInterval, int initVal) Initialize the node Id cache with initial node Id values.
RDD<int[]>	nodeIdsForInstances()
void	updateNodeIndices(RDD<BaggedPoint<TreePoint>> data, scala.collection.mutable.Map<Object,NodeIndexUpdater>[] nodeIdUpdaters, Bin[][] bins) Update the node index values in the cache.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NodeIdCache

public NodeIdCache(RDD<int[]> nodeIdsForInstances,
           scala.Option<String> checkpointDir,
           int checkpointInterval)

Method Detail

init

public static NodeIdCache init(RDD<BaggedPoint<TreePoint>> data,
               int numTrees,
               scala.Option<String> checkpointDir,
               int checkpointInterval,
               int initVal)

Initialize the node Id cache with initial node Id values.

Parameters:

data - The RDD of training rows.

numTrees - The number of trees that we want to create cache for.

checkpointDir - The checkpoint directory where the checkpointed files will be stored.

checkpointInterval - The checkpointing interval (how often should the cache be checkpointed.).

initVal - The initial values in the cache.

Returns:

A node Id cache containing an RDD of initial root node Indices.

nodeIdsForInstances

public RDD<int[]> nodeIdsForInstances()

checkpointDir

public scala.Option<String> checkpointDir()

checkpointInterval

public int checkpointInterval()

updateNodeIndices

public void updateNodeIndices(RDD<BaggedPoint<TreePoint>> data,
                     scala.collection.mutable.Map<Object,NodeIndexUpdater>[] nodeIdUpdaters,
                     Bin[][] bins)

Update the node index values in the cache. This updates the RDD and its lineage. TODO: Passing bin information to executors seems unnecessary and costly.

Parameters:

data - The RDD of training rows.

nodeIdUpdaters - A map of node index updaters. The key is the indices of nodes that we want to update.

bins - Bin information needed to find child node indices.

deleteAllCheckpoints

public void deleteAllCheckpoints()

Call this after training is finished to delete any remaining checkpoints.