org.apache.spark.mllib.clustering

Class DistributedLDAModel

Object

org.apache.spark.mllib.clustering.LDAModel

org.apache.spark.mllib.clustering.DistributedLDAModel

public class DistributedLDAModel
extends LDAModel

:: Experimental ::

Distributed LDA model. This model stores the inferred topics, the full training dataset, and the topic distributions. When computing topics for new documents, it may give more accurate answers than the LocalLDAModel.

Constructor Summary

Constructors

Constructor and Description
DistributedLDAModel(LDA.EMOptimizer state, double[] iterationTimes)

Method Summary

Methods

Modifier and Type	Method and Description
scala.Tuple2<int[],double[]>[]	describeTopics(int maxTermsPerTopic) Return the topics described by weighted terms.
double[]	iterationTimes()
int	k() Number of topics
double	logLikelihood() Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)
double	logPrior() Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
LocalLDAModel	toLocal() Convert model to a local model.
RDD<scala.Tuple2<Object,Vector>>	topicDistributions() For each document in the training set, return the distribution over topics for that document ("theta_doc").
Matrix	topicsMatrix() Inferred topics, where each topic is represented by a distribution over terms.
int	vocabSize() Vocabulary size (number of terms or terms in the vocabulary)

Methods inherited from class org.apache.spark.mllib.clustering.LDAModel

describeTopics

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistributedLDAModel

public DistributedLDAModel(LDA.EMOptimizer state,
                   double[] iterationTimes)

Method Detail

k

public int k()

Description copied from class: LDAModel

Number of topics

Specified by:

k in class LDAModel

vocabSize

public int vocabSize()

Description copied from class: LDAModel

Vocabulary size (number of terms or terms in the vocabulary)

Specified by:

vocabSize in class LDAModel

iterationTimes

public double[] iterationTimes()

toLocal

public LocalLDAModel toLocal()

Convert model to a local model. The local model stores the inferred topics but not the topic distributions for training documents.

topicsMatrix

public Matrix topicsMatrix()

Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.

Specified by:

topicsMatrix in class LDAModel

describeTopics

public scala.Tuple2<int[],double[]>[] describeTopics(int maxTermsPerTopic)

Description copied from class: LDAModel

Return the topics described by weighted terms.

This limits the number of terms per topic. This is approximate; it may not return exactly the top-weighted terms for each topic. To get a more precise set of top terms, increase maxTermsPerTopic.

Specified by:

describeTopics in class LDAModel

Parameters:

maxTermsPerTopic - Maximum number of terms to collect for each topic.

Returns:

Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

logLikelihood

public double logLikelihood()

Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)

Note: - This excludes the prior; for that, use logPrior. - Even with logPrior, this is NOT the same as the data log likelihood given the hyperparameters.

logPrior

public double logPrior()

Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)

topicDistributions

public RDD<scala.Tuple2<Object,Vector>> topicDistributions()

For each document in the training set, return the distribution over topics for that document ("theta_doc").

Returns:

RDD of (document ID, topic distribution) pairs