org.apache.spark.mllib.linalg.distributed

Class RowMatrix

Object

org.apache.spark.mllib.linalg.distributed.RowMatrix

All Implemented Interfaces:

java.io.Serializable, Logging, DistributedMatrix

public class RowMatrix
extends Object
implements DistributedMatrix, Logging

:: Experimental :: Represents a row-oriented distributed Matrix with no meaningful row indices.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
RowMatrix(RDD<Vector> rows) Alternative constructor leaving matrix dimensions to be determined automatically.
RowMatrix(RDD<Vector> rows, long nRows, int nCols)

Method Summary

Methods

Modifier and Type	Method and Description
MultivariateStatisticalSummary	computeColumnSummaryStatistics() Computes column-wise summary statistics.
Matrix	computeCovariance() Computes the covariance matrix, treating each row as an observation.
Matrix	computeGramianMatrix() Computes the Gramian matrix A^T A.
Matrix	computePrincipalComponents(int k) Computes the top k principal components.
SingularValueDecomposition<RowMatrix,Matrix>	computeSVD(int k, boolean computeU, double rCond) Computes the singular value decomposition of this matrix.
RowMatrix	multiply(Matrix B) Multiply this matrix by a local matrix on the right.
long	numCols() Gets or computes the number of columns.
long	numRows() Gets or computes the number of rows.
RDD<Vector>	rows()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initialized, initializeIfNecessary, initializeLogging, initLock, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logTrace, logTrace, logWarning, logWarning

Constructor Detail

RowMatrix

public RowMatrix(RDD<Vector> rows,
         long nRows,
         int nCols)

RowMatrix

public RowMatrix(RDD<Vector> rows)

Alternative constructor leaving matrix dimensions to be determined automatically.

Method Detail

rows

public RDD<Vector> rows()

numCols

public long numCols()

Gets or computes the number of columns.

Specified by:

numCols in interface DistributedMatrix

numRows

public long numRows()

Gets or computes the number of rows.

Specified by:

numRows in interface DistributedMatrix

computeGramianMatrix

public Matrix computeGramianMatrix()

Computes the Gramian matrix A^T A.

computeSVD

public SingularValueDecomposition<RowMatrix,Matrix> computeSVD(int k,
                                                      boolean computeU,
                                                      double rCond)

Computes the singular value decomposition of this matrix. Denote this matrix by A (m x n), this will compute matrices U, S, V such that A = U * S * V'.

There is no restriction on m, but we require n^2 doubles to fit in memory. Further, n should be less than m.

The decomposition is computed by first computing A'A = V S^2 V', computing svd locally on that (since n x n is small), from which we recover S and V. Then we compute U via easy matrix multiplication as U = A * (V * S^-1). Note that this approach requires O(n^3) time on the master node.

At most k largest non-zero singular values and associated vectors are returned. If there are k such values, then the dimensions of the return will be:

U is a RowMatrix of size m x k that satisfies U'U = eye(k), s is a Vector of size k, holding the singular values in descending order, and V is a Matrix of size n x k that satisfies V'V = eye(k).

Parameters:

k - number of singular values to keep. We might return less than k if there are numerically zero singular values. See rCond.

computeU - whether to compute U

rCond - the reciprocal condition number. All singular values smaller than rCond * sigma(0) are treated as zero, where sigma(0) is the largest singular value.

Returns:

SingularValueDecomposition(U, s, V)

computeCovariance

public Matrix computeCovariance()

Computes the covariance matrix, treating each row as an observation.

Returns:

a local dense matrix of size n x n

computePrincipalComponents

public Matrix computePrincipalComponents(int k)

Computes the top k principal components. Rows correspond to observations and columns correspond to variables. The principal components are stored a local matrix of size n-by-k. Each column corresponds for one principal component, and the columns are in descending order of component variance.

Parameters:

k - number of top principal components.

Returns:

a matrix of size n-by-k, whose columns are principal components

computeColumnSummaryStatistics

public MultivariateStatisticalSummary computeColumnSummaryStatistics()

Computes column-wise summary statistics.

multiply

public RowMatrix multiply(Matrix B)

Multiply this matrix by a local matrix on the right.

Parameters:

B - a local matrix whose number of rows must match the number of columns of this matrix

Returns:

a RowMatrix representing the product, which preserves partitioning