pyspark.sql.functions.corr#

pyspark.sql.functions.corr(col1, col2)[source]#

Returns a new Column for the Pearson Correlation Coefficient for col1 and col2.

Added in version 1.6.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters:
col1Column or column name

first column to calculate correlation. A column that evaluates to a numeric.

col2Column or column name

second column to calculate correlation. A column that evaluates to a numeric.

Returns:
Column

Pearson Correlation Coefficient of these two column values.

Examples

>>> from pyspark.sql import functions as sf
>>> a = range(20)
>>> b = [2 * x for x in range(20)]
>>> df = spark.createDataFrame(zip(a, b), ["a", "b"])
>>> df.agg(sf.corr("a", df.b)).show()
+----------+
|corr(a, b)|
+----------+
|       1.0|
+----------+