Class PartitionPredicate
- All Implemented Interfaces:
Serializable,Expression
Table.partitioning().
Connectors are expected to leverage partition predicates for pruning whenever they have
partition metadata to evaluate them. Use eval(InternalRow) to evaluate this
predicate against a single partition's keys.
- Since:
- 4.2.0
- See Also:
-
Field Summary
FieldsFields inherited from interface org.apache.spark.sql.connector.expressions.Expression
EMPTY_EXPRESSION, EMPTY_NAMED_REFERENCE -
Method Summary
Modifier and TypeMethodDescriptionabstract booleaneval(org.apache.spark.sql.catalyst.InternalRow partitionKey) Evaluates this predicate against a single partition's keys.abstract NamedReference[]List of fields or columns that are referenced by this expression.Methods inherited from class org.apache.spark.sql.connector.expressions.GeneralScalarExpression
children, equals, hashCode, nameMethods inherited from class org.apache.spark.sql.internal.connector.ExpressionWithToString
describe, toString
-
Field Details
-
NAME
- See Also:
-
-
Method Details
-
references
List of fields or columns that are referenced by this expression.For PartitionPredicate, returns
PartitionFieldReferenceinstances that identify the partition fields (fromTable.partitioning()) referenced by this predicate. Each reference'sNamedReference.fieldNames()gives the partition field name;PartitionFieldReference.ordinal()gives the 0-based position inTable.partitioning().Example: Suppose
Table.partitioning()returns three partition transforms:[years(ts), months(ts), bucket(32, id)]with ordinals 0, 1, 2. EachPartitionFieldReferencehasNamedReference.fieldNames()(the transform display name, e.g.years(ts)) andPartitionFieldReference.ordinal():years(ts) = 2026returns one reference: (fieldNames=[years(ts)], ordinal=0).years(ts) = 2026 and months(ts) = 01returns two references: (fieldNames=[years(ts)], ordinal=0), (fieldNames=[months(ts)], ordinal=1).bucket(32, id) = 1returns one reference: (fieldNames=[bucket(32, id)], ordinal=2).
Data sources can use these references to decide whether to return a predicate for post-scan filtering. For example, sources supporting partition spec evolution should return PartitionPredicates that reference later-added partition transforms (incompletely partitioned data) to Spark for post-scan filter, while predicates that reference only initially-added partition transforms may be fully pushed.
- Specified by:
referencesin interfaceExpression- Overrides:
referencesin classorg.apache.spark.sql.internal.connector.ExpressionWithToString- Returns:
- array of partition field references
-
eval
public abstract boolean eval(org.apache.spark.sql.catalyst.InternalRow partitionKey) Evaluates this predicate against a single partition's keys.The caller must pass the full partition key: one value per partition transform in
Table.partitioning(), in order. A key for only a subset of referenced fields is not supported.This method may throw when it cannot evaluate the predicate for the given key, for example on an ANSI cast failure. Accepting a predicate lets Spark drop the filter it pushed, which leaves this predicate as the only evaluator, so a caller must let such a failure propagate rather than treat the partition as matching, and must finish evaluating before it mutates any data.
- Parameters:
partitionKey- the full partition key for one partition, ordered according toTable.partitioning().- Returns:
- true if the partition represented by these keys satisfies this predicate.
-