Package org.apache.spark.sql.pipelines.graph

Class CoreDataflowNodeProcessor

Object

org.apache.spark.sql.pipelines.graph.CoreDataflowNodeProcessor

public class CoreDataflowNodeProcessor
extends Object

Processor that is responsible for analyzing each flow and sort the nodes in topological order

Constructor Summary

Constructors

Constructor	Description
CoreDataflowNodeProcessor(DataflowGraph rawGraph)

Method Summary

Modifier and Type	Method	Description
scala.collection.immutable.Seq<GraphElement>	processNode(GraphElement node, scala.collection.immutable.Seq<GraphElement> upstreamNodes)	Processes the node of the graph, re-arranging them if they are not topologically sorted.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CoreDataflowNodeProcessor

public CoreDataflowNodeProcessor(DataflowGraph rawGraph)

Method Details

processNode

public scala.collection.immutable.Seq<GraphElement> processNode(GraphElement node,
 scala.collection.immutable.Seq<GraphElement> upstreamNodes)

Processes the node of the graph, re-arranging them if they are not topologically sorted. Takes care of resolving the flows and virtualizing tables (i.e. removing tables to ensure resolution is internally consistent) if needed for the nodes.

Parameters:

node - The node to process

upstreamNodes - Upstream nodes for the node

Returns:

The resolved nodes generated by processing this element.