org.apache.spark.sql.hive.orc

Class OrcFilters

java.lang.Object

org.apache.spark.sql.hive.orc.OrcFilters

public class OrcFilters
extends java.lang.Object

Helper object for building ORC SearchArguments, which are used for ORC predicate push-down.

Due to limitation of ORC SearchArgument builder, we had to end up with a pretty weird double- checking pattern when converting And/Or/Not filters.

An ORC SearchArgument must be built in one pass using a single builder. For example, you can't build a = 1 and b = 2 first, and then combine them into a = 1 AND b = 2. This is quite different from the cases in Spark SQL or Parquet, where complex filters can be easily built using existing simpler ones.

The annoying part is that, SearchArgument builder methods like startAnd(), startOr(), and startNot() mutate internal state of the builder instance. This forces us to translate all convertible filters with a single builder instance. However, before actually converting a filter, we've no idea whether it can be recognized by ORC or not. Thus, when an inconvertible filter is found, we may already end up with a builder whose internal state is inconsistent.

For example, to convert an And filter with builder b, we call b.startAnd() first, and then try to convert its children. Say we convert left child successfully, but find that right child is inconvertible. Alas, b.startAnd() call can't be rolled back, and b is inconsistent now.

The workaround employed here is that, for And/Or/Not, we first try to convert their children with brand new builders, and only do the actual conversion with the right builder instance when the children are proven to be convertible.

P.S.: Hive seems to use SearchArgument together with ExprNodeGenericFuncDesc only. Usage of builder methods mentioned above can only be found in test code, where all tested filters are known to be convertible.

Constructor Summary

Constructors

Constructor and Description
OrcFilters()

Method Summary

Methods

Modifier and Type	Method and Description
static scala.Option<org.apache.hadoop.hive.ql.io.sarg.SearchArgument>	createFilter(StructType schema, Filter[] filters)
protected static void	initializeLogIfNecessary(boolean isInterpreter)
protected static boolean	isTraceEnabled()
protected static org.slf4j.Logger	log()
protected static void	logDebug(scala.Function0<java.lang.String> msg)
protected static void	logDebug(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logError(scala.Function0<java.lang.String> msg)
protected static void	logError(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logInfo(scala.Function0<java.lang.String> msg)
protected static void	logInfo(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static java.lang.String	logName()
protected static void	logTrace(scala.Function0<java.lang.String> msg)
protected static void	logTrace(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logWarning(scala.Function0<java.lang.String> msg)
protected static void	logWarning(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OrcFilters

public OrcFilters()

Method Detail

createFilter

public static scala.Option<org.apache.hadoop.hive.ql.io.sarg.SearchArgument> createFilter(StructType schema,
                                                                          Filter[] filters)

logName

protected static java.lang.String logName()

log

protected static org.slf4j.Logger log()

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg)

logError

protected static void logError(scala.Function0<java.lang.String> msg)

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg,
           java.lang.Throwable throwable)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg,
              java.lang.Throwable throwable)

logError

protected static void logError(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

isTraceEnabled

protected static boolean isTraceEnabled()

initializeLogIfNecessary

protected static void initializeLogIfNecessary(boolean isInterpreter)