1. [SPARK-24879][SQL] Fix NPE in Hive partition pruning filter pushdown (commit: bd6bfacb2ef473b858cc91e0692591deaea26118) (details)
Commit bd6bfacb2ef473b858cc91e0692591deaea26118 by gatorsmile
[SPARK-24879][SQL] Fix NPE in Hive partition pruning filter pushdown
## What changes were proposed in this pull request? We get a NPE when we
have a filter on a partition column of the form `col in (x, null)`. This
is due to the filter converter in HiveShim not handling `null`s
correctly. This patch fixes this bug while still pushing down as much of
the partition pruning predicates as possible, by filtering out `null`s
from any `in` predicate. Since Hive only supports very simple partition
pruning filters, this change should preserve correctness.
## How was this patch tested? Unit tests, manual tests
Author: William Sheu <>
Closes #21832 from PenguinToast/partition-pruning-npe.
(cherry picked from commit bbd6f0c25fe19dc6c946e63cac7b98d0f78b3463)
Signed-off-by: Xiao Li <>
(commit: bd6bfacb2ef473b858cc91e0692591deaea26118)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/client/FiltersSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala (diff)