FailedChanges

Summary

  1. [SPARK-25368][SQL] Incorrect predicate pushdown returns wrong result (details)
Commit 5ad644a4cefc20e4f198d614c59b8b0f75a228ba by gatorsmile
[SPARK-25368][SQL] Incorrect predicate pushdown returns wrong result
How to reproduce:
```scala val df1 = spark.createDataFrame(Seq(
  (1, 1)
)).toDF("a", "b").withColumn("c", lit(null).cast("int")) val df2 =
df1.union(df1).withColumn("d",
spark_partition_id).filter($"c".isNotNull) df2.show
+---+---+----+---+
|  a|  b|   c|  d|
+---+---+----+---+
|  1|  1|null|  0|
|  1|  1|null|  1|
+---+---+----+---+
```
`filter($"c".isNotNull)` was transformed to `(null <=> c#10)` before
https://github.com/apache/spark/pull/19201, but it is transformed to
`(c#10 = null)` since https://github.com/apache/spark/pull/20155. This
pr revert it to `(null <=> c#10)` to fix this issue.
unit tests
Closes #22368 from wangyum/SPARK-25368.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: gatorsmile
<gatorsmile@gmail.com>
(cherry picked from commit 77c996403d5c761f0dfea64c5b1cb7480ba1d3ac)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala (diff)