SuccessChanges

Summary

  1. [SPARK-32344][SQL][2.4] Unevaluable expr is set to FIRST/LAST (details)
Commit 58c637a213e122156b914c27d51e5eb1ec4e0b4c by dongjoon
[SPARK-32344][SQL][2.4] Unevaluable expr is set to FIRST/LAST
ignoreNullsExpr in distinct aggregates
### What changes were proposed in this pull request?
This PR intends to fix a bug of distinct FIRST/LAST aggregates in
v2.4.6;
``` scala> sql("SELECT FIRST(DISTINCT v) FROM VALUES 1, 2, 3
t(v)").show()
... Caused by: java.lang.UnsupportedOperationException: Cannot evaluate
expression: false#37
at
org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:258)
at
org.apache.spark.sql.catalyst.expressions.AttributeReference.eval(namedExpressions.scala:226)
at
org.apache.spark.sql.catalyst.expressions.aggregate.First.ignoreNulls(First.scala:68)
at
org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions$lzycompute(First.scala:82)
at
org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions(First.scala:81)
at
org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$15.apply(HashAggregateExec.scala:268)
``` A root cause of this bug is that the `Aggregation` strategy replaces
a foldable boolean `ignoreNullsExpr` expr with a `Unevaluable` expr
(`AttributeReference`) for distinct FIRST/LAST aggregate functions. But,
this operation cannot be allowed because the `Analyzer` has checked that
it must be foldabe;
https://github.com/apache/spark/blob/ffdbbae1d465fe2c710d020de62ca1a6b0b924d9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala#L74-L76
So, this PR proposes to change a vriable for `IGNORE NULLS`  from
`Expression` to `Boolean` to avoid the case.
This is the backport of https://github.com/apache/spark/pull/29143.
### Why are the changes needed?
Bugfix.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added a test in `DataFrameAggregateSuite`.
Closes #29157 from maropu/SPARK-32344-BRANCH2.4.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Dongjoon Hyun <dongjoon@apache.org>
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/functions.scala (diff)
The file was addedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/FirstLastTestSuite.scala
The file was removedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/LastTestSuite.scala
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala (diff)