FailedChanges

Summary

  1. [SPARK-30407][SQL] fix the reset metric issue when enable AQE (details)
  2. [SPARK-30403][SQL] fix the NoSuchElementException when enable AQE with (details)
  3. [SPARK-30284][SQL] CREATE VIEW should keep the current catalog and (details)
  4. revert [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax (details)
  5. [SPARK-30341][SQL] Overflow check for interval arithmetic operations (details)
Commit 05f7b57ddc930b8bc4ab4628e3790a6c680998f4 by wenchen
[SPARK-30407][SQL] fix the reset metric issue when enable AQE
### What changes were proposed in this pull request? When working on
[PR#26813](https://github.com/apache/spark/pull/26813), we encounter the
exception in [here(the number of metrics(1) is 2 not 1
)](https://github.com/apache/spark/blob/5d870ef0bc70527fd1bc99a4ad17e4941c923351/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala#L120).
This PR fix the above exception.
### Why are the changes needed? Fix exception
### Does this PR introduce any user-facing change? No
### How was this patch tested?
[this test with enable
AQE](https://github.com/apache/spark/blob/5d870ef0bc70527fd1bc99a4ad17e4941c923351/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala#L120)
Closes #27074 from JkSelf/resetMetricsIssue.
Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala (diff)
Commit 6bd5494f3470025ac93c3b453f72426915cd7a26 by wenchen
[SPARK-30403][SQL] fix the NoSuchElementException when enable AQE with
InSubquery expression
### What changes were proposed in this pull request? This PR aim to fix
the NoSuchElementException  exception when enable AQE with insubquery
expression.
### Why are the changes needed? Fix exception
### Does this PR introduce any user-facing change? No
### How was this patch tested? added new ut
Closes #27068 from JkSelf/fixSubqueryIssue.
Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala (diff)
Commit 1743d5be7f74a5a7ab28dbc5c0f431cf2c2dfbc0 by wenchen
[SPARK-30284][SQL] CREATE VIEW should keep the current catalog and
namespace
### What changes were proposed in this pull request?
Update CREATE VIEW command to store the current catalog and namespace
instead of current database in view metadata. Also update analyzer to
leverage the catalog and namespace in view metastore to resolve
relations inside views.
Note that, this PR still keeps the way we resolve views, by recursively
calling Analyzer. This is necessary because view text may contain CTE,
window spec, etc. which needs rules outside of the main resolution batch
(e.g. `CTESubstitution`)
### Why are the changes needed?
To resolve relations inside view correctly.
### Does this PR introduce any user-facing change?
Yes, fix a bug. Now tables referred by a view can be resolved correctly
even if the current catalog/namespace has been updated.
### How was this patch tested?
a new test
Closes #26923 from cloud-fan/view.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/describe.sql.out (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/create_view.sql.out (diff)
Commit 68260f52973de96df5cecefb4ac9a3e40bb707c2 by wenchen
revert [SPARK-29680][SQL] Remove ALTER TABLE CHANGE COLUMN syntax
### What changes were proposed in this pull request?
Revert https://github.com/apache/spark/pull/26338 , as the syntax is
actually the [hive style ALTER
COLUMN](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ChangeColumnName/Type/Position/Comment).
This PR brings it back, and make it support multi-catalog: 1. renaming
is not allowed as `AlterTableAlterColumnStatement` can't do renaming. 2.
column name should be multi-part
### Why are the changes needed?
to not break hive compatibility.
### Does this PR introduce any user-facing change?
no, as the removal was merged in 3.0.
### How was this patch tested?
new parser tests
Closes #27076 from cloud-fan/alter.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
Commit e04309cb1fa719b48c7ad162919a88a516bc8ad1 by wenchen
[SPARK-30341][SQL] Overflow check for interval arithmetic operations
### What changes were proposed in this pull request?
1. For the interval arithmetic functions, e.g.
`add`/`subtract`/`negative`/`multiply`/`divide`, enable overflow check
when `ANSI` is on.
2. For `multiply`/`divide`,  throw an exception when an overflow happens
in spite of `ANSI` is on/off.
3. `add`/`subtract`/`negative` stay the same for backward compatibility.
4. `divide` by 0 throws ArithmeticException whether `ANSI` or not as
same as numerics.
5. These behaviors fit the numeric type operations fully when ANSI is
on.
6. These behaviors fit the numeric type operations fully when ANSI is
off, except 2 and 4.
### Why are the changes needed?
1. bug fix 2. `ANSI` support
### Does this PR introduce any user-facing change? When `ANSI` is on,
interval `add`/`subtract`/`negative`/`multiply`/`divide` will overflow
if any field overflows
### How was this patch tested?
add unit tests
Closes #26995 from yaooqinn/SPARK-30341.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/interval.sql (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/IntervalExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval.sql.out (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala (diff)