SuccessChanges

Summary

  1. [SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via (commit: efbb0359024d1f77e207ca5fc78b3c32b4138558) (details)
  2. [MINOR][SQL] Make analysis error msg more meaningful on DISTINCT queries (commit: c308ab5a29c3f9f92bc73ac6ae984b465b177e97) (details)
  3. [SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move (commit: c097c555acd0c28d71a223e706e79f4dcf521f3b) (details)
  4. Revert "[SPARK-28527][SQL][TEST] Re-run all the tests in (commit: f0834d3a7f945759bbbcba970d24a49ba1d08421) (details)
  5. [SPARK-28598][SQL] Few date time manipulation functions does not provide (commit: 4ddad7906098ccde8d918f42afe856d9fe23b563) (details)
  6. [SPARK-28390][SQL][PYTHON][TESTS][FOLLOW-UP] Update the TODO with actual (commit: c96b6154b7fa9425130f7675e423aced185bd1d7) (details)
  7. [SPARK-27330][SS] support task abort in foreach writer (commit: d75a11d0596516e73e27aea8f5a7e85df0102d5c) (details)
  8. [SPARK-28734][DOC] Initial table of content in the left hand side bar (commit: a5df5ff0fdec0e1a727cb7f2f9bed178d37b3ee5) (details)
  9. [SPARK-28356][SHUFFLE][FOLLOWUP] Fix case with different pre-shuffle (commit: f999e00e9fc03889106a305eeec6d05b6a2c1de1) (details)
Commit efbb0359024d1f77e207ca5fc78b3c32b4138558 by gatorsmile
[SPARK-28527][SQL][TEST] Re-run all the tests in SQLQueryTestSuite via
Thrift Server
## What changes were proposed in this pull request?
This PR build a test framework that directly re-run all the tests in
`SQLQueryTestSuite` via Thrift Server. But it's a little different from
`SQLQueryTestSuite`: 1. Can not support [UDF
testing](https://github.com/apache/spark/blob/44e607e9213bdceab970606fb15292db2fe157c2/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala#L293-L297).
2. Can not support `DESC` command and `SHOW` command because
`SQLQueryTestSuite` [formatted the
output](https://github.com/apache/spark/blob/1882912cca4921d3d8c8632b3bb34e69e8119791/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala#L38-L50.).
When building this framework, found two bug:
[SPARK-28624](https://issues.apache.org/jira/browse/SPARK-28624):
`make_date` is inconsistent when reading from table
[SPARK-28611](https://issues.apache.org/jira/browse/SPARK-28611):
Histogram's height is different
found two features that ThriftServer can not support:
[SPARK-28636](https://issues.apache.org/jira/browse/SPARK-28636):
ThriftServer can not support decimal type with negative scale
[SPARK-28637](https://issues.apache.org/jira/browse/SPARK-28637):
ThriftServer can not support interval type
Also, found two inconsistent behavior:
[SPARK-28620](https://issues.apache.org/jira/browse/SPARK-28620): Double
type returned for float type in Beeline/JDBC
[SPARK-28619](https://issues.apache.org/jira/browse/SPARK-28619):  The
golden result file is different when tested by `bin/spark-sql`
## How was this patch tested?
N/A
Closes #25373 from wangyum/SPARK-28527.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: gatorsmile
<gatorsmile@gmail.com>
(commit: efbb0359024d1f77e207ca5fc78b3c32b4138558)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala (diff)
The file was modifiedproject/SparkBuild.scala (diff)
The file was addedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
Commit c308ab5a29c3f9f92bc73ac6ae984b465b177e97 by dhyun
[MINOR][SQL] Make analysis error msg more meaningful on DISTINCT queries
## What changes were proposed in this pull request?
This PR makes analysis error messages more meaningful when the function
does not support the modifier DISTINCT:
```sql postgres=# select upper(distinct a) from (values('a'), ('b'))
v(a); ERROR:  DISTINCT specified, but upper is not an aggregate function
LINE 1: select upper(distinct a) from (values('a'), ('b')) v(a);
spark-sql> select upper(distinct a) from (values('a'), ('b')) v(a);
Error in query: upper does not support the modifier DISTINCT; line 1 pos
7 spark-sql>
```
After this pr:
```sql spark-sql> select upper(distinct a) from (values('a'), ('b'))
v(a); Error in query: DISTINCT specified, but upper is not an aggregate
function; line 1 pos 7 spark-sql>
```
## How was this patch tested?
Unit test
Closes #25486 from wangyum/DISTINCT.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
(commit: c308ab5a29c3f9f92bc73ac6ae984b465b177e97)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
Commit c097c555acd0c28d71a223e706e79f4dcf521f3b by sean.owen
[SPARK-21067][DOC] Fix Thrift Server - CTAS fail with Unable to move
source
## What changes were proposed in this pull request?
This PR aims to fix CTAS fails after we closed a session of
ThriftServer.
- sql-distributed-sql-engine.md
![image](https://user-images.githubusercontent.com/25916266/62509628-6f854980-b83e-11e9-9bea-daaf76c8f724.png)
It seems the simplest way to fix
[[SPARK-21067]](https://issues.apache.org/jira/browse/SPARK-21067).
For example : If we use HDFS, we can set the following property in
hive-site.xml.
`<property>`
`  <name>fs.hdfs.impl.disable.cache</name>`
`  <value>true</value>`
`</property>`
## How was this patch tested
Manual.
Closes #25364 from Deegue/fix_add_doc_file_system.
Authored-by: Yizhong Zhang <zyzzxycj@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(commit: c097c555acd0c28d71a223e706e79f4dcf521f3b)
The file was modifieddocs/sql-distributed-sql-engine.md (diff)
Commit f0834d3a7f945759bbbcba970d24a49ba1d08421 by dhyun
Revert "[SPARK-28527][SQL][TEST] Re-run all the tests in
SQLQueryTestSuite via Thrift Server"
This reverts commit efbb0359024d1f77e207ca5fc78b3c32b4138558.
(commit: f0834d3a7f945759bbbcba970d24a49ba1d08421)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was removedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala (diff)
The file was modifiedproject/SparkBuild.scala (diff)
Commit 4ddad7906098ccde8d918f42afe856d9fe23b563 by gurwls223
[SPARK-28598][SQL] Few date time manipulation functions does not provide
versions supporting Column as input through the Dataframe API
## What changes were proposed in this pull request?
Add following functions:
``` def add_months(startDate: Column, numMonths: Column): Column def
date_add(start: Column, days: Column): Column def date_sub(start:
Column, days: Column): Column
```
## How was this patch tested?
UT.
Please review https://spark.apache.org/contributing.html before opening
a pull request.
Closes #25334 from WeichenXu123/datefunc_impr.
Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: 4ddad7906098ccde8d918f42afe856d9fe23b563)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/functions.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala (diff)
Commit c96b6154b7fa9425130f7675e423aced185bd1d7 by gurwls223
[SPARK-28390][SQL][PYTHON][TESTS][FOLLOW-UP] Update the TODO with actual
blocking JIRA IDs
## What changes were proposed in this pull request?
only todo message updated. Need to add udf() for GroupBy Tests, after
resolving following jira
[SPARK-28386] and [SPARK-26741]
## How was this patch tested? NA, only TODO message updated.
Closes #25415 from shivusondur/jiraFollowup.
Authored-by: shivusondur <shivusondur@gmail.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: c96b6154b7fa9425130f7675e423aced185bd1d7)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-select_having.sql (diff)
Commit d75a11d0596516e73e27aea8f5a7e85df0102d5c by wenchen
[SPARK-27330][SS] support task abort in foreach writer
## What changes were proposed in this pull request? in order to address
cases where foreach writer task is failing without calling the close()
method, (for example when a task is interrupted) added the option to
implement an abort() method that will be called when the task is
aborted. users should handle resource cleanup (such as connections) in
the abort() method
## How was this patch tested? update existing unit tests.
Closes #24382 from eyalzit/SPARK-27330-foreach-writer-abort.
Lead-authored-by: Eyal Zituny <eyal.zituny@equalum.io> Co-authored-by:
Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com> Co-authored-by: eyalzit
<eyal.zituny@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: d75a11d0596516e73e27aea8f5a7e85df0102d5c)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala (diff)
Commit a5df5ff0fdec0e1a727cb7f2f9bed178d37b3ee5 by gatorsmile
[SPARK-28734][DOC] Initial table of content in the left hand side bar
for SQL doc
## What changes were proposed in this pull request? This is a initial PR
that creates the table of content for SQL reference guide. The left side
bar will displays additional menu items corresponding to supported SQL
constructs. One this PR is merged, we will fill in the content
incrementally.  Additionally this PR contains a minor change to make the
left sidebar scrollable. Currently it is not possible to scroll in the
left hand side window.
## How was this patch tested? Used jekyll build and serve to verify.
Closes #25459 from dilipbiswal/ref-doc.
Authored-by: Dilip Biswal <dbiswal@us.ibm.com> Signed-off-by: gatorsmile
<gatorsmile@gmail.com>
(commit: a5df5ff0fdec0e1a727cb7f2f9bed178d37b3ee5)
The file was addeddocs/sql-ref-functions-builtin-scalar.md
The file was addeddocs/sql-ref-syntax-qry-select.md
The file was addeddocs/sql-ref-syntax-aux-resource-mgmt.md
The file was addeddocs/sql-ref-syntax-ddl-drop-table.md
The file was addeddocs/sql-ref-syntax-ddl.md
The file was addeddocs/sql-ref-functions-udf-scalar.md
The file was addeddocs/sql-ref-syntax-aux-show-table.md
The file was addeddocs/sql-ref-nan-semantics.md
The file was addeddocs/sql-ref-syntax-qry-select-subqueries.md
The file was addeddocs/sql-ref-syntax-qry-select-distinct.md
The file was addeddocs/sql-ref-syntax-aux-show-functions.md
The file was addeddocs/sql-ref-syntax-aux-conf-mgmt.md
The file was addeddocs/sql-ref-syntax-qry-select-hints.md
The file was addeddocs/sql-ref-syntax-aux-cache-clear-cache.md
The file was addeddocs/sql-ref-syntax-aux-describe-query.md
The file was addeddocs/sql-ref-syntax-dml-load.md
The file was addeddocs/sql-ref.md
The file was addeddocs/sql-ref-syntax-ddl-create-database.md
The file was addeddocs/sql-ref-syntax-qry-window.md
The file was addeddocs/sql-ref-syntax-qry-select-join.md
The file was addeddocs/sql-ref-syntax-aux-resource-mgmt-add-jar.md
The file was addeddocs/sql-ref-functions-builtin-aggregate.md
The file was addeddocs/sql-ref-syntax-dml-insert.md
The file was addeddocs/sql-ref-syntax-aux-show-partitions.md
The file was addeddocs/sql-ref-syntax-qry-select-cte.md
The file was addeddocs/sql-ref-syntax-qry-aggregation.md
The file was addeddocs/sql-ref-syntax-ddl-create-table.md
The file was addeddocs/sql-ref-syntax-aux.md
The file was addeddocs/sql-ref-syntax-aux-cache.md
The file was addeddocs/sql-ref-syntax-qry-select-orderby.md
The file was addeddocs/sql-ref-functions.md
The file was addeddocs/sql-ref-syntax-aux-show.md
The file was addeddocs/sql-ref-syntax-dml.md
The file was addeddocs/sql-ref-syntax-ddl-alter-table.md
The file was addeddocs/sql-ref-syntax-aux-cache-uncache-table.md
The file was addeddocs/sql-ref-syntax-aux-describe-database.md
The file was addeddocs/sql-ref-syntax-aux-conf-mgmt-reset.md
The file was addeddocs/sql-ref-syntax-ddl-create-function.md
The file was addeddocs/sql-ref-syntax-aux-analyze-table.md
The file was addeddocs/sql-ref-syntax-aux-show-create-table.md
The file was modifieddocs/_data/menu-sql.yaml (diff)
The file was addeddocs/sql-ref-syntax-qry-select-setops.md
The file was addeddocs/sql-ref-arithmetic-ops.md
The file was addeddocs/sql-ref-syntax-ddl-truncate-table.md
The file was addeddocs/sql-ref-syntax-aux-show-columns.md
The file was addeddocs/sql-ref-syntax-ddl-drop-view.md
The file was addeddocs/sql-ref-datatypes.md
The file was addeddocs/sql-ref-syntax-aux-resource-mgmt-add-file.md
The file was addeddocs/sql-ref-syntax-qry-select-having.md
The file was addeddocs/sql-ref-syntax-ddl-drop-function.md
The file was addeddocs/sql-ref-syntax-qry-explain.md
The file was addeddocs/sql-ref-syntax-ddl-create-view.md
The file was addeddocs/sql-ref-syntax-qry-sampling.md
The file was addeddocs/sql-ref-syntax-aux-cache-cache-table.md
The file was addeddocs/sql-ref-syntax-aux-describe-table.md
The file was removeddocs/sql-reference.md
The file was addeddocs/sql-ref-functions-udf-aggregate.md
The file was addeddocs/sql-ref-syntax-aux-describe.md
The file was addeddocs/sql-ref-syntax-aux-show-tables.md
The file was addeddocs/sql-ref-syntax-qry.md
The file was addeddocs/sql-ref-syntax-ddl-drop-database.md
The file was addeddocs/sql-ref-syntax-aux-show-tblproperties.md
The file was modifieddocs/css/main.css (diff)
The file was addeddocs/sql-ref-syntax-aux-conf-mgmt-set.md
The file was addeddocs/sql-ref-functions-udf.md
The file was addeddocs/sql-ref-syntax-aux-describe-function.md
The file was addeddocs/sql-ref-syntax-ddl-alter-view.md
The file was addeddocs/sql-ref-syntax-aux-analyze.md
The file was addeddocs/sql-ref-syntax-ddl-alter-database.md
The file was addeddocs/sql-ref-syntax-qry-select-groupby.md
The file was addeddocs/sql-ref-syntax-qry-select-limit.md
The file was addeddocs/sql-ref-functions-builtin.md
The file was addeddocs/sql-ref-syntax-aux-show-databases.md
The file was addeddocs/sql-ref-syntax.md
The file was addeddocs/sql-ref-syntax-ddl-repair-table.md
Commit f999e00e9fc03889106a305eeec6d05b6a2c1de1 by wenchen
[SPARK-28356][SHUFFLE][FOLLOWUP] Fix case with different pre-shuffle
partition numbers
### What changes were proposed in this pull request?
This PR reverts some of the latest changes in
`ReduceNumShufflePartitions` to fix the case when there are different
pre-shuffle partition numbers in the plan. Please see the new UT for an
example.
### Why are the changes needed? Eliminate a bug.
### Does this PR introduce any user-facing change? Yes, some queries
that failed will succeed now.
### How was this patch tested? Added new UT.
Closes #25479 from peter-toth/SPARK-28356-followup.
Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
(commit: f999e00e9fc03889106a305eeec6d05b6a2c1de1)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ReduceNumShufflePartitionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ReduceNumShufflePartitions.scala (diff)