FailedChanges

Summary

  1. [SPARK-28833][DOCS][SQL] Document ALTER VIEW command (details)
  2. [SPARK-29865][K8S] Ensure client-mode executors have same name prefix (details)
  3. [SPARK-29857][WEB UI] Defer render the spark UI dataTables (details)
  4. [SPARK-26923][R][SQL][FOLLOW-UP] Show stderr in the exception whenever (details)
  5. [SPARK-29901][SQL][DOC] Fix broken links in SQL Reference (details)
  6. [SPARK-29889][SQL][TEST] unify the interval tests (details)
  7. [SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.1 (details)
  8. [SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 (details)
  9. [SPARK-29888][SQL] new interval string parser shall handle numeric with (details)
  10. [SPARK-29655][SQL] Read bucketed tables obeys (details)
  11. [SPARK-26499][SQL][FOLLOW-UP] Replace `update` with `setByte` for (details)
  12. [SPARK-29902][DOC][MINOR] Add listener event queue capacity (details)
  13. [SPARK-29829][SQL] SHOW TABLE EXTENDED should do multi-catalog (details)
  14. [SPARK-29833][YARN] Add FileNotFoundException check for spark.yarn.jars (details)
  15. [SPARK-29127][SQL][PYTHON] Add a clue for Python related version (details)
  16. [SPARK-29834][SQL] DESC DATABASE should look up catalog like v2 commands (details)
  17. [SPARK-29343][SQL][FOLLOW-UP] Remove floating-point (details)
  18. [SPARK-29867][ML][PYTHON] Add __repr__ in Python ML Models (details)
  19. [SPARK-29923][SQL][TESTS] Set io.netty.tryReflectionSetAccessible for (details)
  20. [SPARK-29807][SQL] Rename "spark.sql.ansi.enabled" to (details)
  21. [MINOR][TESTS] Ignore GitHub Action and AppVeyor file changes in testing (details)
  22. [SPARK-29476][WEBUI] add tooltip for Thread (details)
  23. [SPARK-29928][SQL][TESTS] Check parsing timestamps up to microsecond (details)
  24. [SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API (details)
  25. [SPARK-29858][SQL] ALTER DATABASE (SET DBPROPERTIES) should look up (details)
  26. [SPARK-29456][WEBUI] Improve tooltip for Session Statistics Table column (details)
  27. [SPARK-29932][R][TESTS] lint-r should do non-zero exit in case of errors (details)
  28. [SPARK-29930][SQL] Remove SQL configs declared to be removed in Spark (details)
  29. [SPARK-16872][ML][PYSPARK] Impl Gaussian Naive Bayes Classifier (details)
  30. [SPARK-29907][SQL] Move DELETE/UPDATE/MERGE relative rules to (details)
  31. [SPARK-29378][R][FOLLOW-UP] Remove manual installation of Arrow (details)
  32. [SPARK-29936][R] Fix SparkR lint errors and add lint-r GitHub Action (details)
  33. [SPARK-25694][SQL] Add a config for `URL.setURLStreamHandlerFactory` (details)
  34. [SPARK-29020][SQL] Improving array_sort behaviour (details)
  35. [SPARK-29783][SQL] Support SQL Standard/ISO_8601 output style for (details)
  36. [SPARK-29870][SQL] Unify the logic of multi-units interval string to (details)
  37. [SPARK-29859][SQL] ALTER DATABASE (SET LOCATION) should look up catalog (details)
  38. [SPARK-29941][SQL] Add ansi type aliases for char and decimal (details)
  39. [SPARK-29873][SQL][TEST][FOLLOWUP] set operations should not escape when (details)
  40. [SPARK-29777][SPARKR] SparkR::cleanClosure aggressively removes a (details)
  41. [SPARK-25694][SQL][FOLLOW-UP] Move (details)
  42. [SPARK-29870][SQL][FOLLOW-UP] Keep CalendarInterval's toString (details)
  43. [SPARK-28527][FOLLOW-UP][SQL][TEST] Add guides for (details)
  44. Revert "[SPARK-29644][SQL] Corrected ShortType and ByteType mapping to (details)
  45. [SPARK-29728][SQL] Datasource V2: Support ALTER TABLE RENAME TO (details)
  46. [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM (details)
  47. [SPARK-29869][SQL] improve error message in (details)
  48. [SPARK-29948][SQL] make the default alias consistent between date, (details)
  49. [SPARK-29918][SQL] RecordBinaryComparator should check endianness when (details)
  50. [SPARK-29893] improve the local shuffle reader performance by changing (details)
  51. [SPARK-29926][SQL] Fix weird interval string whose value is only a (details)
  52. [SPARK-29913][SQL] Improve Exception in postgreCastToBoolean (details)
  53. [SPARK-29906][SQL] AQE should not introduce extra shuffle for outermost (details)
  54. [SPARK-29691][ML][PYTHON] ensure Param objects are valid in fit, (details)
  55. [SPARK-29964][BUILD] lintr github workflows failed due to buggy GnuPG (details)
  56. [SPARK-29963][SQL][TESTS] Check formatting timestamps up to microsecond (details)
  57. [SPARK-29945][SQL] do not handle negative sign specially in the parser (details)
  58. [SPARK-29029][SQL] Use AttributeMap in (details)
  59. [SPARK-29968][SQL] Remove the Predicate code from SparkPlan (details)
  60. [SPARK-18409][ML] LSH approxNearestNeighbors should use approxQuantile (details)
  61. [SPARK-29894][SQL][WEBUI] Add Codegen Stage Id to Spark plan graphs in (details)
  62. [SPARK-28885][SQL][FOLLOW-UP] Re-enable the ported PgSQL regression (details)
  63. [SPARK-29951][SQL] Make the behavior of Postgre dialect independent of (details)
  64. [MINOR][TESTS] Replace JVM assert with JUnit Assert in tests (details)
  65. [SPARK-29911][SQL] Uncache cached tables when session closed (details)
  66. [SPARK-29978][SQL][TESTS] Check `json_tuple` does not truncate results (details)
  67. [SPARK-29961][SQL] Implement builtin function - typeof (details)
  68. [SPARK-22340][PYTHON][FOLLOW-UP] Add a better message and improve (details)
  69. [SPARK-29961][SQL][FOLLOWUP] Remove useless test for VectorUDT (details)
  70. [MINOR][INFRA] Add `io` and `net` to GitHub Action Cache (details)
  71. [SPARK-29885][PYTHON][CORE] Improve the exception message when reading (details)
  72. [SPARK-29942][ML] Impl Complement Naive Bayes Classifier (details)
  73. [SPARK-29960][ML][PYSPARK] MulticlassClassificationEvaluator support (details)
  74. [SPARK-29874][SQL] Optimize Dataset.isEmpty() (details)
  75. [SPARK-29977][SQL] Remove (details)
  76. [SPARK-29248][SQL] provider number of partitions when creating v2 data (details)
  77. [SPARK-29558][SQL] ResolveTables and ResolveRelations should be (details)
  78. [SPARK-29975][SQL] introduce --CONFIG_DIM directive (details)
  79. [SPARK-29911][SQL][FOLLOWUP] Move related unit test to (details)
  80. [SPARK-28023][SQL] Add trim logic in UTF8String's toInt/toLong to make (details)
  81. [SPARK-29893][SQL][FOLLOWUP] code cleanup for local shuffle reader (details)
  82. [SPARK-29427][SQL] Add API to convert RelationalGroupedDataset to (details)
  83. [SPARK-29971][CORE] Fix buffer leaks in (details)
  84. [SPARK-30003][SQL] Do not throw stack overflow exception in non-root (details)
  85. [SPARK-29981][BUILD] Add hive-1.2/2.3 profiles (details)
  86. [SPARK-29981][BUILD][FOLLOWUP] Change hive.version.short (details)
Commit fca0a6c394990b86304a8f9a64bf4c7ec58abbd6 by sean.owen
[SPARK-28833][DOCS][SQL] Document ALTER VIEW command
### What changes were proposed in this pull request? Document ALTER VIEW
statement in the SQL Reference Guide.
### Why are the changes needed? Currently Spark SQL doc lacks
documentation on the supported SQL syntax. This pr is aimed to address
this issue.
### Does this PR introduce any user-facing change? Yes
#### Before: There was no documentation for this.
#### After:
![Screen Shot 2019-11-13 at 10 51 33
PM](https://user-images.githubusercontent.com/7550280/68833575-ac947f80-0668-11ea-910f-c133407ef502.png)
![Screen Shot 2019-11-13 at 10 56 42
PM](https://user-images.githubusercontent.com/7550280/68833597-bae29b80-0668-11ea-9782-b7be94789c12.png)
![Screen Shot 2019-11-13 at 10 56 53
PM](https://user-images.githubusercontent.com/7550280/68833607-be762280-0668-11ea-8a30-5602e755bab8.png)
### How was this patch tested? Tested using jkyll build --serve
Closes #25573 from kevinyu98/spark-28833-alterview.
Authored-by: Kevin Yu <qyu@us.ibm.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifieddocs/sql-ref-syntax-ddl-alter-view.md (diff)
Commit b095232f630221926a9eabb8233c20d03c9a6ebb by eerlands
[SPARK-29865][K8S] Ensure client-mode executors have same name prefix
This basically does what BasicDriverFeatureStep already does to achieve
the same thing in cluster mode; but since that class (or any other
feature) is not invoked in client mode, it needs to be done elsewhere.
I also modified the client mode integration test to check the executor
name prefix; while there I had to fix the minikube backend to parse the
output from newer minikube versions (I have 1.5.2).
Closes #26488 from vanzin/SPARK-29865.
Authored-by: Marcelo Vanzin <vanzin@cloudera.com> Signed-off-by: Erik
Erlandson <eerlands@redhat.com>
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala (diff)
Commit ab981f10a61ac15be14334b015fe2206e088314f by sean.owen
[SPARK-29857][WEB UI] Defer render the spark UI dataTables
### What changes were proposed in this pull request? This PR support
defer render the spark UI page.
### Why are the changes needed? When there are many items, such as tasks
and application lists, the renderer of dataTables is heavy, we can
enable deferRender to optimize it. See details in
https://datatables.net/examples/ajax/defer_render.html
### Does this PR introduce any user-facing change? No.
### How was this patch tested? Not needed.
Closes #26482 from turboFei/SPARK-29857-defer-render.
Authored-by: turbofei <fwang12@ebay.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/stagepage.js (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/executorspage.js (diff)
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/historypage.js (diff)
Commit 17321782deac59516639aab648cb319a51c0bb9b by gurwls223
[SPARK-26923][R][SQL][FOLLOW-UP] Show stderr in the exception whenever
possible in RRunner
### What changes were proposed in this pull request?
This is a followup of https://github.com/apache/spark/pull/23977 I made
a mistake related to this line:
https://github.com/apache/spark/commit/3725b1324f731d57dc776c256bc1a100ec9e6cd0#diff-71c2cad03f08cb5f6c70462aa4e28d3aL112
Previously,
1. the reader iterator for R worker read some initial data eagerly
during RDD materialization. So it read the data before actual execution.
For some reasons, in this case, it showed standard error from R worker.
2. After that, when error happens during actual execution, stderr wasn't
shown:
https://github.com/apache/spark/commit/3725b1324f731d57dc776c256bc1a100ec9e6cd0#diff-71c2cad03f08cb5f6c70462aa4e28d3aL260
After my change
https://github.com/apache/spark/commit/3725b1324f731d57dc776c256bc1a100ec9e6cd0#diff-71c2cad03f08cb5f6c70462aa4e28d3aL112,
it now ignores 1. case and only does 2. of previous code path, because
1. does not happen anymore as I avoided to such eager execution (which
is consistent with PySpark code path).
This PR proposes to do only 1.  before/after execution always because It
is pretty much possible R worker was failed during actual execution and
it's best to show the stderr from R worker whenever possible.
### Why are the changes needed?
It currently swallows standard error from R worker which makes debugging
harder.
### Does this PR introduce any user-facing change?
Yes,
```R df <- createDataFrame(list(list(n=1))) collect(dapply(df,
function(x) {
stop("asdkjasdjkbadskjbsdajbk")
x
}, structType("a double")))
```
**Before:**
``` Error in handleErrors(returnStatus, conn) :
org.apache.spark.SparkException: Job aborted due to stage failure: Task
0 in stage 13.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 13.0 (TID 13, 192.168.35.193, executor driver):
org.apache.spark.SparkException: R worker exited unexpectedly (cranshed)
at org.apache.spark.api.r.RRunner$$anon$1.read(RRunner.scala:130)
at
org.apache.spark.api.r.BaseRRunner$ReaderIterator.hasNext(BaseRRunner.scala:118)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:337)
at org.apache.spark.
```
**After:**
``` Error in handleErrors(returnStatus, conn) :
org.apache.spark.SparkException: Job aborted due to stage failure: Task
0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 1.0 (TID 1, 192.168.35.193, executor driver):
org.apache.spark.SparkException: R unexpectedly exited. R worker
produced errors: Error in computeFunc(inputData) :
asdkjasdjkbadskjbsdajbk
at
org.apache.spark.api.r.BaseRRunner$ReaderIterator$$anonfun$1.applyOrElse(BaseRRunner.scala:144)
at
org.apache.spark.api.r.BaseRRunner$ReaderIterator$$anonfun$1.applyOrElse(BaseRRunner.scala:137)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at org.apache.spark.api.r.RRunner$$anon$1.read(RRunner.scala:128)
at
org.apache.spark.api.r.BaseRRunner$ReaderIterator.hasNext(BaseRRunner.scala:113)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegen
```
### How was this patch tested?
Manually tested and unittest was added.
Closes #26517 from HyukjinKwon/SPARK-26923-followup.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedcore/src/main/scala/org/apache/spark/api/r/BaseRRunner.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/r/RRunner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/r/ArrowRRunner.scala (diff)
The file was modifiedR/pkg/tests/fulltests/test_sparkSQL.R (diff)
Commit d128ef13d8e4ccd46a6eac90834b7520f6684fa4 by gurwls223
[SPARK-29901][SQL][DOC] Fix broken links in SQL Reference
### What changes were proposed in this pull request? Fix broken links
### How was this patch tested? Tested using jykyll build --serve
Closes #26528 from huaxingao/spark-29901.
Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifieddocs/sql-ref-syntax-aux-show-tblproperties.md (diff)
The file was modifieddocs/sql-ref-syntax-ddl-create-view.md (diff)
The file was modifieddocs/sql-ref-syntax-ddl-truncate-table.md (diff)
The file was modifieddocs/sql-ref-syntax-aux-show-databases.md (diff)
Commit bb8b04d4a2b311fbaeb0f4cb9daa6a30e4478532 by wenchen
[SPARK-29889][SQL][TEST] unify the interval tests
### What changes were proposed in this pull request?
move interval tests to `interval.sql`, and import it to
`ansi/interval.sql`
### Why are the changes needed?
improve test coverage
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
N/A
Closes #26515 from cloud-fan/test.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/group-by.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/interval.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/group-by.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/ansi/interval.sql (diff)
Commit 65a189c7a1ddceb8ab482ccc60af5350b8da5ea5 by gurwls223
[SPARK-29376][SQL][PYTHON] Upgrade Apache Arrow to version 0.15.1
### What changes were proposed in this pull request?
Upgrade Apache Arrow to version 0.15.1. This includes Java artifacts and
increases the minimum required version of PyArrow also.
Version 0.12.0 to 0.15.1 includes the following selected
fixes/improvements relevant to Spark users:
* ARROW-6898 - [Java] Fix potential memory leak in ArrowWriter and
several test classes
* ARROW-6874 - [Python] Memory leak in Table.to_pandas() when conversion
to object dtype
* ARROW-5579 - [Java] shade flatbuffer dependency
* ARROW-5843 - [Java] Improve the readability and performance of
BitVectorHelper#getNullCount
* ARROW-5881 - [Java] Provide functionalities to efficiently determine
if a validity buffer has completely 1 bits/0 bits
* ARROW-5893 - [C++] Remove arrow::Column class from C++ library
* ARROW-5970 - [Java] Provide pointer to Arrow buffer
* ARROW-6070 - [Java] Avoid creating new schema before IPC sending
* ARROW-6279 - [Python] Add Table.slice method or allow slices in
\_\_getitem\_\_
* ARROW-6313 - [Format] Tracking for ensuring flatbuffer serialized
values are aligned in stream/files.
* ARROW-6557 - [Python] Always return pandas.Series from
Array/ChunkedArray.to_pandas, propagate field names to Series from
RecordBatch, Table
* ARROW-2015 - [Java] Use Java Time and Date APIs instead of JodaTime
* ARROW-1261 - [Java] Add container type for Map logical type
* ARROW-1207 - [C++] Implement Map logical type
Changelog can be seen at https://arrow.apache.org/release/0.15.0.html
### Why are the changes needed?
Upgrade to get bug fixes, improvements, and maintain compatibility with
future versions of PyArrow.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Existing tests, manually tested with Python 3.7, 3.8
Closes #26133 from BryanCutler/arrow-upgrade-015-SPARK-29376.
Authored-by: Bryan Cutler <cutlerb@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedpom.xml (diff)
The file was modifiedpython/pyspark/sql/utils.py (diff)
The file was modifieddev/deps/spark-deps-hadoop-3.2 (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala (diff)
The file was modifiedpython/setup.py (diff)
The file was modifieddev/deps/spark-deps-hadoop-2.7 (diff)
Commit d1ac25ba33174afaabc51893962f94597d3b1b22 by gurwls223
[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3
### What changes were proposed in this pull request?
This PR proposes to switch `pygments.rb`, which only support Python 2
and seems inactive for the last few years
(https://github.com/tmm1/pygments.rb), to Rouge which is pure Ruby code
highlighter that is compatible with Pygments.
I thought it would be pretty difficult to change but thankfully Rouge
does a great job as the alternative.
### Why are the changes needed?
We're moving to Python 3 and drop Python 2 completely.
### Does this PR introduce any user-facing change?
Maybe a little bit of different syntax style but should not have a
notable change.
### How was this patch tested?
Manually tested the build and checked the documentation.
Closes #26521 from HyukjinKwon/SPARK-28752.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifieddocs/_plugins/include_example.rb (diff)
The file was modifieddocs/css/pygments-default.css (diff)
The file was modifieddev/create-release/spark-rm/Dockerfile (diff)
The file was modifieddocs/README.md (diff)
The file was modifieddocs/_config.yml (diff)
Commit 0c68578fa9d03002a2d1708762c3010fd28430c5 by wenchen
[SPARK-29888][SQL] new interval string parser shall handle numeric with
only fractional part
### What changes were proposed in this pull request?
Current string to interval cast logic does not support i.e. cast('.111
second' as interval) which will fail in SIGN state and return null,
actually, it is 00:00:00.111.
```scala
-- !query 63 select interval '.111 seconds'
-- !query 63 schema struct<0.111 seconds:interval>
-- !query 63 output 0.111 seconds
-- !query 64 select cast('.111 seconds' as interval)
-- !query 64 schema struct<CAST(.111 seconds AS INTERVAL):interval>
-- !query 64 output NULL
````
### Why are the changes needed?
bug fix.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
add ut
Closes #26514 from yaooqinn/SPARK-29888.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala (diff)
Commit 4f10e54ba385daa37598efa49dbfb536a7726dbc by wenchen
[SPARK-29655][SQL] Read bucketed tables obeys
spark.sql.shuffle.partitions
### What changes were proposed in this pull request?
In order to avoid frequently changing the value of
`spark.sql.adaptive.shuffle.maxNumPostShufflePartitions`, we usually set
`spark.sql.adaptive.shuffle.maxNumPostShufflePartitions` much larger
than `spark.sql.shuffle.partitions` after enabling adaptive execution,
which causes some bucket map join lose efficacy and add more
`ShuffleExchange`.
How to reproduce:
```scala val bucketedTableName = "bucketed_table"
spark.range(10000).write.bucketBy(500,
"id").sortBy("id").mode(org.apache.spark.sql.SaveMode.Overwrite).saveAsTable(bucketedTableName)
val bucketedTable = spark.table(bucketedTableName) val df =
spark.range(8)
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
// Spark 2.4. spark.sql.adaptive.enabled=false
// We set spark.sql.shuffle.partitions <= 500 every time based on our
data in this case. spark.conf.set("spark.sql.shuffle.partitions", 500)
bucketedTable.join(df, "id").explain()
// Since 3.0. We enabled adaptive execution and set
spark.sql.adaptive.shuffle.maxNumPostShufflePartitions to a larger
values to fit more cases. spark.conf.set("spark.sql.adaptive.enabled",
true)
spark.conf.set("spark.sql.adaptive.shuffle.maxNumPostShufflePartitions",
1000) bucketedTable.join(df, "id").explain()
```
``` scala> bucketedTable.join(df, "id").explain()
== Physical Plan ==
*(4) Project [id#5L]
+- *(4) SortMergeJoin [id#5L], [id#7L], Inner
  :- *(1) Sort [id#5L ASC NULLS FIRST], false, 0
  :  +- *(1) Project [id#5L]
  :     +- *(1) Filter isnotnull(id#5L)
  :        +- *(1) ColumnarToRow
  :           +- FileScan parquet default.bucketed_table[id#5L] Batched:
true, DataFilters: [isnotnull(id#5L)], Format: Parquet, Location:
InMemoryFileIndex[file:/root/opensource/apache-spark/spark-3.0.0-SNAPSHOT-bin-3.2.0/spark-warehou...,
PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema:
struct<id:bigint>, SelectedBucketsCount: 500 out of 500
  +- *(3) Sort [id#7L ASC NULLS FIRST], false, 0
     +- Exchange hashpartitioning(id#7L, 500), true, [id=#49]
        +- *(2) Range (0, 8, step=1, splits=16)
``` vs
``` scala> bucketedTable.join(df, "id").explain()
== Physical Plan == AdaptiveSparkPlan(isFinalPlan=false)
+- Project [id#5L]
  +- SortMergeJoin [id#5L], [id#7L], Inner
     :- Sort [id#5L ASC NULLS FIRST], false, 0
     :  +- Exchange hashpartitioning(id#5L, 1000), true, [id=#93]
     :     +- Project [id#5L]
     :        +- Filter isnotnull(id#5L)
     :           +- FileScan parquet default.bucketed_table[id#5L]
Batched: true, DataFilters: [isnotnull(id#5L)], Format: Parquet,
Location:
InMemoryFileIndex[file:/root/opensource/apache-spark/spark-3.0.0-SNAPSHOT-bin-3.2.0/spark-warehou...,
PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema:
struct<id:bigint>, SelectedBucketsCount: 500 out of 500
     +- Sort [id#7L ASC NULLS FIRST], false, 0
        +- Exchange hashpartitioning(id#7L, 1000), true, [id=#92]
           +- Range (0, 8, step=1, splits=16)
```
This PR makes read bucketed tables always obeys
`spark.sql.shuffle.partitions` even enabling adaptive execution and set
`spark.sql.adaptive.shuffle.maxNumPostShufflePartitions` to avoid add
more `ShuffleExchange`.
### Why are the changes needed? Do not degrade performance after
enabling adaptive execution.
### Does this PR introduce any user-facing change? No.
### How was this patch tested? Unit test.
Closes #26409 from wangyum/SPARK-29655.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ReduceNumShufflePartitionsSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala (diff)
Commit ee4784bf26421b830f02c6d65faae4e546c622a8 by sean.owen
[SPARK-26499][SQL][FOLLOW-UP] Replace `update` with `setByte` for
ByteType in JdbcUtils.makeGetter
### What changes were proposed in this pull request?
This is a follow-up pr to fix the code coming from #23400; it replaces
`update` with `setByte` for ByteType in `JdbcUtils.makeGetter`.
### Why are the changes needed?
For better code.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #26532 from maropu/SPARK-26499-FOLLOWUP.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Sean
Owen <sean.owen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala (diff)
Commit 15218898cdc540420d6a6c957e5040f78e75cc61 by sean.owen
[SPARK-29902][DOC][MINOR] Add listener event queue capacity
configuration to documentation
### What changes were proposed in this pull request?
Add listener event queue capacity configuration to documentation
### Why are the changes needed?
We some time see many event drops happening in eventLog listener queue.
So, instead of increasing all the queues size, using this config we just
need to increase eventLog queue capacity.
``` scala> sc.parallelize(1 to 100000, 100000).count()
[Stage 0:=================================================>(98299 + 4) /
100000]19/11/14 20:56:35 ERROR AsyncEventQueue: Dropping event from
queue eventLog. This likely means one of the listeners is too slow and
cannot keep up with the rate at which tasks are being started by the
scheduler. 19/11/14 20:56:35 WARN AsyncEventQueue: Dropped 1 events from
eventLog since the application started.
```
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Existing tests
Closes #26529 from shahidki31/master1.
Authored-by: shahid <shahidki31@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifieddocs/configuration.md (diff)
Commit 848bdfa218c5da55d7fbc0cf82866e6b1c4a09e6 by dhyun
[SPARK-29829][SQL] SHOW TABLE EXTENDED should do multi-catalog
resolution
### What changes were proposed in this pull request?
Add ShowTableStatement and make SHOW TABLE EXTENDED go through the same
catalog/table resolution framework of v2 commands.
We don’t have this methods in the catalog to implement an V2 command
- catalog.getPartition
- catalog.getTempViewOrPermanentTableMetadata
### Why are the changes needed?
It's important to make all the commands have the same table resolution
behavior, to avoid confusing
```sql USE my_catalog DESC t // success and describe the table t from
my_catalog SHOW TABLE EXTENDED FROM LIKE 't' // report table not found
as there is no table t in the session catalog
```
### Does this PR introduce any user-facing change?
Yes. When running SHOW TABLE EXTENDED Spark fails the command if the
current catalog is set to a v2 catalog, or the table name specified a v2
catalog.
### How was this patch tested?
Unit tests.
Closes #26540 from planga82/feature/SPARK-29481_ShowTableExtended.
Authored-by: Pablo Langa <soypab@gmail.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
Commit c0507e0f7530032b26db6ac1611a6a53b9802d5a by vanzin
[SPARK-29833][YARN] Add FileNotFoundException check for spark.yarn.jars
### What changes were proposed in this pull request?
When set `spark.yarn.jars=/xxx/xxx` which is just a no schema path,
spark will throw a NullPointerException.
The reason is hdfs will return null if pathFs.globStatus(path) is not
exist, and spark just use `pathFs.globStatus(path).filter(_.isFile())`
without check it.
### Why are the changes needed?
Avoid NullPointerException.
### Does this PR introduce any user-facing change?
Yes. User will get a FileNotFoundException instead NullPointerException
when `spark.yarn.jars` does not have schema and not exists.
### How was this patch tested?
Add UT.
Closes #26462 from ulysses-you/check-yarn-jars-path-exist.
Authored-by: ulysses <youxiduo@weidian.com> Signed-off-by: Marcelo
Vanzin <vanzin@cloudera.com>
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala (diff)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala (diff)
Commit 7720781695d47fe0375f6e1150f6981b886686bd by dhyun
[SPARK-29127][SQL][PYTHON] Add a clue for Python related version
information in integrated UDF tests
### What changes were proposed in this pull request?
This PR proposes to show Python, pandas and PyArrow versions in
integrated UDF tests as a clue so when the test cases fail, it show the
related version information.
I think we don't really need this kind of version information in the
test case name for now since I intend that integrated SQL test cases do
not target to test different combinations of Python, Pandas and PyArrow.
### Why are the changes needed?
To make debug easier.
### Does this PR introduce any user-facing change?
It will change test name to include related Python, pandas and PyArrow
versions.
### How was this patch tested?
Manually tested:
```
[info] - udf/postgreSQL/udf-case.sql - Scala UDF *** FAILED *** (8
seconds, 229 milliseconds)
[info]   udf/postgreSQL/udf-case.sql - Scala UDF
...
[info] - udf/postgreSQL/udf-case.sql - Regular Python UDF *** FAILED ***
(6 seconds, 298 milliseconds)
[info]   udf/postgreSQL/udf-case.sql - Regular Python UDF
[info]   Python: 3.7
...
[info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF *** FAILED ***
(6 seconds, 376 milliseconds)
[info]   udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
[info]   Python: 3.7 Pandas: 0.25.3 PyArrow: 0.14.0
```
Closes #26538 from HyukjinKwon/investigate-flaky-test.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala (diff)
Commit 16e7195299d864b9e98ed17a9747d53c6a001024 by dhyun
[SPARK-29834][SQL] DESC DATABASE should look up catalog like v2 commands
### What changes were proposed in this pull request? Add
DescribeNamespaceStatement, DescribeNamespace and DescribeNamespaceExec
to make "DESC DATABASE" look up catalog like v2 commands.
### Why are the changes needed? It's important to make all the commands
have the same catalog/namespace resolution behavior, to avoid confusing
end-users.
### Does this PR introduce any user-facing change? Yes, add "DESC
NAMESPACE" whose function is same as "DESC DATABASE" and "DESC SCHEMA".
### How was this patch tested? New unit test
Closes #26513 from fuwhu/SPARK-29834.
Authored-by: fuwhu <bestwwg@163.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was addedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeNamespaceExec.scala
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala (diff)
Commit 6d6b233791a0a18713a3234f10ba234d92083d68 by dhyun
[SPARK-29343][SQL][FOLLOW-UP] Remove floating-point
Sum/Average/CentralMomentAgg from order-insensitive aggregates
### What changes were proposed in this pull request?
This pr is to remove floating-point `Sum/Average/CentralMomentAgg` from
order-insensitive aggregates in `EliminateSorts`.
This pr comes from the gatorsmile suggestion:
https://github.com/apache/spark/pull/26011#discussion_r344583899
### Why are the changes needed?
Bug fix.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added tests in `SubquerySuite`.
Closes #26534 from maropu/SPARK-29343-FOLLOWUP.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Dongjoon Hyun <dhyun@apple.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala (diff)
Commit 1112fc6029f48918cdd05d0736e02f446262f398 by dhyun
[SPARK-29867][ML][PYTHON] Add __repr__ in Python ML Models
### What changes were proposed in this pull request? Add ```__repr__```
in Python ML Models
### Why are the changes needed? In Python ML Models, some of them have
```__repr__```, others don't. In the doctest, when calling Model.setXXX,
some of the Models print out the xxxModel... correctly, some of them
can't because of lacking the  ```__repr__``` method. For example:
```
   >>> gm = GaussianMixture(k=3, tol=0.0001, seed=10)
   >>> model = gm.fit(df)
   >>> model.setPredictionCol("newPrediction")
   GaussianMixture...
``` After the change, the above code will become the following:
```
   >>> gm = GaussianMixture(k=3, tol=0.0001, seed=10)
   >>> model = gm.fit(df)
   >>> model.setPredictionCol("newPrediction")
   GaussianMixtureModel...
```
### Does this PR introduce any user-facing change? Yes.
### How was this patch tested? doctest
Closes #26489 from huaxingao/spark-29876.
Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedpython/pyspark/ml/feature.py (diff)
The file was modifiedpython/pyspark/ml/tree.py (diff)
The file was modifiedpython/pyspark/ml/wrapper.py (diff)
The file was modifiedpython/pyspark/ml/regression.py (diff)
The file was modifiedpython/pyspark/ml/classification.py (diff)
The file was modifiedpython/pyspark/ml/fpm.py (diff)
The file was modifiedpython/pyspark/ml/recommendation.py (diff)
The file was modifiedpython/pyspark/ml/clustering.py (diff)
Commit f77c10de38d0563b2e42d1200a1fbbdb3018c2e9 by dhyun
[SPARK-29923][SQL][TESTS] Set io.netty.tryReflectionSetAccessible for
Arrow on JDK9+
### What changes were proposed in this pull request?
This PR aims to add `io.netty.tryReflectionSetAccessible=true` to the
testing configuration for JDK11 because this is an officially documented
requirement of Apache Arrow.
Apache Arrow community documented this requirement at `0.15.0`
([ARROW-6206](https://github.com/apache/arrow/pull/5078)).
> #### For java 9 or later, should set
"-Dio.netty.tryReflectionSetAccessible=true".
> This fixes `java.lang.UnsupportedOperationException: sun.misc.Unsafe
or java.nio.DirectByteBuffer.(long, int) not available`. thrown by
netty.
### Why are the changes needed?
After ARROW-3191, Arrow Java library requires the property
`io.netty.tryReflectionSetAccessible` to be set to true for JDK >= 9.
After https://github.com/apache/spark/pull/26133, JDK11 Jenkins job seem
to fail.
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/676/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/677/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/678/
```scala Previous exception in task: sun.misc.Unsafe or
java.nio.DirectByteBuffer.<init>(long, int) not available&#010;
io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:473)&#010;
io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)&#010;
io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)&#010;
io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)&#010;
org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)&#010;
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the Jenkins with JDK11.
Closes #26552 from dongjoon-hyun/SPARK-ARROW-JDK11.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedproject/SparkBuild.scala (diff)
The file was modifiedpython/run-tests.py (diff)
The file was modifiedpom.xml (diff)
The file was modifiedsql/catalyst/pom.xml (diff)
The file was modifiedR/run-tests.sh (diff)
The file was modifiedsql/core/pom.xml (diff)
The file was modifiedsql/hive/pom.xml (diff)
Commit 40ea4a11d7f1534023669f0b81faf5d398174e46 by wenchen
[SPARK-29807][SQL] Rename "spark.sql.ansi.enabled" to
"spark.sql.dialect.spark.ansi.enabled"
### What changes were proposed in this pull request? Rename config
"spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled"
### Why are the changes needed? The relation between
"spark.sql.ansi.enabled" and "spark.sql.dialect" is confusing, since the
"PostgreSQL" dialect should contain the features of
"spark.sql.ansi.enabled".
To make things clearer, we can rename the "spark.sql.ansi.enabled" to
"spark.sql.dialect.spark.ansi.enabled", thus the option
"spark.sql.dialect.spark.ansi.enabled" is only for Spark dialect.
For the casting and arithmetic operations, runtime exceptions should be
thrown if "spark.sql.dialect" is "spark" and
"spark.sql.dialect.spark.ansi.enabled" is true or "spark.sql.dialect" is
PostgresSQL.
### Does this PR introduce any user-facing change? Yes, the config name
changed.
### How was this patch tested? Existing UT.
Closes #26444 from xuanyuanking/SPARK-29807.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala (diff)
The file was modifieddocs/sql-keywords.md (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/text.sql (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DecimalExpressionSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql.out (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala (diff)
Commit d0470d639412ecbe6e126f8d8abf5a5819b9e278 by dhyun
[MINOR][TESTS] Ignore GitHub Action and AppVeyor file changes in testing
### What changes were proposed in this pull request?
This PR aims to ignore `GitHub Action` and `AppVeyor` file changes. When
we touch these files, Jenkins job should not trigger a full testing.
### Why are the changes needed?
Currently, these files are categorized to `root` and trigger the full
testing and ends up wasting the Jenkins resources.
- https://github.com/apache/spark/pull/26555
```
[info] Using build tool sbt with Hadoop profile hadoop2.7 under
environment amplab_jenkins From https://github.com/apache/spark
* [new branch]      master     -> master
[info] Found the following changed modules: sparkr, root
[info] Setup the following environment variables for tests:
```
### Does this PR introduce any user-facing change?
No. (Jenkins testing only).
### How was this patch tested?
Manually.
```
$ dev/run-tests.py -h -v
... Trying:
   [x.name for x in
determine_modules_for_files([".github/workflows/master.yml",
"appveyor.xml"])] Expecting:
   []
...
```
Closes #26556 from dongjoon-hyun/SPARK-IGNORE-APPVEYOR.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifieddev/run-tests.py (diff)
Commit 53364730049981e3d36b6e6a405b1df5314d321d by sean.owen
[SPARK-29476][WEBUI] add tooltip for Thread
### What changes were proposed in this pull request? Adding tooltip for
Thread Dump - Thread Locks
Before:
![Screenshot from 2019-11-04
17-11-22](https://user-images.githubusercontent.com/51401130/68127349-b963f580-ff3b-11e9-8547-e01907382632.png)
After:
![Screenshot from 2019-11-13
18-12-54](https://user-images.githubusercontent.com/51401130/68768698-08e7a700-0649-11ea-804b-2eb4d5f162b4.png)
### Why are the changes needed? Thread Dump tab do not have any tooltip
for the columns, Some page provide tooltip , inorder to resolve the
inconsistency and for better user experience.
### Does this PR introduce any user-facing change? No
### How was this patch tested? Manual
Closes #26386 from PavithraRamachandran/threadDump_tooltip.
Authored-by: Pavithra Ramachandran <pavi.rams@gmail.com> Signed-off-by:
Sean Owen <sean.owen@databricks.com>
The file was modifiedcore/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala (diff)
Commit e88267cb5a26b687f69977c9792af18fc3833db1 by dhyun
[SPARK-29928][SQL][TESTS] Check parsing timestamps up to microsecond
precision by JSON/CSV datasource
### What changes were proposed in this pull request? In the PR, I
propose to add tests from the commit
https://github.com/apache/spark/commit/9c7e8be1dca8285296f3052c41f35043699d7d10
for Spark 2.4 that check parsing of timestamp strings for various
seconds fractions.
### Why are the changes needed? To make sure that current behavior is
the same as in Spark 2.4
### Does this PR introduce any user-facing change? No
### How was this patch tested? By running `CSVSuite`,
`JsonFunctionsSuite` and `TimestampFormatterSuite`.
Closes #26558 from MaxGekk/parse-timestamp-micros-tests.
Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala (diff)
Commit cc12cf6029b1b119599a381f4e4600c0a6525ff2 by dhyun
[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API
### What changes were proposed in this pull request?
[[SPARK-29376] Upgrade Apache Arrow to version
0.15.1](https://github.com/apache/spark/pull/26133) upgrades to Arrow
0.15 at Scala/Java/Python. This PR aims to upgrade `SparkR` to use Arrow
0.15 API. Currently, it's broken.
### Why are the changes needed?
First of all, it turns out that our Jenkins jobs (including PR builder)
ignores Arrow test. Arrow 0.15 has a breaking R API changes at
[ARROW-5505](https://issues.apache.org/jira/browse/ARROW-5505) and we
missed that. AppVeyor was the only one having SparkR Arrow tests but
it's broken now.
**Jenkins**
``` Skipped
------------------------------------------------------------------------
1. createDataFrame/collect Arrow optimization (test_sparkSQL_arrow.R#25)
- arrow not installed
```
Second, Arrow throws OOM on AppVeyor environment (Windows JDK8) like the
following because it still has Arrow 0.14.
``` Warnings
-----------------------------------------------------------------------
1. createDataFrame/collect Arrow optimization (test_sparkSQL_arrow.R#39)
- createDataFrame attempted Arrow optimization because
'spark.sql.execution.arrow.sparkr.enabled' is set to true; however,
failed, attempting non-optimization. Reason: Error in
handleErrors(returnStatus, conn): java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at
org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:669)
at
org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$3.readNextBatch(ArrowConverters.scala:243)
```
It is due to the version mismatch.
```java int messageLength =
MessageSerializer.bytesToInt(buffer.array()); if (messageLength ==
IPC_CONTINUATION_TOKEN) {
buffer.clear();
// ARROW-6313, if the first 4 bytes are continuation message, read the
next 4 for the length
if (in.readFully(buffer) == 4) {
   messageLength = MessageSerializer.bytesToInt(buffer.array());
}
}
// Length of 0 indicates end of stream if (messageLength != 0) {
// Read the message into the buffer.
ByteBuffer messageBuffer = ByteBuffer.allocate(messageLength);
```
After upgrading this to 0.15, we are hitting ARROW-5505. This PR
upgrades Arrow version in AppVeyor and fix the issue.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the AppVeyor.
This PR passed here.
-
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/28909044
``` SparkSQL Arrow optimization: Spark package found in SPARK_HOME:
C:\projects\spark\bin\..
................
```
Closes #26555 from dongjoon-hyun/SPARK-R-TEST.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedR/pkg/R/SQLContext.R (diff)
The file was modifiedR/pkg/R/deserialize.R (diff)
The file was modifiedappveyor.yml (diff)
Commit 388a737b985b048ed4330764f1788160c278d929 by dhyun
[SPARK-29858][SQL] ALTER DATABASE (SET DBPROPERTIES) should look up
catalog like v2 commands
### What changes were proposed in this pull request? Add
AlterNamespaceSetPropertiesStatement, AlterNamespaceSetProperties and
AlterNamespaceSetPropertiesExec to make ALTER DATABASE (SET
DBPROPERTIES) command look up catalog like v2 commands.
### Why are the changes needed? It's important to make all the commands
have the same catalog/namespace resolution behavior, to avoid confusing
end-users.
### Does this PR introduce any user-facing change? Yes, add "ALTER
NAMESPACE ... SET (DBPROPERTIES | PROPERTIES) ..." whose function is
same as "ALTER DATABASE ... SET DBPROPERTIES ..." and "ALTER SCHEMA ...
SET DBPROPERTIES ...".
### How was this patch tested? New unit test
Closes #26551 from fuwhu/SPARK-29858.
Authored-by: fuwhu <bestwwg@163.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
The file was addedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterNamespaceSetPropertiesExec.scala
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
Commit a9959be2bcf55f753ef48e0a4daea6abe0d63c1c by sean.owen
[SPARK-29456][WEBUI] Improve tooltip for Session Statistics Table column
in JDBC/ODBC Server Tab
What changes were proposed in this pull request? Some of the columns of
JDBC/ODBC tab  Session info in Web UI are hard to understand.
Add tool tip for Start time, finish time , Duration and Total Execution
![Screenshot from 2019-10-16
12-33-17](https://user-images.githubusercontent.com/51401130/66901981-76d68980-f01d-11e9-9686-e20346a38c25.png)
Why are the changes needed? To improve the understanding of the WebUI
Does this PR introduce any user-facing change? No
How was this patch tested? manual test
Closes #26138 from PavithraRamachandran/JDBC_tooltip.
Authored-by: Pavithra Ramachandran <pavi.rams@gmail.com> Signed-off-by:
Sean Owen <sean.owen@databricks.com>
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ToolTips.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala (diff)
Commit e1fc38b3e409e8a2c65d0cc1fc2ec63da527bbc6 by dhyun
[SPARK-29932][R][TESTS] lint-r should do non-zero exit in case of errors
### What changes were proposed in this pull request?
This PR aims to make `lint-r` exits with non-zero in case of errors.
Please note that `lint-r` works correctly when everything are installed
correctly.
### Why are the changes needed?
There are two cases which hide errors from
Jenkins/AppVeyor/GitHubAction. 1. `lint-r` exits with zero if there is
no R installation.
```bash
$ dev/lint-r dev/lint-r: line 25: type: Rscript: not found ERROR: You
should install R
$ echo $? 0
```
2. `lint-r` exits with zero if we didn't do `R/install-dev.sh`.
```bash
$ dev/lint-r Error: You should install SparkR in a local directory with
`R/install-dev.sh`. In addition: Warning message: In library(SparkR,
lib.loc = LOCAL_LIB_LOC, logical.return = TRUE) :
no library trees found in 'lib.loc' Execution halted lintr checks
passed.        // <=== Please note here
$ echo $? 0
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Manually check the above two cases.
Closes #26561 from dongjoon-hyun/SPARK-29932.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifieddev/lint-r (diff)
Commit 5eb8973f871fef557fb4ca3f494406ed676a431a by dhyun
[SPARK-29930][SQL] Remove SQL configs declared to be removed in Spark
3.0
### What changes were proposed in this pull request? In the PR, I
propose to remove the following SQL configs: 1.
`spark.sql.fromJsonForceNullableSchema` 2.
`spark.sql.legacy.compareDateTimestampInTimestamp` 3.
`spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation`
that are declared to be removed in Spark 3.0
### Why are the changes needed? To make code cleaner and improve
maintainability.
### Does this PR introduce any user-facing change? Yes
### How was this patch tested? By `TypeCoercionSuite`,
`JsonExpressionsSuite` and `DDLSuite`.
Closes #26559 from MaxGekk/remove-sql-configs.
Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala (diff)
Commit c5f644c6ebb871d76f6b6b12b341ba761427492c by ruifengz
[SPARK-16872][ML][PYSPARK] Impl Gaussian Naive Bayes Classifier
### What changes were proposed in this pull request? support `modelType`
`gaussian`
### Why are the changes needed? current modelTypes do not support
continuous data
### Does this PR introduce any user-facing change? yes, add a
`modelType` option
### How was this patch tested? existing testsuites and added ones
Closes #26413 from zhengruifeng/gnb.
Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by:
zhengruifeng <ruifengz@foxmail.com>
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala (diff)
The file was modifiedpython/pyspark/ml/classification.py (diff)
The file was modifieddocs/ml-classification-regression.md (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala (diff)
The file was modifiedproject/MimaExcludes.scala (diff)
Commit d83cacfcf5510e0b466bbe17459811bb42d72250 by wenchen
[SPARK-29907][SQL] Move DELETE/UPDATE/MERGE relative rules to
dmlStatementNoWith to support cte
### What changes were proposed in this pull request?
SPARK-27444 introduced `dmlStatementNoWith` so that any dml that needs
cte support can leverage it. It be better if we move DELETE/UPDATE/MERGE
rules to `dmlStatementNoWith`.
### Why are the changes needed? Wit this change, we can support syntax
like "With t AS (SELECT) DELETE FROM xxx", and so as UPDATE/MERGE.
### Does this PR introduce any user-facing change? No.
### How was this patch tested?
New cases added.
Closes #26536 from xianyinxin/SPARK-29907.
Authored-by: xy_xin <xianyin.xxy@alibaba-inc.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
Commit f280c6aa54d80251da66ab370d32a7d93b01f225 by gurwls223
[SPARK-29378][R][FOLLOW-UP] Remove manual installation of Arrow
dependencies in AppVeyor build
### What changes were proposed in this pull request?
This PR remove manual installation of Arrow dependencies in AppVeyor
build
### Why are the changes needed?
It's unnecessary. See
https://github.com/apache/spark/pull/26555#discussion_r347178368
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
AppVeyor will test.
Closes #26566 from HyukjinKwon/SPARK-29378.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedappveyor.yml (diff)
Commit 42f8f79ff0311f7de629177b8609b2cbc73ad1c4 by dhyun
[SPARK-29936][R] Fix SparkR lint errors and add lint-r GitHub Action
### What changes were proposed in this pull request?
This PR fixes SparkR lint errors and adds `lint-r` GitHub Action to
protect the branch.
### Why are the changes needed?
It turns out that we currently don't run it. It's recovered yesterday.
However, after that, our Jenkins linter jobs (`master`/`branch-2.4`) has
been broken on `lint-r` tasks.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action on this PR in addition to Jenkins R and AppVeyor
R.
Closes #26564 from dongjoon-hyun/SPARK-29936.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedR/pkg/inst/worker/worker.R (diff)
The file was modifieddev/lint-r.R (diff)
The file was modified.github/workflows/master.yml (diff)
The file was modifiedR/pkg/R/SQLContext.R (diff)
The file was modifiedR/pkg/tests/fulltests/test_sparkSQL.R (diff)
The file was modifiedR/pkg/R/context.R (diff)
The file was modifiedR/pkg/.lintr (diff)
The file was modifiedR/pkg/R/utils.R (diff)
The file was modifiedR/pkg/R/DataFrame.R (diff)
The file was modifiedR/pkg/R/group.R (diff)
Commit ee3bd6d76887ccc4961fd520c5d03f7edd3742ac by d_tsai
[SPARK-25694][SQL] Add a config for `URL.setURLStreamHandlerFactory`
### What changes were proposed in this pull request?
Add a property `spark.fsUrlStreamHandlerFactory.enabled` to allow users
turn off the default registration of
`org.apache.hadoop.fs.FsUrlStreamHandlerFactory`
### Why are the changes needed?
This [SPARK-25694](https://issues.apache.org/jira/browse/SPARK-25694) is
a long-standing issue. Originally, [[SPARK-12868][SQL] Allow adding jars
from hdfs](https://github.com/apache/spark/pull/17342 ) added this for
better Hive support. However, this have a side-effect when the users use
Apache Spark without `-Phive`. This causes exceptions when the users
tries to use another custom factories or 3rd party library (trying to
set this). This configuration will unblock those non-hive users.
### Does this PR introduce any user-facing change?
Yes. This provides a new user-configurable property. By default, the
behavior is unchanged.
### How was this patch tested?
Manual testing.
**BEFORE**
```
$ build/sbt package
$ bin/spark-shell scala> sql("show tables").show
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+
scala> java.net.URL.setURLStreamHandlerFactory(new
org.apache.hadoop.fs.FsUrlStreamHandlerFactory()) java.lang.Error:
factory already defined
at java.net.URL.setURLStreamHandlerFactory(URL.java:1134)
... 47 elided
```
**AFTER**
```
$ build/sbt package
$ bin/spark-shell --conf
spark.sql.defaultUrlStreamHandlerFactory.enabled=false scala> sql("show
tables").show
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+
scala> java.net.URL.setURLStreamHandlerFactory(new
org.apache.hadoop.fs.FsUrlStreamHandlerFactory())
```
Closes #26530 from jiangzho/master.
Lead-authored-by: Zhou Jiang <zhou_jiang@apple.com> Co-authored-by:
Dongjoon Hyun <dhyun@apple.com> Co-authored-by: zhou-jiang
<zhou_jiang@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala (diff)
The file was addedsql/core/src/main/scala/org/apache/spark/sql/internal/config/package.scala
Commit 73912379d06e49e29aa8192ac4fb746d73c97fc5 by gurwls223
[SPARK-29020][SQL] Improving array_sort behaviour
### What changes were proposed in this pull request? I've noticed that
there are two functions to sort arrays sort_array and array_sort.
sort_array is from 1.5.0 and it has the possibility of ordering both
ascending and descending
array_sort is from 2.4.0 and it only has the possibility of ordering in
ascending.
Basically I just added the possibility of ordering either ascending or
descending using array_sort.
I think it would be good to have unified behaviours and not having to
user sort_array when you want to order in descending order. Imagine that
you are new to spark, I'd like to be able to sort array using the newest
spark functions.
### Why are the changes needed? Basically to be able to sort the array
in descending order using *array_sort* instead of using *sort_array*
from 1.5.0
### Does this PR introduce any user-facing change? Yes, now you are able
to sort the array in descending order. Note that it has the same
behaviour with nulls than sort_array
### How was this patch tested? Test's added
This is the link to the
[jira](https://issues.apache.org/jira/browse/SPARK-29020)
Closes #25728 from Gschiavon/improving-array-sort.
Lead-authored-by: gschiavon <german.schiavon@lifullconnect.com>
Co-authored-by: Takuya UESHIN <ueshin@databricks.com> Co-authored-by:
gschiavon <Gschiavon@users.noreply.github.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/functions.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HigherOrderFunctionsSuite.scala (diff)
Commit 5cebe587c7132fa6ea502084d45e0d8b203481b8 by wenchen
[SPARK-29783][SQL] Support SQL Standard/ISO_8601 output style for
interval type
### What changes were proposed in this pull request?
Add 3 interval output types which are named as `SQL_STANDARD`,
`ISO_8601`, `MULTI_UNITS`. And we add a new conf
`spark.sql.dialect.intervalOutputStyle` for this. The `MULTI_UNITS`
style displays the interval values in the former behavior and it is the
default. The newly added `SQL_STANDARD`, `ISO_8601` styles can be found
in the following table.
Style | conf | Year-Month Interval | Day-Time Interval | Mixed Interval
-- | -- | -- | -- | -- Format With Time Unit Designators | MULTI_UNITS |
1 year 2 mons | 1 days 2 hours 3 minutes 4.123456 seconds | interval 1
days 2 hours 3 minutes 4.123456 seconds SQL STANDARD  | SQL_STANDARD |
1-2 | 3 4:05:06 | -1-2 3 -4:05:06 ISO8601 Basic Format| ISO_8601| P1Y2M|
P3DT4H5M6S|P-1Y-2M3D-4H-5M-6S
### Why are the changes needed?
for ANSI SQL support
### Does this PR introduce any user-facing change?
yes,interval out now has 3 output styles
### How was this patch tested?
add new unit tests
cc cloud-fan maropu MaxGekk HyukjinKwon thanks.
Closes #26418 from yaooqinn/SPARK-29783.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
The file was modifiedcommon/unsafe/src/test/java/org/apache/spark/unsafe/types/CalendarIntervalSuite.java (diff)
The file was addedsql/core/src/test/resources/sql-tests/results/interval-display.sql.out
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/inputs/interval-display-iso_8601.sql
The file was addedsql/core/src/test/resources/sql-tests/results/interval-display-sql_standard.sql.out
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/inputs/interval-display.sql
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/interval.sql.out (diff)
The file was addedsql/core/src/test/resources/sql-tests/inputs/interval-display-sql_standard.sql
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/interval.sql (diff)
The file was modifiedcommon/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java (diff)
The file was addedsql/core/src/test/resources/sql-tests/results/interval-display-iso_8601.sql.out
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala (diff)
Commit 50f6d930dafc508d1e5162dd0dd580d1f91bfbd1 by wenchen
[SPARK-29870][SQL] Unify the logic of multi-units interval string to
CalendarInterval
### What changes were proposed in this pull request?
We now have two different implementation for multi-units interval
strings to CalendarInterval type values.
One is used to covert interval string literals to CalendarInterval. This
approach will re-delegate the interval string to spark parser which
handles the string as a `singleInterval` -> `multiUnitsInterval` ->
eventually call `IntervalUtils.fromUnitStrings`
The other is used in `Cast`, which eventually calls
`IntervalUtils.stringToInterval`. This approach is ~10 times faster than
the other.
We should unify these two for better performance and simple logic. this
pr uses the 2nd approach.
### Why are the changes needed?
We should unify these two for better performance and simple logic.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
we shall not fail on existing uts
Closes #26491 from yaooqinn/SPARK-29870.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/GroupStateImpl.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/IntervalExpressionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimeWindow.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala (diff)
Commit c32e2286898fe8660c7deed9303f1c5c15b60757 by wenchen
[SPARK-29859][SQL] ALTER DATABASE (SET LOCATION) should look up catalog
like v2 commands
### What changes were proposed in this pull request? Add
AlterNamespaceSetLocationStatement, AlterNamespaceSetLocation,
AlterNamespaceSetLocationExec to make ALTER DATABASE (SET LOCATION) look
up catalog like v2 commands. And also refine the code of
AlterNamespaceSetProperties, AlterNamespaceSetPropertiesExec,
DescribeNamespace, DescribeNamespaceExec to use SupportsNamespaces
instead of CatalogPlugin for catalog parameter.
### Why are the changes needed? It's important to make all the commands
have the same catalog/namespace resolution behavior, to avoid confusing
end-users.
### Does this PR introduce any user-facing change? Yes, add "ALTER
NAMESPACE ... SET LOCATION" whose function is same as "ALTER DATABASE
... SET LOCATION" and "ALTER SCHEMA ... SET LOCATION".
### How was this patch tested? New unit tests
Closes #26562 from fuwhu/SPARK-29859.
Authored-by: fuwhu <bestwwg@163.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AlterNamespaceSetPropertiesExec.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeNamespaceExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
Commit ae6b711b262de21b701434b8e4d031ca265cc376 by wenchen
[SPARK-29941][SQL] Add ansi type aliases for char and decimal
### What changes were proposed in this pull request?
Checked with SQL Standard and PostgreSQL
> CHAR is equivalent to CHARACTER. DEC is equivalent to DECIMAL. INT is
equivalent to INTEGER. VARCHAR is equivalent to CHARACTER VARYING. ...
```sql postgres=# select dec '1.0'; numeric
--------- 1.0
(1 row)
postgres=# select CHARACTER '. second';
bpchar
----------
. second
(1 row)
postgres=# select CHAR '. second';
bpchar
----------
. second
(1 row)
```
### Why are the changes needed?
For better ansi support
### Does this PR introduce any user-facing change?
yes, we add character as char and dec as decimal
### How was this patch tested?
add ut
Closes #26574 from yaooqinn/SPARK-29941.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
Commit ea010a2bc226197b3161a5c4db9cfe5a4444c7a4 by wenchen
[SPARK-29873][SQL][TEST][FOLLOWUP] set operations should not escape when
regen golden file with --SET --import both specified
### What changes were proposed in this pull request?
When regenerating golden files, the set operations via `--SET` will not
be done, but those with --import should be exceptions because we need
the set command.
### Why are the changes needed?
fix test tool.
### Does this PR introduce any user-facing change?
### How was this patch tested?
add ut, but I'm not sure we need these tests for tests itself. cc maropu
cloud-fan
Closes #26557 from yaooqinn/SPARK-29873.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/interval-display-iso_8601.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/ansi/higher-order-functions.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/ansi/interval.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/interval-display-sql_standard.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/ansi/literals.sql (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
Commit 9514b822a70d77a6298ece48e6c053200360302c by gurwls223
[SPARK-29777][SPARKR] SparkR::cleanClosure aggressively removes a
function required by user function
### What changes were proposed in this pull request? The implementation
for walking through the user function AST and picking referenced
variables and functions, had an optimization to skip a branch if it had
already seen it. This runs into an interesting problem in the following
example
``` df <- createDataFrame(data.frame(x=1)) f1 <- function(x) x + 1 f2 <-
function(x) f1(x) + 2 dapplyCollect(df, function(x) { f1(x); f2(x) })
``` Results in error:
``` org.apache.spark.SparkException: R computation failed with
Error in f1(x) : could not find function "f1" Calls: compute ->
computeFunc -> f2
```
### Why are the changes needed? Bug fix
### Does this PR introduce any user-facing change? No
### How was this patch tested? Unit tests in `test_utils.R`
Closes #26429 from falaki/SPARK-29777.
Authored-by: Hossein <hossein@databricks.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedR/pkg/R/utils.R (diff)
The file was modifiedR/pkg/tests/fulltests/test_utils.R (diff)
Commit 8469614c0513fbed87977d4e741649db3fdd8add by gurwls223
[SPARK-25694][SQL][FOLLOW-UP] Move
'spark.sql.defaultUrlStreamHandlerFactory.enabled' into
StaticSQLConf.scala
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/26530 and
proposes to move the configuration
`spark.sql.defaultUrlStreamHandlerFactory.enabled` to
`StaticSQLConf.scala` for consistency.
### Why are the changes needed?
To put the similar configurations together and for readability.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Manually tested as described in
https://github.com/apache/spark/pull/26530.
Closes #26570 from HyukjinKwon/SPARK-25694.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala (diff)
The file was removedsql/core/src/main/scala/org/apache/spark/sql/internal/config/package.scala
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala (diff)
Commit 882f54b0a323fb5cd827d600b3c3332e1fcdf65a by gurwls223
[SPARK-29870][SQL][FOLLOW-UP] Keep CalendarInterval's toString
### What changes were proposed in this pull request?
This is a followup of https://github.com/apache/spark/pull/26418. This
PR removed `CalendarInterval`'s `toString` with an unfinished changes.
### Why are the changes needed?
1. Ideally we should make each PR isolated and separate targeting one
issue without touching unrelated codes.
2. There are some other places where the string formats were exposed to
users. For example:
    ```scala
   scala> sql("select interval 1 days as
a").selectExpr("to_csv(struct(a))").show()
   ```
   ```
   +--------------------------+
   |to_csv(named_struct(a, a))|
   +--------------------------+
   |      "CalendarInterval...|
   +--------------------------+
   ```
3.  Such fixes:
    ```diff
    private def writeMapData(
       map: MapData, mapType: MapType, fieldWriter: ValueWriter): Unit =
{
     val keyArray = map.keyArray()
   + val keyString = mapType.keyType match {
   +   case CalendarIntervalType =>
   +    (i: Int) =>
IntervalUtils.toMultiUnitsString(keyArray.getInterval(i))
   +   case _ => (i: Int) => keyArray.get(i, mapType.keyType).toString
   + }
   ```
    can cause performance regression due to type dispatch for each map.
### Does this PR introduce any user-facing change?
Yes, see 2. case above.
### How was this patch tested?
Manually tested.
Closes #26572 from HyukjinKwon/SPARK-29783.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
The file was modifiedcommon/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala (diff)
Commit 28a502c6e92ce44601e798d203364422063c1e07 by dhyun
[SPARK-28527][FOLLOW-UP][SQL][TEST] Add guides for
ThriftServerQueryTestSuite
### What changes were proposed in this pull request? This PR add guides
for `ThriftServerQueryTestSuite`.
### Why are the changes needed? Add guides
### Does this PR introduce any user-facing change? No.
### How was this patch tested? N/A
Closes #26587 from wangyum/SPARK-28527-FOLLOW-UP.
Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
Commit a834dba120e3569e44c5e4b9f8db9c6eef58161b by dhyun
Revert "[SPARK-29644][SQL] Corrected ShortType and ByteType mapping to
SmallInt and TinyInt in JDBCUtils
This reverts commit f7e53865 i.e PR #26301 from master
Closes #26583 from shivsood/revert_29644_master.
Authored-by: shivsood <shivsood@microsoft.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedexternal/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala (diff)
The file was modifiedexternal/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala (diff)
Commit 3d45779b687e3f8008044aa251d57024f2a31807 by wenchen
[SPARK-29728][SQL] Datasource V2: Support ALTER TABLE RENAME TO
### What changes were proposed in this pull request?
This PR adds `ALTER TABLE a.b.c RENAME TO x.y.x` support for V2
catalogs.
### Why are the changes needed?
The current implementation doesn't support this command V2 catalogs.
### Does this PR introduce any user-facing change?
Yes, now the renaming table works for v2 catalogs:
``` scala> spark.sql("SHOW TABLES IN testcat.ns1.ns2").show
+---------+---------+
|namespace|tableName|
+---------+---------+
|  ns1.ns2|      old|
+---------+---------+
scala> spark.sql("ALTER TABLE testcat.ns1.ns2.old RENAME TO
testcat.ns1.ns2.new").show
scala> spark.sql("SHOW TABLES IN testcat.ns1.ns2").show
+---------+---------+
|namespace|tableName|
+---------+---------+
|  ns1.ns2|      new|
+---------+---------+
```
### How was this patch tested?
Added unit tests.
Closes #26539 from imback82/rename_table.
Authored-by: Terry Kim <yuminkim@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala (diff)
The file was addedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RenameTableExec.scala
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
Commit 2e71a6e7ba20743e22a234f5209c1b120f2a7948 by xingbo.jiang
[SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM
exception
### What changes were proposed in this pull request?
When a task fails with OOM exception, the `UnsafeInMemorySorter.array`
could be `null`. In the meanwhile, the `cleanupResources()` on task
completion would call `UnsafeInMemorySorter.getMemoryUsage` in turn, and
that lead to another NPE thrown.
### Why are the changes needed?
Check if `array` is null in `UnsafeInMemorySorter.getMemoryUsage` and it
should help to avoid NPE.
### Does this PR introduce any user-facing change? No.
### How was this patch tested? It was tested manually.
Closes #26349 from ayudovin/fix-npe-in-listener.
Authored-by: yudovin <artsiom.yudovin@profitero.com> Signed-off-by:
Xingbo Jiang <xingbo.jiang@databricks.com>
The file was modifiedcore/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java (diff)
Commit 5ac37a82656f440e8f58564974668ed9e0ca6b72 by wenchen
[SPARK-29869][SQL] improve error message in
HiveMetastoreCatalog#convertToLogicalRelation
### What changes were proposed in this pull request? In our production,
HiveMetastoreCatalog#convertToLogicalRelation throws AssertError
occasionally:
```sql scala> spark.table("hive_table").show java.lang.AssertionError:
assertion failed
at scala.Predef$.assert(Predef.scala:208)
at
org.apache.spark.sql.hive.HiveMetastoreCatalog.convertToLogicalRelation(HiveMetastoreCatalog.scala:261)
at
org.apache.spark.sql.hive.HiveMetastoreCatalog.convert(HiveMetastoreCatalog.scala:137)
at
org.apache.spark.sql.hive.RelationConversions$$anonfun$apply$4.applyOrElse(HiveStrategies.scala:220)
at
org.apache.spark.sql.hive.RelationConversions$$anonfun$apply$4.applyOrElse(HiveStrategies.scala:207)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$2(AnalysisHelper.scala:108)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:108)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$4(AnalysisHelper.scala:113)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:376)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:214)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:374)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:327)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:113)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:73)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:72)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:29)
at
org.apache.spark.sql.hive.RelationConversions.apply(HiveStrategies.scala:207)
at
org.apache.spark.sql.hive.RelationConversions.apply(HiveStrategies.scala:191)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:130)
at
scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
at
scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:49)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:127)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:119)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:119)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:168)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:162)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:122)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:98)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:98)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:146)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:145)
at
org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:66)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:63)
at
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:63)
at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:55)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:86)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:585)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:581)
... 47 elided
```` Most of cases occurred in reading a table which created by an old
Spark version. After recreated the table, the issue will be gone.
After deep dive, the root cause is this external table is a
non-partitioned table but the `LOCATION` set to a partitioned path
{{/tablename/dt=yyyymmdd}}. The partitionSpec is inferred.
### Why are the changes needed? Above error message is very confused. We
need more details about assert failure information.
This issue caused by `PartitioningAwareFileIndex#inferPartitioning()`.
For non-HiveMetastore Spark, it's useful. But for Hive table, it
shouldn't infer partition if Hive tell us it's a non partitioned table.
(new added)
### Does this PR introduce any user-facing change? No
### How was this patch tested? Add UT.
Closes #26499 from LantaoJin/SPARK-29869.
Authored-by: LantaoJin <jinlantao@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala (diff)
Commit 16134d6d0f349a6880c90770ab7e393a266b978d by wenchen
[SPARK-29948][SQL] make the default alias consistent between date,
timestamp and interval
### What changes were proposed in this pull request?
Update `Literal.sql` to make date, timestamp and interval consistent.
They should all use the `TYPE 'value'` format.
### Why are the changes needed?
Make the default alias consistent. For example, without this patch we
will see
``` scala> sql("select interval '1 day', date '2000-10-10'").show
+------+-----------------+
|1 days|DATE '2000-10-10'|
+------+-----------------+
|1 days|       2000-10-10|
+------+-----------------+
```
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
existing tests
Closes #26579 from cloud-fan/sql.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/literals.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/literals.sql.out (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionSQLBuilderSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/cast.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/window.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-joins.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/datetime.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display-sql_standard.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/typeCoercion/native/dateTimeOperations.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/udf/udf-window.sql.out (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/array.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display-iso_8601.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala (diff)
Commit ffc97530371433bc0221e06d8c1d11af8d92bd94 by wenchen
[SPARK-29918][SQL] RecordBinaryComparator should check endianness when
compared by long
### What changes were proposed in this pull request? This PR try to make
sure the comparison results of  `compared by 8 bytes at a time` and
`compared by bytes wise` in RecordBinaryComparator is *consistent*, by
reverse long bytes if it is little-endian and using
Long.compareUnsigned.
### Why are the changes needed? If the architecture supports unaligned
or the offset is 8 bytes aligned, `RecordBinaryComparator` compare 8
bytes at a time by reading 8 bytes as a long.  Related code is
```
   if (Platform.unaligned() || (((leftOff + i) % 8 == 0) && ((rightOff +
i) % 8 == 0))) {
     while (i <= leftLen - 8) {
       final long v1 = Platform.getLong(leftObj, leftOff + i);
       final long v2 = Platform.getLong(rightObj, rightOff + i);
       if (v1 != v2) {
         return v1 > v2 ? 1 : -1;
       }
       i += 8;
     }
   }
```
Otherwise, it will compare bytes by bytes.  Related code is
```
   while (i < leftLen) {
     final int v1 = Platform.getByte(leftObj, leftOff + i) & 0xff;
     final int v2 = Platform.getByte(rightObj, rightOff + i) & 0xff;
     if (v1 != v2) {
       return v1 > v2 ? 1 : -1;
     }
     i += 1;
   }
```
However, on little-endian machine,  the result of *compared by a long
value* and *compared bytes by bytes* maybe different.
For two same records, its offsets may vary in the first run and second
run, which will lead to compare them using long comparison or
byte-by-byte comparison, the result maybe different.
### Does this PR introduce any user-facing change? No
### How was this patch tested? Add new test cases in
RecordBinaryComparatorSuite
Closes #26548 from WangGuangxin/binary_comparator.
Authored-by: wangguangxin.cn <wangguangxin.cn@bytedance.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/main/java/org/apache/spark/sql/execution/RecordBinaryComparator.java (diff)
The file was modifiedsql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java (diff)
Commit a8d98833b88e7a366734c18acec36236357a41e4 by wenchen
[SPARK-29893] improve the local shuffle reader performance by changing
the reading task number from 1 to multi
### What changes were proposed in this pull request? This PR update the
local reader task number from 1 to multi `partitionStartIndices.length`.
### Why are the changes needed? Improve the performance of local shuffle
reader.
### Does this PR introduce any user-facing change? No
### How was this patch tested? Existing UTs
Closes #26516 from JkSelf/improveLocalShuffleReader.
Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedcore/src/main/scala/org/apache/spark/MapOutputTracker.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LocalShuffledRowRDD.scala (diff)
Commit 79ed4ae2dbdbc73c4225fbabeac5a1ddb10c65d1 by wenchen
[SPARK-29926][SQL] Fix weird interval string whose value is only a
dangling decimal point
### What changes were proposed in this pull request?
Currently, we support to parse '1. second' to 1s or even '. second' to
0s.
```sql
-- !query 118 select interval '1. seconds'
-- !query 118 schema struct<1 seconds:interval>
-- !query 118 output 1 seconds
-- !query 119 select interval '. seconds'
-- !query 119 schema struct<0 seconds:interval>
-- !query 119 output 0 seconds
```
```sql postgres=# select interval '1. second'; ERROR:  invalid input
syntax for type interval: "1. second" LINE 1: select interval '1.
second';
postgres=# select interval '. second'; ERROR:  invalid input syntax for
type interval: ". second" LINE 1: select interval '. second';
``` We fix this by fixing the new interval parser's
VALUE_FRACTIONAL_PART state
With further digging, we found that 1. is valid in python, r, scala, and
presto and so on... so this PR ONLY forbid the invalid interval value in
the form of  '. seconds'.
### Why are the changes needed?
bug fix
### Does this PR introduce any user-facing change?
yes, now we treat '. second' .... as invalid intervals
### How was this patch tested?
add ut
Closes #26573 from yaooqinn/SPARK-29926.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala (diff)
Commit 6fb8b8606544f26dc2d9719a2d009eb5aea65ba2 by wenchen
[SPARK-29913][SQL] Improve Exception in postgreCastToBoolean
### What changes were proposed in this pull request? Exception
improvement.
### Why are the changes needed? After selecting pgSQL dialect, queries
which are failing because of wrong syntax will give long exception stack
trace. For example,
`explain select cast ("abc" as boolean);`
Current output:
> ERROR SparkSQLDriver: Failed in [explain select cast ("abc" as
boolean)]
> java.lang.IllegalArgumentException: invalid input syntax for type
boolean: abc
> at
org.apache.spark.sql.catalyst.expressions.postgreSQL.PostgreCastToBoolean.$anonfun$castToBoolean$2(PostgreCastToBoolean.scala:51)
> at
org.apache.spark.sql.catalyst.expressions.CastBase.buildCast(Cast.scala:277)
> at
org.apache.spark.sql.catalyst.expressions.postgreSQL.PostgreCastToBoolean.$anonfun$castToBoolean$1(PostgreCastToBoolean.scala:44)
> at
org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:773)
> at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:460)
> at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1$$anonfun$applyOrElse$1.applyOrElse(expressions.scala:52)
> at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1$$anonfun$applyOrElse$1.applyOrElse(expressions.scala:45)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:286)
> at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:286)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:291)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:376)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:214)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:374)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:327)
> at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:291)
> at org.apache.spark.sql.catalyst.plans.QueryPlan.
>       .
>       .
>       .
### Does this PR introduce any user-facing change? Yes. After this PR,
output for above query will be:
> == Physical Plan ==
> org.apache.spark.sql.AnalysisException: invalid input syntax for type
boolean: abc;
>
> Time taken: 0.044 seconds, Fetched 1 row(s)
> 19/11/15 15:38:57 INFO SparkSQLCLIDriver: Time taken: 0.044 seconds,
Fetched 1 row(s)
### How was this patch tested? Updated existing test cases.
Closes #26546 from jobitmathew/pgsqlexception.
Authored-by: Jobit Mathew <jobit.mathew@huawei.com> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala (diff)
Commit 3d2a6f464f98c04f011509733a15972fd2b6701d by gatorsmile
[SPARK-29906][SQL] AQE should not introduce extra shuffle for outermost
limit
### What changes were proposed in this pull request?
`AdaptiveSparkPlanExec` should forward `executeCollect` and
`executeTake` to the underlying physical plan.
### Why are the changes needed?
some physical plan has optimization in `executeCollect` and
`executeTake`. For example, `CollectLimitExec` won't do shuffle for
outermost limit.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
a new test
This closes #26560
Closes #26576 from cloud-fan/aqe.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Xiao Li
<gatorsmile@gmail.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala (diff)
Commit e804ed5e330c7dc6cd46812b520dd6b610a584b6 by cutlerb
[SPARK-29691][ML][PYTHON] ensure Param objects are valid in fit,
transform
modify Param._copyValues to check valid Param objects supplied as extra
### What changes were proposed in this pull request?
Estimator.fit() and Model.transform() accept a dictionary of extra
parameters whose values are used to overwrite those supplied at
initialization or by default.  Additionally, the
ParamGridBuilder.addGrid accepts a parameter and list of values. The
keys are presumed to be valid Param objects. This change adds a check
that only Param objects are supplied as keys.
### Why are the changes needed?
Param objects are created by and bound to an instance of Params
(Estimator, Model, or Transformer). They may be obtained from their
parent as attributes, or by name through getParam.
The documentation does not state that keys must be valid Param objects,
nor describe how one may be obtained. The current behavior is to
silently ignore keys which are not valid Param objects.
### Does this PR introduce any user-facing change?
If the user does not pass in a Param object as required for keys in
`extra` for Estimator.fit() and Model.transform(), and `param` for
ParamGridBuilder.addGrid, an error will be raised indicating it is an
invalid object.
### How was this patch tested?
Added method test_copy_param_extras_check to test_param.py.   Tested
with Python 3.7
Closes #26527 from JohnHBauer/paramExtra.
Authored-by: John Bauer <john.h.bauer@gmail.com> Signed-off-by: Bryan
Cutler <cutlerb@gmail.com>
The file was modifiedpython/pyspark/ml/param/__init__.py (diff)
The file was modifiedpython/pyspark/ml/tuning.py (diff)
The file was modifiedpython/pyspark/ml/tests/test_param.py (diff)
The file was modifiedpython/pyspark/ml/tests/test_tuning.py (diff)
Commit e753aa30e659706c3fa3414bf38566a79e0af8d6 by dhyun
[SPARK-29964][BUILD] lintr github workflows failed due to buggy GnuPG
### What changes were proposed in this pull request?
Linter (R) github workflows failed sometimes like:
https://github.com/apache/spark/pull/26509/checks?check_run_id=310718016
Failed message:
``` Executing: /tmp/apt-key-gpghome.8r74rQNEjj/gpg.1.sh --keyserver
keyserver.ubuntu.com --recv-keys
E298A3A825C0D65DFD57CBB651716619E084DAB9 gpg: connecting dirmngr at
'/tmp/apt-key-gpghome.8r74rQNEjj/S.dirmngr' failed: IPC connect call
failed gpg: keyserver receive failed: No dirmngr
##[error]Process completed with exit code 2.
```
It is due to a buggy GnuPG. Context:
https://github.com/sbt/website/pull/825
https://github.com/sbt/sbt/issues/4261
https://github.com/microsoft/WSL/issues/3286
### Why are the changes needed?
Make lint-r github workflows work.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Pass github workflows.
Closes #26602 from viirya/SPARK-29964.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modified.github/workflows/master.yml (diff)
Commit 40b8a08b8b7f71685c5703605fd8a8b101de32f7 by gurwls223
[SPARK-29963][SQL][TESTS] Check formatting timestamps up to microsecond
precision by JSON/CSV datasource
### What changes were proposed in this pull request? In the PR, I
propose to add tests from the commit
https://github.com/apache/spark/commit/47cb1f359af62383e24198dbbaa0b4503348cd04
for Spark 2.4 that check formatting of timestamp strings for various
seconds fractions.
### Why are the changes needed? To make sure that current behavior is
the same as in Spark 2.4
### Does this PR introduce any user-facing change? No
### How was this patch tested? By running `CSVSuite`,
`JsonFunctionsSuite` and `TimestampFormatterSuite`.
Closes #26601 from MaxGekk/format-timestamp-micros-tests.
Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala (diff)
Commit 9e58b10c8e1e066e3daae06f57e48f369a24e9ec by yamamuro
[SPARK-29945][SQL] do not handle negative sign specially in the parser
### What changes were proposed in this pull request?
Remove the special handling of the negative sign in the parser (interval
literal and type constructor)
### Why are the changes needed?
The negative sign is an operator (UnaryMinus). We don't need to handle
it specially, which is kind of doing constant folding at parser side.
### Does this PR introduce any user-facing change?
The error message becomes a little different. Now it reports type
mismatch for the `-` operator.
### How was this patch tested?
existing tests
Closes #26578 from cloud-fan/interval.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Takeshi
Yamamuro <yamamuro@apache.org>
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display-iso_8601.sql.out (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/interval.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display-sql_standard.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/literals.sql (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/interval-display.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/ansi/literals.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/literals.sql.out (diff)
Commit 5a70af7a6c271e12a1f8b508dcede804e30a1b75 by dhyun
[SPARK-29029][SQL] Use AttributeMap in
PhysicalOperation.collectProjectsAndFilters
### What changes were proposed in this pull request?
This PR fixes the issue of substituting aliases while collecting filters
in  `PhysicalOperation.collectProjectsAndFilters`. When the
`AttributeReference` in alias map differs from the `AttributeReference`
in filter condition only in qualifier, it does not substitute alias and
throws exception saying `key videoid#47L not found` in the following
scenario.
```
[1] Project [userid#0]
+- [2] Filter (isnotnull(videoid#47L) && NOT (videoid#47L = 30))
  +- [3] Project [factorial(videoid#1) AS videoid#47L, userid#0]
     +- [4] Filter (isnotnull(avebitrate#2) && (avebitrate#2 < 10))
        +- [5] Relation[userid#0,videoid#1,avebitrate#2]
```
### Why are the changes needed?
We need to use `AttributeMap` where the key is `AttributeReference`'s
`ExprId` instead of `Map[Attribute, Expression]` while collecting and
substituting aliases in `PhysicalOperation.collectProjectsAndFilters`.
### Does this PR introduce any user-facing change? No
### How was this patch tested? New unit tests were added in
`TestPhysicalOperation` which reproduces the bug
Closes #25761 from
nikitagkonda/SPARK-29029-use-attributemap-for-aliasmap-in-physicaloperation.
Authored-by: Nikita Konda <nikita.konda@workday.com> Signed-off-by:
Dongjoon Hyun <dhyun@apple.com>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala (diff)
Commit 0032d85153e34b9ac69598b7dff530094ed0f640 by wenchen
[SPARK-29968][SQL] Remove the Predicate code from SparkPlan
### What changes were proposed in this pull request?
This is to refactor Predicate code; it mainly removed `newPredicate`
from `SparkPlan`. Modifications are listed below;
- Move `Predicate` from
`o.a.s.sqlcatalyst.expressions.codegen.GeneratePredicate.scala` to
`o.a.s.sqlcatalyst.expressions.predicates.scala`
- To resolve the name conflict,  rename
`o.a.s.sqlcatalyst.expressions.codegen.Predicate` to
`o.a.s.sqlcatalyst.expressions.BasePredicate`
- Extend `CodeGeneratorWithInterpretedFallback ` for `BasePredicate`
This comes from the cloud-fan suggestion:
https://github.com/apache/spark/pull/26420#discussion_r348005497
### Why are the changes needed?
For better code/test coverage.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #26604 from maropu/RefactorPredicate.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProductExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/PredicateSuite.scala (diff)
Commit 56a65b971d4f1b43c37a5664a3a4e2e4fcad1c14 by sean.owen
[SPARK-18409][ML] LSH approxNearestNeighbors should use approxQuantile
instead of sort
### What changes were proposed in this pull request?
```LSHModel.approxNearestNeighbors``` sorts the full dataset on the
hashDistance in order to find a threshold. This PR uses approxQuantile
instead.
### Why are the changes needed? To improve performance.
### Does this PR introduce any user-facing change? Yes. Changed
```LSH``` to make it extend ```HasRelativeError```
```LSH``` and ```LSHModel``` have new APIs
```setRelativeError/getRelativeError```
### How was this patch tested? Existing tests. Also added a couple doc
test in python to test newly added ```getRelativeError```
Closes #26415 from huaxingao/spark-18409.
Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala (diff)
Commit b5df40bd87656ee71210db1008ce8b89b5a777d1 by wenchen
[SPARK-29894][SQL][WEBUI] Add Codegen Stage Id to Spark plan graphs in
Web UI SQL Tab
### What changes were proposed in this pull request? The Web UI SQL Tab
provides information on the executed SQL using plan graphs and by
reporting SQL execution plans. Both sources provide useful information.
Physical execution plans report Codegen Stage Ids. This PR adds Codegen
Stage Ids to the plan graphs.
### Why are the changes needed? It is useful to have Codegen Stage Id
information also reported in plan graphs, this allows to more easily
match physical plans and graphs with metrics when troubleshooting SQL
execution. Example snippet to show the proposed change:
![](https://issues.apache.org/jira/secure/attachment/12985837/snippet__plan_graph_with_Codegen_Stage_Id_Annotated.png)
Example of the current state:
![](https://issues.apache.org/jira/secure/attachment/12985838/snippet_plan_graph_before_patch.png)
Physical plan:
![](https://issues.apache.org/jira/secure/attachment/12985932/Physical_plan_Annotated.png)
### Does this PR introduce any user-facing change? This PR adds Codegen
Stage Id information to SQL plan graphs in the Web UI/SQL Tab.
### How was this patch tested? Added a test + manually tested
Closes #26519 from LucaCanali/addCodegenStageIdtoWEBUIGraphs.
Authored-by: Luca Canali <luca.canali@cern.ch> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifieddocs/img/webui-sql-dag.png (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala (diff)
The file was modifieddocs/web-ui.md (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala (diff)
Commit 6eeb131941e3a4afb6f8c55de06e9c26f942e155 by dhyun
[SPARK-28885][SQL][FOLLOW-UP] Re-enable the ported PgSQL regression
tests of SQLQueryTestSuite
### What changes were proposed in this pull request?
SPARK-28885(#26107) has supported the ANSI store assignment rules and
stopped running some ported PgSQL regression tests that violate the
rules. To re-activate these tests, this pr is to modify them for passing
tests with the rules.
### Why are the changes needed?
To make the test coverage better.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #26492 from maropu/SPARK-28885-FOLLOWUP.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Dongjoon Hyun <dhyun@apple.com>
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/timestamp.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/float4.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/int4.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/numeric.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/date.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/float8.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/int2.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/float4.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/numeric.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/date.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/int2.sql (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/float8.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/timestamp.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/int8.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/int8.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/int4.sql.out (diff)
Commit 23b3c4fafdf37a482b3f823a5701d99d96236519 by wenchen
[SPARK-29951][SQL] Make the behavior of Postgre dialect independent of
ansi mode config
### What changes were proposed in this pull request? Fix the
inconsistent behavior of build-in function SQL LEFT/RIGHT.
### Why are the changes needed? As the comment in
https://github.com/apache/spark/pull/26497#discussion_r345708065,
Postgre dialect should not be affected by the ANSI mode config. During
reran the existing tests, only the LEFT/RIGHT build-in SQL function
broke the assumption. We fix this by following
https://www.postgresql.org/docs/12/sql-keywords-appendix.html:
`LEFT/RIGHT reserved (can be function or type)`
### Does this PR introduce any user-facing change? Yes, the Postgre
dialect will not be affected by the ANSI mode config.
### How was this patch tested? Existing UT.
Closes #26584 from xuanyuanking/SPARK-29951.
Authored-by: Yuanjian Li <xyliyuanjian@gmail.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifieddocs/sql-keywords.md (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/postgreSQL/text.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/text.sql (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
Commit 1febd373ea806326d269a60048ee52543a76c918 by sean.owen
[MINOR][TESTS] Replace JVM assert with JUnit Assert in tests
### What changes were proposed in this pull request?
Use JUnit assertions in tests uniformly, not JVM assert() statements.
### Why are the changes needed?
assert() statements do not produce as useful errors when they fail, and,
if they were somehow disabled, would fail to test anything.
### Does this PR introduce any user-facing change?
No. The assertion logic should be identical.
### How was this patch tested?
Existing tests.
Closes #26581 from srowen/assertToJUnit.
Authored-by: Sean Owen <sean.owen@databricks.com> Signed-off-by: Sean
Owen <sean.owen@databricks.com>
The file was modifiedmllib/src/test/java/org/apache/spark/ml/stat/JavaKolmogorovSmirnovTestSuite.java (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/feature/LSHTest.scala (diff)
The file was modifiedcommon/network-shuffle/src/test/java/org/apache/spark/network/shuffle/TestShuffleDataContext.java (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/DistributedSuite.scala (diff)
The file was modifiedresource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/Utils.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/security/HiveHadoopDelegationTokenManagerSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/shuffle/ShuffleDriverComponentsSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/ProbabilisticClassifierSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/columnar/compression/CompressionSchemeBenchmark.scala (diff)
The file was modifiedgraphx/src/test/scala/org/apache/spark/graphx/util/PeriodicGraphCheckpointerSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/MasterFailureTest.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/util/PeriodicRDDCheckpointerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/JoinBenchmark.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/clustering/LDASuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/tree/EnsembleTestHelper.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/WideSchemaBenchmark.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/HashedRelationMetricsBenchmark.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/benchmark/Benchmark.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala (diff)
The file was modifiedsql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/rpc/TestRpcEndpoint.scala (diff)
The file was modifiedexternal/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/java/org/apache/spark/sql/streaming/JavaOutputModeSuite.java (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTable.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/MapStatusesSerDeserBenchmark.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala (diff)
Commit 06e203b85682b63ee250b96520558fc79aae0a17 by sean.owen
[SPARK-29911][SQL] Uncache cached tables when session closed
### What changes were proposed in this pull request? The local temporary
view is session-scoped. Its lifetime is the lifetime of the session that
created it.  But now cache data is cross-session. Its lifetime is the
lifetime of the Spark application. That's will cause the memory leak if
cache a local temporary view in memory when the session closed. In this
PR, we uncache the cached data of local temporary view when session
closed. This PR doesn't impact the cached data of global temp view and
persisted view.
How to reproduce: 1. create a local temporary view v1 2. cache it in
memory 3. close session without drop table v1.
The application will hold the memory forever. In a long running thrift
server scenario. It's worse.
```shell 0: jdbc:hive2://localhost:10000> CACHE TABLE testCacheTable AS
SELECT 1; CACHE TABLE testCacheTable AS SELECT 1;
+---------+--+
| Result  |
+---------+--+
+---------+--+ No rows selected (1.498 seconds) 0:
jdbc:hive2://localhost:10000> !close
!close Closing: 0: jdbc:hive2://localhost:10000 0:
jdbc:hive2://localhost:10000 (closed)> !connect
'jdbc:hive2://localhost:10000'
!connect 'jdbc:hive2://localhost:10000' Connecting to
jdbc:hive2://localhost:10000 Enter username for
jdbc:hive2://localhost:10000: lajin Enter password for
jdbc:hive2://localhost:10000:
*** Connected to: Spark SQL (version 3.0.0-SNAPSHOT) Driver: Hive JDBC
(version 1.2.1.spark2) Transaction isolation:
TRANSACTION_REPEATABLE_READ 1: jdbc:hive2://localhost:10000> select *
from testCacheTable; select * from testCacheTable; Error: Error running
query: org.apache.spark.sql.AnalysisException: Table or view not found:
testCacheTable; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation [testCacheTable] (state=,code=0)
```
<img width="1047" alt="Screen Shot 2019-11-15 at 2 03 49 PM"
src="https://user-images.githubusercontent.com/1853780/68923527-7ca8c180-07b9-11ea-9cc7-74f276c46840.png">
### Why are the changes needed? Resolve memory leak for thrift server
### Does this PR introduce any user-facing change? No
### How was this patch tested? Manual test in UI storage tab And add an
UT
Closes #26543 from LantaoJin/SPARK-29911.
Authored-by: LantaoJin <jinlantao@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
Commit e6b157cf704544a6217a9f9d191f542518581040 by gurwls223
[SPARK-29978][SQL][TESTS] Check `json_tuple` does not truncate results
### What changes were proposed in this pull request? I propose to add a
test from the commit
https://github.com/apache/spark/commit/a9365221133caadffbbbbce1aae1ace799a588a3
for 2.4. I extended the test by a few more lengths of requested field to
cover more code branches in Jackson Core. In particular, [the
optimization](https://github.com/apache/spark/blob/5eb8973f871fef557fb4ca3f494406ed676a431a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L473-L476)
calls Jackson's method
https://github.com/FasterXML/jackson-core/blob/42b8b566845e8f8d77537f51187a439029ed9bff/src/main/java/com/fasterxml/jackson/core/json/UTF8JsonGenerator.java#L742-L746
where the internal buffer size is **8000**. In this way:
- 2000 to check 2000+2000+2000 < 8000
- 2800 from the 2.4 commit. It covers the specific case:
https://github.com/FasterXML/jackson-core/blob/42b8b566845e8f8d77537f51187a439029ed9bff/src/main/java/com/fasterxml/jackson/core/json/UTF8JsonGenerator.java#L746
- 8000-1, 8000, 8000+1 are sizes around the size of the internal buffer
- 65535 to test an outstanding large field.
### Why are the changes needed? To be sure that the current
implementation and future versions of Spark don't have the bug fixed in
2.4.
### Does this PR introduce any user-facing change? No
### How was this patch tested? By running `JsonFunctionsSuite`.
Closes #26613 from MaxGekk/json_tuple-test.
Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala (diff)
Commit 7a706703455d218a874cafe114ac8d720700a802 by gurwls223
[SPARK-29961][SQL] Implement builtin function - typeof
### What changes were proposed in this pull request? Add typeof function
for Spark to get the underlying type of value.
```sql
-- !query 0 select typeof(1)
-- !query 0 schema struct<typeof(1):string>
-- !query 0 output int
-- !query 1 select typeof(1.2)
-- !query 1 schema struct<typeof(1.2):string>
-- !query 1 output decimal(2,1)
-- !query 2 select typeof(array(1, 2))
-- !query 2 schema struct<typeof(array(1, 2)):string>
-- !query 2 output array<int>
-- !query 3 select typeof(a) from (values (1), (2), (3.1)) t(a)
-- !query 3 schema struct<typeof(a):string>
-- !query 3 output decimal(11,1) decimal(11,1) decimal(11,1)
```
##### presto
```sql presto> select typeof(array[1]);
    _col0
----------------
array(integer)
(1 row)
```
##### PostgreSQL
```sql postgres=# select pg_typeof(a) from (values (1), (2), (3.0))
t(a);
pg_typeof
-----------
numeric
numeric
numeric
(3 rows)
```
##### impala https://issues.apache.org/jira/browse/IMPALA-1597
### Why are the changes needed? a function which is better we have to
help us debug, test, develop ...
### Does this PR introduce any user-facing change?
add a new function
### How was this patch tested?
add ut and example
Closes #26599 from yaooqinn/SPARK-29961.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala (diff)
The file was addedsql/core/src/test/resources/sql-tests/inputs/misc-functions.sql
The file was addedsql/core/src/test/resources/sql-tests/results/misc-functions.sql.out
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala (diff)
Commit 74cb1ffd686d67188a4540c74c6111affd6cce90 by gurwls223
[SPARK-22340][PYTHON][FOLLOW-UP] Add a better message and improve
documentation for pinned thread mode
### What changes were proposed in this pull request?
This PR proposes to show different warning message when the pinned
thread mode is enabled:
When enabled:
> PYSPARK_PIN_THREAD feature is enabled. However, note that it cannot
inherit the local properties from the parent thread although it isolates
each thread on PVM and JVM with its own local properties.
> To work around this, you should manually copy and set the local
properties from the parent thread to the child thread when you create
another thread.
When disabled:
> Currently, 'setLocalProperty' (set to local properties) with multiple
threads does not properly work.
> Internally threads on PVM and JVM are not synced, and JVM thread can
be reused for multiple threads on PVM, which fails to isolate local
properties for each thread on PVM.
> To work around this, you can set PYSPARK_PIN_THREAD to true (see
SPARK-22340). However, note that it cannot inherit the local properties
from the parent thread although it isolates each thread on PVM and JVM
with its own local properties.
> To work around this, you should manually copy and set the local
properties from the parent thread to the child thread when you create
another thread.
### Why are the changes needed?
Currently, it shows the same warning message regardless of
PYSPARK_PIN_THREAD being set. In the warning message it says "you can
set PYSPARK_PIN_THREAD to true ..." which is confusing.
### Does this PR introduce any user-facing change?
Documentation and warning message as shown above.
### How was this patch tested?
Manually tested.
```bash
$ PYSPARK_PIN_THREAD=true ./bin/pyspark
```
```python sc.setJobGroup("a", "b")
```
```
.../pyspark/util.py:141: UserWarning: PYSPARK_PIN_THREAD feature is
enabled. However, note that it cannot inherit the local properties from
the parent thread although it isolates each thread on PVM and JVM with
its own local properties. To work around this, you should manually copy
and set the local properties from the parent thread to the child thread
when you create another thread.
warnings.warn(msg, UserWarning)
```
```bash
$ ./bin/pyspark
```
```python sc.setJobGroup("a", "b")
```
```
.../pyspark/util.py:141: UserWarning: Currently, 'setJobGroup' (set to
local properties) with multiple threads does not properly work.
Internally threads on PVM and JVM are not synced, and JVM thread can be
reused for multiple threads on PVM, which fails to isolate local
properties for each thread on PVM. To work around this, you can set
PYSPARK_PIN_THREAD to true (see SPARK-22340). However, note that it
cannot inherit the local properties from the parent thread although it
isolates each thread on PVM and JVM with its own local properties. To
work around this, you should manually copy and set the local properties
from the parent thread to the child thread when you create another
thread.
warnings.warn(msg, UserWarning)
```
Closes #26588 from HyukjinKwon/SPARK-22340.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedpython/pyspark/context.py (diff)
The file was modifiedpython/pyspark/util.py (diff)
Commit d555f8fcc964e6e81f2d849de62d06877dca70c5 by dhyun
[SPARK-29961][SQL][FOLLOWUP] Remove useless test for VectorUDT
### What changes were proposed in this pull request?
A follow-up to rm useless test in VectorUDTSuite
### Why are the changes needed?
rm useless test, which is already covered.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
no
Closes #26620 from yaooqinn/SPARK-29961-f.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/linalg/VectorUDTSuite.scala (diff)
Commit affaefe1f3e623e6ae2042648ab78ee7b89d1ed6 by gurwls223
[MINOR][INFRA] Add `io` and `net` to GitHub Action Cache
### What changes were proposed in this pull request?
This PR aims to cache `~/.m2/repository/net` and `~/.m2/repository/io`
to reduce the flakiness.
### Why are the changes needed?
This will stabilize GitHub Action more before adding `hive-1.2` and
`hive-2.3` combination.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
After the GitHub Action on this PR passes, check the log.
Closes #26621 from dongjoon-hyun/SPARK-GHA-CACHE.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modified.github/workflows/master.yml (diff)
Commit 85c004d5b0303435dc207e139cdc51f0f2d3e160 by gurwls223
[SPARK-29885][PYTHON][CORE] Improve the exception message when reading
the daemon port
### What changes were proposed in this pull request? In production
environment, my PySpark application occurs an exception and it's message
as below:
``` 19/10/28 16:15:03 ERROR Executor: Exception in task 0.0 in stage 0.0
(TID 0) org.apache.spark.SparkException: No port number in
pyspark.daemon's stdout
at
org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:204)
at
org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:122)
at
org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:95)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at
org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:337)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:335)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1182)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
``` At first, I think a physical node has many ports are occupied by a
large number of processes. But I found the total number of ports in use
is only 671.
```
[yarnr1115 ~]$ netstat -a | wc -l 671 671
``` I  checked the code of PythonWorkerFactory in line 204 and found:
```
daemon = pb.start()
val in = new DataInputStream(daemon.getInputStream)
try {
daemonPort = in.readInt()
} catch {
case _: EOFException =>
throw new SparkException(s"No port number in $daemonModule's stdout")
}
``` I added some code here:
``` logError("Meet EOFException, daemon is alive: ${daemon.isAlive()}")
logError("Exit value: ${daemon.exitValue()}")
``` Then I recurrent the exception and it's message as below:
``` 19/10/28 16:15:03 ERROR PythonWorkerFactory: Meet EOFException,
daemon is alive: false 19/10/28 16:15:03 ERROR PythonWorkerFactory: Exit
value: 139 19/10/28 16:15:03 ERROR Executor: Exception in task 0.0 in
stage 0.0 (TID 0) org.apache.spark.SparkException: No port number in
pyspark.daemon's stdout
at
org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:206)
at
org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:122)
at
org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:95)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at
org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:337)
at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:335)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1182)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
at
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
``` I think the exception message has caused me a lot of confusion. This
PR will add meaningful log for exception information.
### Why are the changes needed? In order to clarify the exception and
try three times default.
### Does this PR introduce any user-facing change? No.
### How was this patch tested? Exists UT.
Closes #26510 from beliefer/improve-except-message.
Authored-by: gengjiaan <gengjiaan@360.cn> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala (diff)
Commit 297cbab98e68fed08cd790c24020b3df416e1e03 by ruifengz
[SPARK-29942][ML] Impl Complement Naive Bayes Classifier
### What changes were proposed in this pull request? Impl Complement
Naive Bayes Classifier as a `modelType` option in `NaiveBayes`
### Why are the changes needed? 1, it is a better choice for text
classification: it is said in
[scikit-learn](https://scikit-learn.org/stable/modules/naive_bayes.html#complement-naive-bayes)
that 'CNB regularly outperforms MNB (often by a considerable margin) on
text classification tasks.' 2, CNB is highly similar to existing MNB,
only a small part of existing MNB need to be changed, so it is a easy
win to support CNB.
### Does this PR introduce any user-facing change? yes, a new
`modelType` is supported
### How was this patch tested? added testsuites
Closes #26575 from zhengruifeng/cnb.
Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by:
zhengruifeng <ruifengz@foxmail.com>
The file was modifieddocs/ml-classification-regression.md (diff)
The file was modifiedpython/pyspark/ml/classification.py (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala (diff)
Commit 0f40d2a6ee0bba3f95785b8c36171e104c12615a by ruifengz
[SPARK-29960][ML][PYSPARK] MulticlassClassificationEvaluator support
hammingLoss
### What changes were proposed in this pull request?
MulticlassClassificationEvaluator support hammingLoss
### Why are the changes needed? 1, it is an easy to compute hammingLoss
based on confusion matrix 2, scikit-learn supports it
### Does this PR introduce any user-facing change? yes
### How was this patch tested? added testsuites
Closes #26597 from zhengruifeng/multi_class_hamming_loss.
Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by:
zhengruifeng <ruifengz@foxmail.com>
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/evaluation/MulticlassMetrics.scala (diff)
The file was modifiedpython/pyspark/ml/evaluation.py (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala (diff)
Commit 6146dc4562739c1c947eb944897c2fe85d1016e0 by wenchen
[SPARK-29874][SQL] Optimize Dataset.isEmpty()
### What changes were proposed in this pull request? In  origin way to
judge if a DataSet is empty by
```
def isEmpty: Boolean = withAction("isEmpty",
limit(1).groupBy().count().queryExecution) { plan =>
   plan.executeCollect().head.getLong(0) == 0
}
``` will add two shuffles by `limit()`, `groupby() and count()`, then
collect all data to driver. In this way we can avoid `oom` when collect
data to driver. But it will trigger all partitions calculated and add
more shuffle process.
We change it to
```
def isEmpty: Boolean = withAction("isEmpty", select().queryExecution) {
plan =>
   plan.executeTake(1).isEmpty
}
``` After these pr, we will add a column pruning to origin LogicalPlan
and use `executeTake()` API. then we won't add more shuffle process and
just compute only one partition's data in last stage. In this way we can
reduce cost when we call `DataSet.isEmpty()` and won't bring memory
issue to driver side.
### Why are the changes needed? Optimize Dataset.isEmpty()
### Does this PR introduce any user-facing change? No
### How was this patch tested? Origin UT
Closes #26500 from AngersZhuuuu/SPARK-29874.
Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala (diff)
Commit cdcd43cbf2479b258f4c5cfa0f6306f475d25cf2 by wenchen
[SPARK-29977][SQL] Remove
newMutableProjection/newOrdering/newNaturalAscendingOrdering  from
SparkPlan
### What changes were proposed in this pull request?
This is to refactor `SparkPlan` code; it mainly removed
`newMutableProjection`/`newOrdering`/`newNaturalAscendingOrdering` from
`SparkPlan`. The other modifications are listed below;
- Move `BaseOrdering` from
`o.a.s.sqlcatalyst.expressions.codegen.GenerateOrdering.scala` to
`o.a.s.sqlcatalyst.expressions.ordering.scala`
- `RowOrdering` extends `CodeGeneratorWithInterpretedFallback ` for
`BaseOrdering`
- Remove the unused variables (`subexpressionEliminationEnabled` and
`codeGenFallBack`) from `SparkPlan`
### Why are the changes needed?
For better code/test coverage.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing.
Closes #26615 from maropu/RefactorOrdering.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectHashAggregateExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ReferenceSort.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala (diff)
The file was modifiedsql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala (diff)
Commit 54c5087a3ae306ad766df81a5a6279f219b2ea47 by wenchen
[SPARK-29248][SQL] provider number of partitions when creating v2 data
writer factory
### What changes were proposed in this pull request? When implementing a
ScanBuilder, we require the implementor to provide the schema of the
data and the number of partitions.
However, when someone is implementing WriteBuilder we only pass them the
schema, but not the number of partitions. This is an asymetrical
developer experience.
This PR adds a PhysicalWriteInfo interface that is passed to
createBatchWriterFactory and createStreamingWriterFactory that adds the
number of partitions of the data that is going to be written.
### Why are the changes needed? Passing in the number of partitions on
the WriteBuilder would enable data sources to provision their write
targets before starting to write. For example:
it could be used to provision a Kafka topic with a specific number of
partitions it could be used to scale a microservice prior to sending the
data to it it could be used to create a DsV2 that sends the data to
another spark cluster (currently not possible since the reader wouldn't
be able to know the number of partitions)
### Does this PR introduce any user-facing change? No
### How was this patch tested? Tests passed
Closes #26591 from edrevo/temp.
Authored-by: Ximo Guanter <joaquin.guantergonzalbez@telefonica.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/MicroBatchWrite.scala (diff)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/write/streaming/StreamingWrite.java (diff)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/write/DataWriterFactory.java (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/noop/NoopDataSource.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/InMemoryTable.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ConsoleWrite.scala (diff)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/write/streaming/StreamingDataWriterFactory.java (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchWrite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileBatchWrite.scala (diff)
The file was addedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/write/PhysicalWriteInfoImpl.scala
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memory.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/sources/StreamingDataSourceV2Suite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/SimpleWritableDataSource.scala (diff)
The file was addedsql/catalyst/src/main/java/org/apache/spark/sql/connector/write/PhysicalWriteInfo.java
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/write/BatchWrite.java (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaStreamingWrite.scala (diff)
Commit 6b4b6a87cde8e29da5cbc2ee00242ec74d5477b2 by blue
[SPARK-29558][SQL] ResolveTables and ResolveRelations should be
order-insensitive
### What changes were proposed in this pull request?
Make `ResolveRelations` call `ResolveTables` at the beginning, and make
`ResolveTables` call `ResolveTempViews`(newly added) at the beginning,
to ensure the relation resolution priority.
### Why are the changes needed?
To resolve an `UnresolvedRelation`, the general process is: 1. try to
resolve to (global) temp view first. If it's not a temp view, move on 2.
if the table name specifies a catalog, lookup the table from the
specified catalog. Otherwise, lookup table from the current catalog. 3.
when looking up table from session catalog, return a v1 relation if the
table provider is v1.
Currently, this process is done by 2 rules: `ResolveTables` and
`ResolveRelations`. To avoid rule conflicts, we add a lot of checks: 1.
`ResolveTables` only resolves `UnresolvedRelation` if it's not a temp
view and the resolved table is not v1. 2. `ResolveRelations` only
resolves `UnresolvedRelation` if the table name has less than 2 parts.
This requires to run `ResolveTables` before `ResolveRelations`,
otherwise we may resolve a v2 table to a v1 relation.
To clearly guarantee the resolution priority, and avoid massive changes,
this PR proposes to call one rule in another rule to ensure the rule
execution order. Now the process is simple: 1. first run
`ResolveTempViews`, see if we can resolve relation to temp view 2. then
run `ResolveTables`, see if we can resolve relation to v2 tables. 3.
finally run `ResolveRelations`, see if we can resolve relation to v1
tables.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
existing tests
Closes #26214 from cloud-fan/resolve.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Ryan
Blue <blue@apache.org>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala (diff)
Commit e2f056f4a89b1bd9864be8c111d39af6558c839b by gurwls223
[SPARK-29975][SQL] introduce --CONFIG_DIM directive
### What changes were proposed in this pull request?
allow the sql test files to specify different dimensions of config sets
during testing. For example,
```
--CONFIG_DIM1 a=1
--CONFIG_DIM1 b=2,c=3
--CONFIG_DIM2 x=1
--CONFIG_DIM2 y=1,z=2
```
This example defines 2 config dimensions, and each dimension defines 2
config sets. We will run the queries 4 times: 1. a=1, x=1 2. a=1, y=1,
z=2 3. b=2, c=3, x=1 4. b=2, c=3, y=1, z=2
### Why are the changes needed?
Currently `SQLQueryTestSuite` takes a long time. This is because we run
each test at least 3 times, to check with different codegen modes. This
is not necessary for most of the tests, e.g. DESC TABLE. We should only
check these codegen modes for certain tests.
With the --CONFIG_DIM directive, we can do things like: test different
join operator(broadcast or shuffle join) X different codegen modes.
After reducing testing time, we should be able to run thrifter server
SQL tests with config settings.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
test only
Closes #26612 from cloud-fan/test.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/inner-join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/window_part3.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/order-by-nulls-ordering.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/group-by.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/window_part2.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/aggregates_part4.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/join-empty-relation.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-group-by.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-order-by.sql (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/aggregates_part1.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/window_part1.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/window_part4.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-group-by.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/udf/udf-natural-join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/outer-join.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/natural-join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/aggregates_part3.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/udaf.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/outer-join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/aggregates_part2.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/window.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-orderby-limit.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/udf/udf-outer-join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/postgreSQL/join.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-joins.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/udf/udf-join-empty-relation.sql (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql (diff)
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/subquery/exists-subquery/exists-aggregate.sql (diff)
Commit 9ec2a4e58caa4128e9c690d72239cebd6b732084 by gurwls223
[SPARK-29911][SQL][FOLLOWUP] Move related unit test to
ThriftServerWithSparkContextSuite
### What changes were proposed in this pull request? This is follow up
of #26543
See https://github.com/apache/spark/pull/26543#discussion_r348934276
### Does this PR introduce any user-facing change? No
### How was this patch tested? Exist UT.
Closes #26628 from LantaoJin/SPARK-29911_FOLLOWUP.
Authored-by: LantaoJin <jinlantao@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala (diff)
The file was addedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala
Commit 2dd6807e421c96d0aaafc57ceb48f50f66f9d2e7 by wenchen
[SPARK-28023][SQL] Add trim logic in UTF8String's toInt/toLong to make
it consistent with other string-numeric casting
### What changes were proposed in this pull request?
Modify `UTF8String.toInt/toLong` to support trim spaces for both sides
before converting it to byte/short/int/long.
With this kind of "cheap" trim can help improve performance for casting
string to integrals. The idea is from
https://github.com/apache/spark/pull/24872#issuecomment-556917834
### Why are the changes needed?
make the behavior consistent.
### Does this PR introduce any user-facing change? yes, cast string to
an integral type, and binary comparison between string and integrals
will trim spaces first. their behavior will be consistent with float and
double.
### How was this patch tested? 1. add ut. 2. benchmark tests
the benchmark is modified based on
https://github.com/apache/spark/pull/24872#issuecomment-503827016
```scala
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements.  See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version
2.0
* (the "License"); you may not use this file except in compliance with
* the License.  You may obtain a copy of the License at
*
*    http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.sql.execution.benchmark
import org.apache.spark.benchmark.Benchmark
/**
* Benchmark trim the string when casting string type to Boolean/Numeric
types.
* To run this benchmark:
* {{{
*   1. without sbt:
*      bin/spark-submit --class <this class> --jars <spark core test
jar> <spark sql test jar>
*   2. build/sbt "sql/test:runMain <this class>"
*   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain <this class>"
*      Results will be written to
"benchmarks/CastBenchmark-results.txt".
* }}}
*/ object CastBenchmark extends SqlBasedBenchmark { This conversation
was marked as resolved by yaooqinn
  override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
   val title = "Cast String to Integral"
   runBenchmark(title) {
     withTempPath { dir =>
       val N = 500L << 14
       val df = spark.range(N)
       val types = Seq("int", "long")
       (1 to 5).by(2).foreach { i =>
         df.selectExpr(s"concat(id, '${" " * i}') as str")
           .write.mode("overwrite").parquet(dir + i.toString)
       }
        val benchmark = new Benchmark(title, N, minNumIters = 5, output
= output)
       Seq(true, false).foreach { trim =>
         types.foreach { t =>
           val str = if (trim) "trim(str)" else "str"
           val expr = s"cast($str as $t) as c_$t"
           (1 to 5).by(2).foreach { i =>
             benchmark.addCase(expr + s" - with $i spaces") { _ =>
               spark.read.parquet(dir +
i.toString).selectExpr(expr).collect()
             }
           }
         }
       }
       benchmark.run()
     }
   }
}
}
```
#### benchmark result. normal trim v.s. trim in toInt/toLong
```java
================================================================================================
Cast String to Integral
================================================================================================
Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1
Intel(R) Core(TM) i5-5287U CPU  2.90GHz Cast String to Integral:       
         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per
Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
cast(trim(str) as int) as c_int - with 1 spaces          10220        
12994        1337          0.8        1247.5       1.0X cast(trim(str)
as int) as c_int - with 3 spaces           4763           8356       
357          1.7         581.4       2.1X cast(trim(str) as int) as
c_int - with 5 spaces           4791           8042         NaN        
1.7         584.9       2.1X cast(trim(str) as long) as c_long - with 1
spaces           4014           6755         NaN          2.0       
490.0       2.5X cast(trim(str) as long) as c_long - with 3 spaces     
    4737           6938         NaN          1.7         578.2     
2.2X cast(trim(str) as long) as c_long - with 5 spaces           4478  
       6919        1404          1.8         546.6       2.3X cast(str
as int) as c_int - with 1 spaces           4443           6222       
NaN          1.8         542.3       2.3X cast(str as int) as c_int -
with 3 spaces           3659           3842         170          2.2   
    446.7       2.8X cast(str as int) as c_int - with 5 spaces         
4372           7996         NaN          1.9         533.7       2.3X
cast(str as long) as c_long - with 1 spaces           3866         
5838         NaN          2.1         471.9       2.6X cast(str as long)
as c_long - with 3 spaces           3793           5449         NaN    
    2.2         463.0       2.7X cast(str as long) as c_long - with 5
spaces           4947           5961        1198          1.7       
603.9       2.1X
```
Closes #26622 from yaooqinn/cheapstringtrim.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/core/src/test/resources/sql-tests/results/comparator.sql.out (diff)
The file was modifieddocs/sql-migration-guide.md (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/results/cast.sql.out (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/comparator.sql (diff)
The file was modifiedcommon/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/cast.sql (diff)
Commit 6e581cf164c3a2930966b270ac1406dc1195c942 by gatorsmile
[SPARK-29893][SQL][FOLLOWUP] code cleanup for local shuffle reader
### What changes were proposed in this pull request?
A few cleanups for https://github.com/apache/spark/pull/26516: 1. move
the calculating of partition start indices from the RDD to the rule. We
can reuse code from "shrink number of reducers" in the future if we
split partitions by size. 2. only check extra shuffles when adding local
readers to the probe side. 3. add comments. 4. simplify the config name:
`optimizedLocalShuffleReader` -> `localShuffleReader`
### Why are the changes needed?
make code more maintainable.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
existing tests
Closes #26625 from cloud-fan/aqe.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Xiao Li
<gatorsmile@gmail.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LocalShuffledRowRDD.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
Commit 6b0e391aa49acd5029d00fefc0c90fcdfdf88cb6 by dhyun
[SPARK-29427][SQL] Add API to convert RelationalGroupedDataset to
KeyValueGroupedDataset
### What changes were proposed in this pull request?
This PR proposes to add `as` API to RelationalGroupedDataset. It creates
KeyValueGroupedDataset instance using given grouping expressions,
instead of a typed function in groupByKey API. Because it can leverage
existing columns, it can use existing data partition, if any, when doing
operations like cogroup.
### Why are the changes needed?
Currently if users want to do cogroup on DataFrames, there is no good
way to do except for KeyValueGroupedDataset.
1. KeyValueGroupedDataset ignores existing data partition if any. That
is a problem. 2. groupByKey calls typed function to create additional
keys. You can not reuse existing columns, if you just need grouping by
them.
```scala
// df1 and df2 are certainly partitioned and sorted. val df1 = Seq((1,
2, 3), (2, 3, 4)).toDF("a", "b", "c")
.repartition($"a").sortWithinPartitions("a") val df2 = Seq((1, 2, 4),
(2, 3, 5)).toDF("a", "b", "c")
.repartition($"a").sortWithinPartitions("a")
```
```scala
// This groupBy.as.cogroup won't unnecessarily repartition the data val
df3 = df1.groupBy("a").as[Int]
.cogroup(df2.groupBy("a").as[Int]) { case (key, data1, data2) =>
   data1.zip(data2).map { p =>
     p._1.getInt(2) + p._2.getInt(2)
   }
}
```
```
== Physical Plan ==
*(5) SerializeFromObject [input[0, int, false] AS value#11247]
+- CoGroup
org.apache.spark.sql.DataFrameSuite$$Lambda$4922/12067092816eec1b6f,
a#11209: int, createexternalrow(a#11209, b#11210, c#11211,
StructField(a,IntegerType,false), StructField(b,IntegerType,false),
StructField(c,IntegerType,false)), createexternalrow(a#11225, b#11226,
c#11227, StructField(a,IntegerType,false),
StructField(b,IntegerType,false), StructField(c,IntegerType,false)),
[a#11209], [a#11225], [a#11209, b#11210, c#11211], [a#11225, b#11226,
c#11227], obj#11246: int
  :- *(2) Sort [a#11209 ASC NULLS FIRST], false, 0
  :  +- Exchange hashpartitioning(a#11209, 5), false, [id=#10218]
  :     +- *(1) Project [_1#11202 AS a#11209, _2#11203 AS b#11210,
_3#11204 AS c#11211]
  :        +- *(1) LocalTableScan [_1#11202, _2#11203, _3#11204]
  +- *(4) Sort [a#11225 ASC NULLS FIRST], false, 0
     +- Exchange hashpartitioning(a#11225, 5), false, [id=#10223]
        +- *(3) Project [_1#11218 AS a#11225, _2#11219 AS b#11226,
_3#11220 AS c#11227]
           +- *(3) LocalTableScan [_1#11218, _2#11219, _3#11220]
```
```scala
// Current approach creates additional AppendColumns and repartition
data again val df4 = df1.groupByKey(r =>
r.getInt(0)).cogroup(df2.groupByKey(r => r.getInt(0))) {
case (key, data1, data2) =>
   data1.zip(data2).map { p =>
     p._1.getInt(2) + p._2.getInt(2)
}
}
```
```
== Physical Plan ==
*(7) SerializeFromObject [input[0, int, false] AS value#11257]
+- CoGroup
org.apache.spark.sql.DataFrameSuite$$Lambda$4933/138102700737171997,
value#11252: int, createexternalrow(a#11209, b#11210, c#11211,
StructField(a,IntegerType,false), StructField(b,IntegerType,false),
StructField(c,IntegerType,false)), createexternalrow(a#11225, b#11226,
c#11227, StructField(a,IntegerType,false),
StructField(b,IntegerType,false), StructField(c,IntegerType,false)),
[value#11252], [value#11254], [a#11209, b#11210, c#11211], [a#11225,
b#11226, c#11227], obj#11256: int
  :- *(3) Sort [value#11252 ASC NULLS FIRST], false, 0
  :  +- Exchange hashpartitioning(value#11252, 5), true, [id=#10302]
  :     +- AppendColumns
org.apache.spark.sql.DataFrameSuite$$Lambda$4930/19529195347ce07f47,
createexternalrow(a#11209, b#11210, c#11211,
StructField(a,IntegerType,false), StructField(b,IntegerType,false),
StructField(c,IntegerType,false)), [input[0, int, false] AS value#11252]
  :        +- *(2) Sort [a#11209 ASC NULLS FIRST], false, 0
  :           +- Exchange hashpartitioning(a#11209, 5), false,
[id=#10297]
  :              +- *(1) Project [_1#11202 AS a#11209, _2#11203 AS
b#11210, _3#11204 AS c#11211]
  :                 +- *(1) LocalTableScan [_1#11202, _2#11203,
_3#11204]
  +- *(6) Sort [value#11254 ASC NULLS FIRST], false, 0
     +- Exchange hashpartitioning(value#11254, 5), true, [id=#10312]
        +- AppendColumns
org.apache.spark.sql.DataFrameSuite$$Lambda$4932/15265288491f0e0c1f,
createexternalrow(a#11225, b#11226, c#11227,
StructField(a,IntegerType,false), StructField(b,IntegerType,false),
StructField(c,IntegerType,false)), [input[0, int, false] AS value#11254]
           +- *(5) Sort [a#11225 ASC NULLS FIRST], false, 0
              +- Exchange hashpartitioning(a#11225, 5), false,
[id=#10307]
                 +- *(4) Project [_1#11218 AS a#11225, _2#11219 AS
b#11226, _3#11220 AS c#11227]
                    +- *(4) LocalTableScan [_1#11218, _2#11219,
_3#11220]
```
### Does this PR introduce any user-facing change?
Yes, this adds a new `as` API to RelationalGroupedDataset. Users can use
it to create KeyValueGroupedDataset and do cogroup.
### How was this patch tested?
Unit tests.
Closes #26509 from viirya/SPARK-29427-2.
Lead-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by:
Liang-Chi Hsieh <liangchi@uber.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala (diff)
Commit f28eab2de72f7da883b970fc19edd4f569340bd7 by vanzin
[SPARK-29971][CORE] Fix buffer leaks in
`TransportFrameDecoder/TransportCipher`
### What changes were proposed in this pull request?
- Correctly release `ByteBuf` in `TransportCipher` in all cases
- Move closing / releasing logic to `handlerRemoved(...)` so we are
guaranteed that is always called.
- Correctly release `frameBuf` it is not null when the handler is
removed (and so also when the channel becomes inactive)
### Why are the changes needed?
We need to carefully manage the ownership / lifecycle of `ByteBuf`
instances so we don't leak any of these. We did not correctly do this in
all cases:
- when end up in invalid cipher state.
- when partial data was received and the channel is closed before the
full frame is decoded
Fixes https://github.com/netty/netty/issues/9784.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the newly added UTs.
Closes #26609 from normanmaurer/fix_leaks.
Authored-by: Norman Maurer <norman_maurer@apple.com> Signed-off-by:
Marcelo Vanzin <vanzin@cloudera.com>
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/crypto/TransportCipher.java (diff)
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/util/ByteArrayReadableChannel.java (diff)
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java (diff)
The file was addedcommon/network-common/src/test/java/org/apache/spark/network/crypto/TransportCipherSuite.java
Commit fc7a37b147d76edcd0de97ca1645dba736317a96 by gurwls223
[SPARK-30003][SQL] Do not throw stack overflow exception in non-root
unknown hint resolution
### What changes were proposed in this pull request? This is rather a
followup of https://github.com/apache/spark/pull/25464 (see
https://github.com/apache/spark/pull/25464/files#r349543286)
It will cause an infinite recursion via mapping children - we should
return the hint rather than its parent plan in unknown hint resolution.
### Why are the changes needed?
Prevent Stack over flow during hint resolution.
### Does this PR introduce any user-facing change?
Yes, it avoids stack overflow exception It was caused by
https://github.com/apache/spark/pull/25464 and this is only in the
master.
No behaviour changes to end users as it happened only in the master.
### How was this patch tested?
Unittest was added.
Closes #26642 from HyukjinKwon/SPARK-30003.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala (diff)
Commit c98e5eb3396a6db92f2420e743afa9ddff319ca2 by dhyun
[SPARK-29981][BUILD] Add hive-1.2/2.3 profiles
### What changes were proposed in this pull request?
This PR aims the followings.
- Add two profiles, `hive-1.2` and `hive-2.3` (default)
- Validate if we keep the existing combination at least. (Hadoop-2.7 +
Hive 1.2 / Hadoop-3.2 + Hive 2.3).
For now, we assumes that `hive-1.2` is explicitly used with `hadoop-2.7`
and `hive-2.3` with `hadoop-3.2`. The followings are beyond the scope of
this PR.
- SPARK-29988 Adjust Jenkins jobs for `hive-1.2/2.3` combination
- SPARK-29989 Update release-script for `hive-1.2/2.3` combination
- SPARK-29991 Support `hive-1.2/2.3` in PR Builder
### Why are the changes needed?
This will help to switch our dependencies to update the exposed
dependencies.
### Does this PR introduce any user-facing change?
This is a dev-only change that the build profile combinations are
changed.
- `-Phadoop-2.7` => `-Phadoop-2.7 -Phive-1.2`
- `-Phadoop-3.2` => `-Phadoop-3.2 -Phive-2.3`
### How was this patch tested?
Pass the Jenkins with the dependency check and tests to make it sure we
don't change anything for now.
- [Jenkins (-Phadoop-2.7
-Phive-1.2)](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114192/consoleFull)
- [Jenkins (-Phadoop-3.2
-Phive-2.3)](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114192/consoleFull)
Also, from now, GitHub Action validates the following combinations.
![gha](https://user-images.githubusercontent.com/9700541/69355365-822d5e00-0c36-11ea-93f7-e00e5459e1d0.png)
Closes #26619 from dongjoon-hyun/SPARK-29981.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was modifiedappveyor.yml (diff)
The file was modifiedsql/hive/pom.xml (diff)
The file was modifieddev/run-tests.py (diff)
The file was modifiedpom.xml (diff)
The file was modified.github/workflows/master.yml (diff)
The file was modifieddev/test-dependencies.sh (diff)
Commit 6625b69027661d34352aa22e75dee6f31d069b41 by dhyun
[SPARK-29981][BUILD][FOLLOWUP] Change hive.version.short
### What changes were proposed in this pull request?
This is a follow-up according to liancheng 's advice.
- https://github.com/apache/spark/pull/26619#discussion_r349326090
### Why are the changes needed?
Previously, we chose the full version to be carefully. As of today, it
seems that `Apache Hive 2.3` branch seems to become stable.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the compile combination on GitHub Action. 1.
hadoop-2.7/hive-1.2/JDK8 2. hadoop-2.7/hive-2.3/JDK8 3.
hadoop-3.2/hive-2.3/JDK8 4. hadoop-3.2/hive-2.3/JDK11
Also, pass the Jenkins with `hadoop-2.7` and `hadoop-3.2` for (1) and
(4).
(2) and (3) is not ready in Jenkins.
Closes #26645 from dongjoon-hyun/SPARK-RENAME-HIVE-DIRECTORY.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/scala/org/apache/spark/sql/hive/thriftserver/ThriftserverShimUtils.scala
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/OperationState.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchResultsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetOperationStatusReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoType.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetResultSetMetadataResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/OperationHandle.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCLIServiceConstants.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TStatusCode.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoValue.java
The file was removedsql/core/v2.3.5/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcShimUtils.scala
The file was addedsql/core/v1.2/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/OperationType.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/CompositeService.java
The file was removedsql/core/v2.3.5/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TUnionTypeEntry.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TFetchOrientation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/auth/TSetIpAddressProcessor.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/FetchOrientation.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetPrimaryKeysReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetPrimaryKeysOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetColumnsResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeEntry.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TDoubleColumn.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TRenewDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TDoubleColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/SQLOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/FetchOrientation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/auth/TSetIpAddressProcessor.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSession.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeQualifiers.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetPrimaryKeysOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TOperationHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetFunctionsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetResultSetMetadataReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetOperationStatusReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TBoolValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/TableTypeMapping.java
The file was addedsql/core/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TDoubleValue.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetDelegationTokenResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TColumnDesc.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTypeInfoResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ICLIService.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/ClassicTableTypeMapping.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TStatus.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/Operation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TRow.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/ServiceOperations.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TDoubleColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetCatalogsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI64Value.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/SessionHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/MetadataOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TPrimitiveTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/CLIService.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TStringValue.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TSessionHandle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetResultSetMetadataResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TBoolColumn.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/SessionHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TBinaryColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetOperationStatusReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSession.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/OperationType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TColumnValue.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI64Column.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TStringColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/OperationType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCloseOperationResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTypeQualifierValue.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetResultSetMetadataReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetColumnsReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI16Value.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/HiveSQLException.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/ColumnDescriptor.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/GetInfoValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTablesResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/RowSetFactory.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TRenewDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TPrimitiveTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TDoubleValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTableTypesResp.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/SessionHandle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetResultSetMetadataReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetFunctionsResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCancelDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TFetchResultsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/TSetIpAddressProcessor.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TSessionHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/TableSchema.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTypeDesc.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetOperationStatusReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TDoubleValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TSessionHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TJobExecutionStatus.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/HttpAuthUtils.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI16Column.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCancelDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/ServiceUtils.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/Type.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TByteColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI32Column.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/HandleIdentifier.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetColumnsResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TOpenSessionReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TUserDefinedTypeEntry.java
The file was modifiedpom.xml (diff)
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/Column.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/PatternOrIdentifier.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TRenewDelegationTokenReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/Operation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI64Value.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/CLIService.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/Handle.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TRenewDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/auth/KerberosSaslHelper.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/TableTypeMapping.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI64Column.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCancelDelegationTokenReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
The file was addedsql/hive-thriftserver/v2.3/if/TCLIService.thrift
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTypeInfoReq.java
The file was removedsql/core/v1.2.1/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TPrimitiveTypeEntry.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ICLIService.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TOpenSessionResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTablesReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI32Value.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/auth/PlainSaslHelper.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/TypeQualifiers.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TStructTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetSchemasResp.java
The file was addedsql/core/v2.3/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TRowSet.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/SessionManager.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI16Column.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTypeQualifierValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/CookieSigner.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/ServiceOperations.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TBinaryColumn.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TUnionTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/ICLIService.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTableTypesReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TProgressUpdateResp.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/HiveSession.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/session/SessionManager.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetCatalogsReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TBoolColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTableTypesReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelOperationResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchResultsResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TStatusCode.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/THandleIdentifier.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/FetchOrientation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/GetInfoValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/server/HiveServer2.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTableTypesResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/HiveSQLException.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/RowSet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/OperationState.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTypeDesc.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/HandleIdentifier.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/Handle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
The file was removedsql/hive-thriftserver/v1.2.1/if/TCLIService.thrift
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/ColumnBasedSet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseOperationResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TFetchResultsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/HandleIdentifier.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/ColumnValue.java
The file was addedsql/core/v2.3/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TFetchResultsReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TByteValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/SessionHandle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetFunctionsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TColumnDesc.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationType.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TDoubleColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TBinaryColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TUnionTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/ClassicTableTypeMapping.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TBoolValue.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCancelOperationResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI16Value.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TStringValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTypeId.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/RowSet.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TColumn.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TStatus.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
The file was removedsql/core/v2.3.5/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationHandle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnBasedSet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseOperationReq.java
The file was removedsql/core/v1.2.1/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TStructTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI64Value.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI16Column.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TUserDefinedTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseSessionResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/AbstractService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TI32Column.java
The file was addedsql/core/v2.3/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcShimUtils.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/session/HiveSession.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TMapTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TBoolValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/server/HiveServer2.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TExecuteStatementReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/OperationHandle.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchOrientation.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI16Column.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TByteColumn.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchResultsResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseSessionReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeQualifiers.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoType.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TStringColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/RowSetFactory.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java
The file was removedsql/core/v1.2.1/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcShimUtils.scala
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnBasedSet.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TStatus.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCloseSessionReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TOperationType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnDescriptor.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationState.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetDelegationTokenResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TStringValue.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCancelOperationReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCloseSessionResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TRenewDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/THandleIdentifier.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/GetInfoType.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/Handle.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeQualifierValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TArrayTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TProtocolVersion.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetPrimaryKeysReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TColumnDesc.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetFunctionsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/auth/KerberosSaslHelper.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TOpenSessionResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TBoolColumn.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/ServiceOperations.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/ColumnDescriptor.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/TableSchema.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/HiveTableTypeMapping.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTypeInfoResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TStatusCode.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TStatusCode.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TExecuteStatementReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCloseSessionReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetCrossReferenceOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
The file was removedsql/core/v1.2.1/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/ClassicTableTypeMapping.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI64Column.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/server/HiveServer2.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Handle.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetColumnsResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TDoubleValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/auth/PlainSaslHelper.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetSchemasResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetColumnsResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TBoolValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TStringColumn.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TJobExecutionStatus.java
The file was removedsql/hive-thriftserver/v2.3.5/if/TCLIService.thrift
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/PatternOrIdentifier.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TMapTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetOperationStatusResp.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/CLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetFunctionsResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/TableSchema.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoResp.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/SQLOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCrossReferenceResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCancelOperationResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/TypeDescriptor.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSessionHookContext.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TFetchResultsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/ServiceUtils.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/HiveSQLException.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCancelOperationReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TRowSet.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/OperationHandle.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/TableSchema.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI64Value.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TByteColumn.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetPrimaryKeysResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeQualifierValue.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/GetInfoType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TFetchOrientation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TByteColumn.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSessionHookContextImpl.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTableTypesReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TUserDefinedTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/HandleIdentifier.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCancelDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Column.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetInfoValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/TableTypeMapping.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/KerberosSaslHelper.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TRow.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseOperationReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/TypeQualifiers.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseSessionResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/Operation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TBoolColumn.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/RowBasedSet.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/PlainSaslHelper.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/MetadataOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/HiveTableTypeMapping.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/ServiceUtils.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTypeInfoReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI16Value.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TStructTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/HiveTableTypeMapping.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetFunctionsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/ServiceOperations.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/Operation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCancelOperationResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTableSchema.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/SQLOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TByteValue.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetColumnsReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetResultSetMetadataReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/AbstractService.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTableTypesResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TProtocolVersion.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetPrimaryKeysResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/RowSet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/CookieSigner.java
The file was addedsql/core/v2.3/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTablesReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TRowSet.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/ColumnDescriptor.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCrossReferenceResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetResultSetMetadataResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationState.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/auth/PlainSaslHelper.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TPrimitiveTypeEntry.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TSessionHandle.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TByteValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/HiveSQLException.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TArrayTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI32Value.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/RowSet.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/OperationState.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/MetadataOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/CompositeService.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/MetadataOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/TypeQualifiers.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI32Value.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetCrossReferenceOperation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/RowBasedSet.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TRow.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI32Value.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCatalogsResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TColumnDesc.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TOperationType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/TypeQualifiers.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTableTypesReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/FetchOrientation.java
The file was removedsql/core/v2.3.5/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseSessionReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TByteValue.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/auth/HttpAuthUtils.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/SessionManager.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTableSchema.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCloseSessionResp.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTypeQualifiers.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeId.java
The file was addedsql/hive-thriftserver/v1.2/src/main/scala/org/apache/spark/sql/hive/thriftserver/ThriftserverShimUtils.scala
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TArrayTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTableSchema.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/RowBasedSet.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/AbstractService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/auth/HttpAuthUtils.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTypeInfoResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTypeInfoReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeId.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/CLIService.java
The file was addedsql/hive-thriftserver/v1.2/if/TCLIService.thrift
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeDesc.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TStringColumn.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCLIServiceConstants.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetSchemasReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/ClassicTableTypeMapping.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TRow.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TOperationState.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/SQLOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/TypeDescriptor.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/CompositeService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/scala/org/apache/spark/sql/hive/thriftserver/ThriftserverShimUtils.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI16Value.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TBinaryColumn.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/GetInfoValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TCloseOperationResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/ServiceUtils.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCrossReferenceReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TI32Column.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/CookieSigner.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/TypeDescriptor.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TUnionTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TCLIService.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeDesc.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSessionHookContext.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTypeInfoResp.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/ColumnBasedSet.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/AbstractService.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/RowBasedSet.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/TypeDescriptor.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/CookieSigner.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/server/HiveServer2.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationHandle.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetCatalogsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTypeQualifiers.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TRenewDelegationTokenResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TRowSet.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetSchemasResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/ICLIService.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/OperationHandle.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetCatalogsResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TStringValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTypeInfoReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetDelegationTokenReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TUserDefinedTypeEntry.java
The file was addedsql/hive-thriftserver/v2.3/src/main/scala/org/apache/spark/sql/hive/thriftserver/ThriftserverShimUtils.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TRenewDelegationTokenReq.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/SessionManager.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/auth/HttpAuthUtils.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/GetInfoType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/Type.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TOperationHandle.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/operation/TableTypeMapping.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetTablesResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetSchemasResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/RowSetFactory.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/auth/KerberosSaslHelper.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchOrientation.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTablesResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TColumnValue.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TStructTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/RowSetFactory.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TTypeEntry.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCloseOperationResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TI32Column.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TArrayTypeEntry.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/OperationType.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TFetchResultsReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TCloseOperationReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TGetSchemasReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/GetInfoValue.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TOpenSessionReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TCloseOperationReq.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TI64Column.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCatalogsResp.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/GetInfoType.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/OperationState.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetCrossReferenceReq.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/CompositeService.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/operation/HiveTableTypeMapping.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/THandleIdentifier.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TColumnValue.java
The file was addedsql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TTableSchema.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TGetTablesResp.java
The file was addedsql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/ColumnValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/cli/session/HiveSessionHookContextImpl.java
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TOperationState.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TOperationType.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TStatus.java
The file was addedsql/core/v1.2/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/THandleIdentifier.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/auth/TSetIpAddressProcessor.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TColumnValue.java
The file was removedsql/hive-thriftserver/v2.3.5/src/main/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TProgressUpdateResp.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetTableTypesResp.java
The file was addedsql/hive-thriftserver/v2.3/src/gen/java/org/apache/hive/service/rpc/thrift/TRenewDelegationTokenReq.java
The file was removedsql/hive-thriftserver/v1.2.1/src/gen/java/org/apache/hive/service/cli/thrift/TGetDelegationTokenResp.java
The file was addedsql/core/v1.2/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcShimUtils.scala
The file was addedsql/hive-thriftserver/v1.2/src/gen/java/org/apache/hive/service/cli/thrift/TTypeId.java
The file was removedsql/hive-thriftserver/v2.3.5/src/gen/java/org/apache/hive/service/rpc/thrift/TGetResultSetMetadataResp.java