SuccessChanges

Summary

  1. [MINOR][DOCS] Fix Spark hive example. (commit: 533d603cebd2fe5197f4b4d40e1f54fa94c74f36) (details)
  2. [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor (commit: a73cb040cc24c4e6a0b4a56eff26af61d0ec618b) (details)
  3. [SPARK-27800][SQL][HOTFIX][FOLLOWUP] Fix wrong answer on BitwiseXor test (commit: db33bd23b7bd0d518ab8dc8098f46c617b8b79cc) (details)
  4. [SPARK-25139][SPARK-18406][CORE][BRANCH-2.3] Avoid NonFatals to kill the (commit: e56a9b19685b0ba089a0d6979d74f6592b9029d1) (details)
Commit 533d603cebd2fe5197f4b4d40e1f54fa94c74f36 by gurwls223
[MINOR][DOCS] Fix Spark hive example.
## What changes were proposed in this pull request?
Documentation has an error,
https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#hive-tables.
The example:
```scala scala> val dataDir = "/tmp/parquet_data" dataDir: String =
/tmp/parquet_data
scala> spark.range(10).write.parquet(dataDir)
scala> sql(s"CREATE EXTERNAL TABLE hive_ints(key int) STORED AS PARQUET
LOCATION '$dataDir'") res6: org.apache.spark.sql.DataFrame = []
scala> sql("SELECT * FROM hive_ints").show()
+----+
| key|
+----+
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
+----+
```
Range does not emit `key`, but `id` instead.
Closes #24657 from ScrapCodes/fix_hive_example.
Lead-authored-by: Prashant Sharma <prashant@apache.org> Co-authored-by:
Prashant Sharma <prashsh1@in.ibm.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(cherry picked from commit 5f4b50513cd34cd3dcf7f72972bfcd1f51031723)
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
(commit: 533d603cebd2fe5197f4b4d40e1f54fa94c74f36)
The file was modifiedexamples/src/main/scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala (diff)
Commit a73cb040cc24c4e6a0b4a56eff26af61d0ec618b by dhyun
[SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
## What changes were proposed in this pull request?
Fix example for bitwise xor function. 3 ^ 5 should be 6 rather than 2.
- See https://spark.apache.org/docs/latest/api/sql/index.html#_14
## How was this patch tested?
manual tests
Closes #24669 from alex-lx/master.
Authored-by: Liu Xiao <hhdxlx@gmail.com> Signed-off-by: Dongjoon Hyun
<dhyun@apple.com>
(cherry picked from commit bf617996aa95dd51be15da64ba9254f98697560c)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(commit: a73cb040cc24c4e6a0b4a56eff26af61d0ec618b)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala (diff)
Commit db33bd23b7bd0d518ab8dc8098f46c617b8b79cc by dhyun
[SPARK-27800][SQL][HOTFIX][FOLLOWUP] Fix wrong answer on BitwiseXor test
cases
This PR is a follow up of https://github.com/apache/spark/pull/24669 to
fix the wrong answers used in test cases.
Closes #24674 from dongjoon-hyun/SPARK-27800.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(cherry picked from commit a24cdc00bfa4eb011022aedec22a345a0e0e981d)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(commit: db33bd23b7bd0d518ab8dc8098f46c617b8b79cc)
The file was modifiedsql/hive/src/test/resources/golden/udf_bitwise_xor-1-8fc9b9bf1aced42754419310784e0a9f (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala (diff)
Commit e56a9b19685b0ba089a0d6979d74f6592b9029d1 by dhyun
[SPARK-25139][SPARK-18406][CORE][BRANCH-2.3] Avoid NonFatals to kill the
Executor in PythonRunner
## What changes were proposed in this pull request?
Python uses a prefetch approach to read the result from upstream and
serve them in another thread, thus it's possible that if the children
operator doesn't consume all the data then the Task cleanup may happen
before Python side read process finishes, this in turn create a race
condition that the block read locks are freed during Task cleanup and
then the reader try to release the read lock it holds and find it has
been released, in this case we shall hit a AssertionError.
We shall catch the AssertionError in PythonRunner and prevent this kill
the Executor.
## How was this patch tested?
Hard to write a unit test case for this case, manually verified with
failed job.
Closes #24670 from rezasafi/branch-2.3.
Authored-by: Xingbo Jiang <xingbo.jiang@databricks.com> Signed-off-by:
Dongjoon Hyun <dhyun@apple.com>
(commit: e56a9b19685b0ba089a0d6979d74f6592b9029d1)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRunner.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRDD.scala (diff)