SuccessChanges

Summary

  1. [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python (commit: 0a05cf917a2f701def38968e59217a98b6da8d8b) (details)
  2. [SPARK-25854][BUILD] fix `build/mvn` not to fail during Zinc server (commit: 3afb3a20e670c73677ab96d6fe5fcb3380800f33) (details)
Commit 0a05cf917a2f701def38968e59217a98b6da8d8b by ueshin
[SPARK-25822][PYSPARK] Fix a race condition when releasing a Python
worker
## What changes were proposed in this pull request?
There is a race condition when releasing a Python worker. If
`ReaderIterator.handleEndOfDataSection` is not running in the task
thread, when a task is early terminated (such as `take(N)`), the task
completion listener may close the worker but "handleEndOfDataSection"
can still put the worker into the worker pool to reuse.
https://github.com/zsxwing/spark/commit/0e07b483d2e7c68f3b5c3c118d0bf58c501041b7
is a patch to reproduce this issue.
I also found a user reported this in the mail list:
http://mail-archives.apache.org/mod_mbox/spark-user/201610.mbox/%3CCAAUq=H+YLUEpd23nwvq13Ms5hOStkhX3ao4f4zQV6sgO5zM-xAmail.gmail.com%3E
This PR fixes the issue by using `compareAndSet` to make sure we will
never return a closed worker to the work pool.
## How was this patch tested?
Jenkins.
Closes #22816 from zsxwing/fix-socket-closed.
Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Takuya
UESHIN <ueshin@databricks.com>
(cherry picked from commit 86d469aeaa492c0642db09b27bb0879ead5d7166)
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
(commit: 0a05cf917a2f701def38968e59217a98b6da8d8b)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRunner.scala (diff)
Commit 3afb3a20e670c73677ab96d6fe5fcb3380800f33 by sean.owen
[SPARK-25854][BUILD] fix `build/mvn` not to fail during Zinc server
shutdown
the final line in the mvn helper script in build/ attempts to shut down
the zinc server.  due to the zinc server being set up w/a 30min timeout,
by the time the mvn test instantiation finishes, the server times out.
this means that when the mvn script tries to shut down zinc, it returns
w/an exit code of 1.  this will then automatically fail the entire build
(even if the build passes).
i set up a test build:
https://amplab.cs.berkeley.edu/jenkins/job/sknapp-testing-spark-branch-2.4-test-maven-hadoop-2.7/
Closes #22854 from shaneknapp/fix-mvn-helper-script.
Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(cherry picked from commit 6aa506394958bfb30cd2a9085a5e8e8be927de51)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 3afb3a20e670c73677ab96d6fe5fcb3380800f33)
The file was modifiedbuild/mvn (diff)