1. [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python (commit: 0a05cf917a2f701def38968e59217a98b6da8d8b) (details)
  2. [SPARK-25854][BUILD] fix `build/mvn` not to fail during Zinc server (commit: 3afb3a20e670c73677ab96d6fe5fcb3380800f33) (details)
Commit 0a05cf917a2f701def38968e59217a98b6da8d8b by ueshin
[SPARK-25822][PYSPARK] Fix a race condition when releasing a Python
## What changes were proposed in this pull request?
There is a race condition when releasing a Python worker. If
`ReaderIterator.handleEndOfDataSection` is not running in the task
thread, when a task is early terminated (such as `take(N)`), the task
completion listener may close the worker but "handleEndOfDataSection"
can still put the worker into the worker pool to reuse.
is a patch to reproduce this issue.
I also found a user reported this in the mail list:
This PR fixes the issue by using `compareAndSet` to make sure we will
never return a closed worker to the work pool.
## How was this patch tested?
Closes #22816 from zsxwing/fix-socket-closed.
Authored-by: Shixiong Zhu <> Signed-off-by: Takuya
(cherry picked from commit 86d469aeaa492c0642db09b27bb0879ead5d7166)
Signed-off-by: Takuya UESHIN <>
(commit: 0a05cf917a2f701def38968e59217a98b6da8d8b)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRunner.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala (diff)
Commit 3afb3a20e670c73677ab96d6fe5fcb3380800f33 by sean.owen
[SPARK-25854][BUILD] fix `build/mvn` not to fail during Zinc server
the final line in the mvn helper script in build/ attempts to shut down
the zinc server.  due to the zinc server being set up w/a 30min timeout,
by the time the mvn test instantiation finishes, the server times out.
this means that when the mvn script tries to shut down zinc, it returns
w/an exit code of 1.  this will then automatically fail the entire build
(even if the build passes).
i set up a test build:
Closes #22854 from shaneknapp/fix-mvn-helper-script.
Authored-by: shane knapp <> Signed-off-by: Sean Owen
(cherry picked from commit 6aa506394958bfb30cd2a9085a5e8e8be927de51)
Signed-off-by: Sean Owen <>
(commit: 3afb3a20e670c73677ab96d6fe5fcb3380800f33)
The file was modifiedbuild/mvn (diff)