1. [SPARK-27338][CORE] Fix deadlock in (details)
Commit 93c14c650bd37c1d5d8b6aab954004f801eb623f by wenchen
[SPARK-27338][CORE] Fix deadlock in
UnsafeExternalSorter.SpillableIterator when locking both
UnsafeExternalSorter.SpillableIterator and TaskMemoryManager
## What changes were proposed in this pull request?
In `UnsafeExternalSorter.SpillableIterator#loadNext()` takes lock on the
`UnsafeExternalSorter` and calls `freePage` once the `lastPage` is
consumed which needs to take a lock on `TaskMemoryManager`. At the same
time, there can be another MemoryConsumer using `UnsafeExternalSorter`
as part of sorting can try to `allocatePage` needs to get lock on
`TaskMemoryManager` which can cause spill to happen which requires lock
on `UnsafeExternalSorter` again causing deadlock. This is a classic
deadlock situation happening similar to the SPARK-26265.
To fix this, we can move the `freePage` call in `loadNext` outside of
`Synchronized` block similar to the fix in SPARK-26265
## How was this patch tested?
Manual tests were being done and will also try to add a test.
Closes #24265 from venkata91/deadlock-sorter.
Authored-by: Venkata krishnan Sowrirajan <>
Signed-off-by: Wenchen Fan <>
(cherry picked from commit 6c4552c65045cfe82ed95212ee7cff684e44288b)
Signed-off-by: Wenchen Fan <>
The file was modifiedcore/src/main/java/org/apache/spark/util/collection/unsafe/sort/ (diff)