SuccessChanges

Summary

  1. [SPARK-27338][CORE] Fix deadlock in (commit: 049a8883129c995b937943fc5d9228a9bbc1fdea) (details)
  2. [SPARK-27338][CORE][FOLLOWUP] remove trailing space (commit: b95a58e0feabb1451198bbd7266c01004206e958) (details)
  3. [SPARK-27216][CORE][BACKPORT-2.3] Upgrade RoaringBitmap to 0.7.45 to fix (commit: 9446da10ddfad7afe5f8ca9386d46607e489f742) (details)
Commit 049a8883129c995b937943fc5d9228a9bbc1fdea by wenchen
[SPARK-27338][CORE] Fix deadlock in
UnsafeExternalSorter.SpillableIterator when locking both
UnsafeExternalSorter.SpillableIterator and TaskMemoryManager
## What changes were proposed in this pull request?
In `UnsafeExternalSorter.SpillableIterator#loadNext()` takes lock on the
`UnsafeExternalSorter` and calls `freePage` once the `lastPage` is
consumed which needs to take a lock on `TaskMemoryManager`. At the same
time, there can be another MemoryConsumer using `UnsafeExternalSorter`
as part of sorting can try to `allocatePage` needs to get lock on
`TaskMemoryManager` which can cause spill to happen which requires lock
on `UnsafeExternalSorter` again causing deadlock. This is a classic
deadlock situation happening similar to the SPARK-26265.
To fix this, we can move the `freePage` call in `loadNext` outside of
`Synchronized` block similar to the fix in SPARK-26265
## How was this patch tested?
Manual tests were being done and will also try to add a test.
Closes #24265 from venkata91/deadlock-sorter.
Authored-by: Venkata krishnan Sowrirajan <vsowrirajan@qubole.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 6c4552c65045cfe82ed95212ee7cff684e44288b)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: 049a8883129c995b937943fc5d9228a9bbc1fdea)
The file was modifiedcore/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java (diff)
Commit b95a58e0feabb1451198bbd7266c01004206e958 by wenchen
[SPARK-27338][CORE][FOLLOWUP] remove trailing space
## What changes were proposed in this pull request?
https://github.com/apache/spark/pull/24265 breaks the lint check,
because it has trailing space. (not sure why it passed jenkins). This PR
fixes it.
## How was this patch tested?
N/A
Closes #24289 from cloud-fan/fix.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen
Fan <wenchen@databricks.com>
(commit: b95a58e0feabb1451198bbd7266c01004206e958)
The file was modifiedcore/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java (diff)
Commit 9446da10ddfad7afe5f8ca9386d46607e489f742 by sean.owen
[SPARK-27216][CORE][BACKPORT-2.3] Upgrade RoaringBitmap to 0.7.45 to fix
Kryo unsafe ser/dser issue
## What changes were proposed in this pull request?
Back-port of #24264 to branch-2.3.
HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks.
But RoaringBitmap couldn't be ser/deser with unsafe KryoSerializer.
It's a bug of RoaringBitmap-0.5.11 and fixed in latest version.
## How was this patch tested?
Add a UT
Closes #24291 from LantaoJin/SPARK-27216_BACKPORT-2.3.
Authored-by: LantaoJin <jinlantao@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(commit: 9446da10ddfad7afe5f8ca9386d46607e489f742)
The file was modifieddev/deps/spark-deps-hadoop-2.6 (diff)
The file was modifiedpom.xml (diff)
The file was modifieddev/deps/spark-deps-hadoop-2.7 (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala (diff)