1. [SPARK-26665][CORE] Fix a bug that BlockTransferService.fetchBlockSync (commit: b88067bd0f7b9466a89ce6458cb7766a24283b13) (details)
  2. [SPARK-26228][MLLIB] OOM issue encountered when computing Gramian matrix (commit: 98d48c7ad2883a1dbaa81828184c3e1d76e3190a) (details)
Commit b88067bd0f7b9466a89ce6458cb7766a24283b13 by zsxwing
[SPARK-26665][CORE] Fix a bug that BlockTransferService.fetchBlockSync
may hang forever
## What changes were proposed in this pull request?
`ByteBuffer.allocate` may throw `OutOfMemoryError` when the block is
large but no enough memory is available. However, when this happens,
right now BlockTransferService.fetchBlockSync will just hang forever as
its `BlockFetchingListener. onBlockFetchSuccess` doesn't complete
This PR catches `Throwable` and uses the error to complete `Promise`.
## How was this patch tested?
Added a unit test. Since I cannot make `ByteBuffer.allocate` throw
`OutOfMemoryError`, I passed a negative size to make
`ByteBuffer.allocate` fail. Although the error type is different, it
should trigger the same code path.
Closes #23590 from zsxwing/SPARK-26665.
Authored-by: Shixiong Zhu <> Signed-off-by: Shixiong
Zhu <>
(commit: b88067bd0f7b9466a89ce6458cb7766a24283b13)
The file was addedcore/src/test/scala/org/apache/spark/network/BlockTransferServiceSuite.scala
The file was modifiedcore/src/main/scala/org/apache/spark/network/BlockTransferService.scala (diff)
Commit 98d48c7ad2883a1dbaa81828184c3e1d76e3190a by sean.owen
[SPARK-26228][MLLIB] OOM issue encountered when computing Gramian matrix
Avoid memory problems in closure cleaning when handling large Gramians
(>= 16K rows/cols) by using null as zeroValue
Existing tests. Note that it's hard to test the case that triggers this
issue as it would require a large amount of memory and run a while. I
confirmed locally that a 16K x 16K Gramian failed with tons of driver
memory before, and didn't fail upfront after this change.
Closes #23600 from srowen/SPARK-26228.
Authored-by: Sean Owen <> Signed-off-by: Sean
Owen <>
(cherry picked from commit 6dcad38ba3393188084f378b7ff6dfc12b685b13)
Signed-off-by: Sean Owen <>
(commit: 98d48c7ad2883a1dbaa81828184c3e1d76e3190a)
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala (diff)