1. [SPARK-24578][CORE] Cap sub-region's size of returned nio buffer (commit: 8928de3cd448922d43f9ae80cea7138ecbea0d24) (details)
Commit 8928de3cd448922d43f9ae80cea7138ecbea0d24 by zsxwing
[SPARK-24578][CORE] Cap sub-region's size of returned nio buffer
## What changes were proposed in this pull request? This PR tries to fix
the performance regression introduced by SPARK-21517.
In our production job, we performed many parallel computations, with
high possibility, some task could be scheduled to a host-2 where it
needs to read the cache block data from host-1. Often, this big transfer
makes the cluster suffer time out issue (it will retry 3 times, each
with 120s timeout, and then do recompute to put the cache block into the
local MemoryStore).
The root cause is that we don't do `consolidateIfNeeded` anymore as we
are using
``` Unpooled.wrappedBuffer(chunks.length, getChunks(): _*)
``` in ChunkedByteBuffer. If we have many small chunks, it could cause
the `buf.notBuffer(...)` have very bad performance in the case that we
have to call `copyByteBuf(...)` many times.
## How was this patch tested? Existing unit tests and also test in
Author: Wenbo Zhao <>
Closes #21593 from WenboZhao/spark-24578.
(cherry picked from commit 3f4bda7289f1bfbbe8b9bc4b516007f569c44d2e)
Signed-off-by: Shixiong Zhu <>
(commit: 8928de3cd448922d43f9ae80cea7138ecbea0d24)
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/protocol/ (diff)