1. [SPARK-26734][STREAMING] Fix StackOverflowError with large block queue (commit: 38ade4268b585ebc750ccdee77b6e57d2f9e5820) (details)
Commit 38ade4268b585ebc750ccdee77b6e57d2f9e5820 by sean.owen
[SPARK-26734][STREAMING] Fix StackOverflowError with large block queue
## What changes were proposed in this pull request?
SPARK-23991 introduced a bug in
`ReceivedBlockTracker#allocateBlocksToBatch`: when a queue with more
than a few thousand blocks are in the queue, serializing the queue
throws a StackOverflowError.  This change just adds `dequeueAll` to the
new `clone` operation on the queue so that the fix in 23991 is preserved
but the serialized data comes from an ArrayBuffer which doesn't have the
serialization problems that mutable.Queue has.
## How was this patch tested?
A unit test was added.
Closes #23716 from rlodge/SPARK-26734.
Authored-by: Ross Lodge <> Signed-off-by: Sean
Owen <>
(cherry picked from commit 8427e9ba5cae28233d1bdc54208b46889b83a821)
Signed-off-by: Sean Owen <>
(commit: 38ade4268b585ebc750ccdee77b6e57d2f9e5820)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala (diff)