1. [SPARK-23438][DSTREAMS] Fix DStreams data loss with WAL when driver (commit: 1f180cd121b13ecd455bee55ed2224936a2f3b2a) (details)
  2. [SPARK-23449][K8S] Preserve extraJavaOptions ordering (commit: 6eee545f9ad4552057bb51daa866d68b08f27c0f) (details)
Commit 1f180cd121b13ecd455bee55ed2224936a2f3b2a by vanzin
[SPARK-23438][DSTREAMS] Fix DStreams data loss with WAL when driver
There is a race condition introduced in SPARK-11141 which could cause
data loss. The problem is that ReceivedBlockTracker.insertAllocatedBatch
function assumes that all blocks from streamIdToUnallocatedBlockQueues
allocated to the batch and clears the queue.
In this PR only the allocated blocks will be removed from the queue
which will prevent data loss.
Additional unit test + manually.
Author: Gabor Somogyi <>
Closes #20620 from gaborgsomogyi/SPARK-23438.
(cherry picked from commit b308182f233b8840dfe0e6b5736d2f2746f40757)
Signed-off-by: Marcelo Vanzin <>
(commit: 1f180cd121b13ecd455bee55ed2224936a2f3b2a)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala (diff)
Commit 6eee545f9ad4552057bb51daa866d68b08f27c0f by vanzin
[SPARK-23449][K8S] Preserve extraJavaOptions ordering
For some JVM options, like `-XX:+UnlockExperimentalVMOptions` ordering
is necessary.
## What changes were proposed in this pull request?
Keep original `extraJavaOptions` ordering, when passing them through
environment variables inside the Docker container.
## How was this patch tested?
Ran base branch a couple of times and checked startup command in logs.
Ordering differed every time. Added sorting, ordering was consistent to
what user had in `extraJavaOptions`.
Author: Andrew Korzhuev <>
Closes #20628 from andrusha/patch-2.
(cherry picked from commit 185f5bc7dd52cebe8fac9393ecb2bd0968bc5867)
Signed-off-by: Marcelo Vanzin <>
(commit: 6eee545f9ad4552057bb51daa866d68b08f27c0f)
The file was modifiedresource-managers/kubernetes/docker/src/main/dockerfiles/spark/ (diff)