1. [SPARK-25837][CORE] Fix potential slowdown in AppStatusListener when (commit: 49e1eb8bdeff2ea13e235ed3a82173887c48643e) (details)
Commit 49e1eb8bdeff2ea13e235ed3a82173887c48643e by vanzin
[SPARK-25837][CORE] Fix potential slowdown in AppStatusListener when
cleaning up stages
## What changes were proposed in this pull request?
* Update `AppStatusListener` `cleanupStages` method to remove tasks for
those stages in a single pass instead of 1 for each stage.
* This fixes an issue where the cleanupStages method would get backed
up, causing a backup in the executor in ElementTrackingStore, resulting
in stages and jobs not getting cleaned up properly.
Tasks seem most susceptible to this as there are a lot of them, however
a similar issue could arise in other locations the `KVStore` `view`
method is used. A broader fix might involve updates to `KVStoreView` and
`InMemoryView` as it appears this interface and implementation can lead
to multiple and inefficient traversals of the stored data.
## How was this patch tested?
Using existing tests in AppStatusListenerSuite
This is my original work and I license the work to the project under the
project’s open source license.
Closes #22883 from patrickbrownsync/cleanup-stages-fix.
Authored-by: Patrick Brown <> Signed-off-by:
Marcelo Vanzin <>
(cherry picked from commit e9d3ca0b7993995f24f5c555a570bc2521119e12)
Signed-off-by: Marcelo Vanzin <>
(commit: 49e1eb8bdeff2ea13e235ed3a82173887c48643e)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)