1. [SPARK-24889][CORE] Update block info when unpersist rdds (details)
Commit 18688d370399dcf92f4228db6c7e3cb186804c18 by vanzin
[SPARK-24889][CORE] Update block info when unpersist rdds
We will update block info coming from executors, at the timing like
caching a RDD. However, when removing RDDs with unpersisting, we don't
ask to update block info. So the block info is not updated.
We can fix this with few options:
1. Ask to update block info when unpersisting
This is simplest but changes driver-executor communication a bit.
2. Update block info when processing the event of unpersisting RDD
We send a `SparkListenerUnpersistRDD` event when unpersisting RDD. When
processing this event, we can update block info of the RDD. This only
changes event processing code so the risk seems to be lower.
Currently this patch takes option 2 for lower risk. If we agree first
option has no risk, we can change to it.
Unit tests.
Closes #22341 from viirya/SPARK-24889.
Authored-by: Liang-Chi Hsieh <> Signed-off-by: Marcelo
Vanzin <>
(cherry picked from commit 14f3ad20932535fe952428bf255e7eddd8fa1b58)
Signed-off-by: Marcelo Vanzin <>
The file was modifiedcore/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/LiveEntity.scala (diff)