SuccessChanges

Summary

  1. [SPARK-24415][CORE] Fixed the aggregated stage metrics by retaining (details)
Commit 5b8b6b4e9e36228e993a15cab19c80e7fad43786 by tgraves
[SPARK-24415][CORE] Fixed the aggregated stage metrics by retaining
stage objects in liveStages until all tasks are complete
The problem occurs because stage object is removed from liveStages in
AppStatusListener onStageCompletion. Because of this any onTaskEnd event
received after onStageCompletion event do not update stage metrics.
The fix is to retain stage objects in liveStages until all tasks are
complete.
1. Fixed the reproducible example posted in the JIRA 2. Added unit test
Closes #22209 from ankuriitg/ankurgupta/SPARK-24415.
Authored-by: ankurgupta <ankur.gupta@cloudera.com> Signed-off-by:
Marcelo Vanzin <vanzin@cloudera.com>
(cherry picked from commit 39a02d8f75def7191c66d388729ba1721c92188d)
Signed-off-by: Thomas Graves <tgraves@apache.org>
The file was modifiedcore/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala (diff)