1. [SPARK-22371][CORE] Return None instead of throwing an exception when an (details)
Commit 895c95e5bde86099c0ed01edce00d5d87b192cbb by wenchen
[SPARK-22371][CORE] Return None instead of throwing an exception when an
accumulator is garbage collected.
## What changes were proposed in this pull request?
There's a period of time when an accumulator has been garbage collected,
but hasn't been removed from AccumulatorContext.originals by
ContextCleaner. When an update is received for such accumulator it will
throw an exception and kill the whole job. This can happen when a stage
completes, but there're still running tasks from other attempts,
speculation etc. Since AccumulatorContext.get() returns an option we can
just return None in such case.
## How was this patch tested?
Unit test.
Author: Artem Rudoy <>
Closes #21114 from artemrd/SPARK-22371.
(cherry picked from commit 6c35865d949a8b46f654cd53c7e5f3288def18d0)
Signed-off-by: Wenchen Fan <>
The file was modifiedcore/src/test/scala/org/apache/spark/AccumulatorSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/AccumulatorV2.scala (diff)