1. [SPARK-22371][CORE] Return None instead of throwing an exception when an (commit: 895c95e5bde86099c0ed01edce00d5d87b192cbb) (details)
  2. [SPARK-23850][SQL] Add separate config for SQL options redaction. (commit: d88f3e474063010231a5a5d95d192c7796af01e4) (details)
Commit 895c95e5bde86099c0ed01edce00d5d87b192cbb by wenchen
[SPARK-22371][CORE] Return None instead of throwing an exception when an
accumulator is garbage collected.
## What changes were proposed in this pull request?
There's a period of time when an accumulator has been garbage collected,
but hasn't been removed from AccumulatorContext.originals by
ContextCleaner. When an update is received for such accumulator it will
throw an exception and kill the whole job. This can happen when a stage
completes, but there're still running tasks from other attempts,
speculation etc. Since AccumulatorContext.get() returns an option we can
just return None in such case.
## How was this patch tested?
Unit test.
Author: Artem Rudoy <>
Closes #21114 from artemrd/SPARK-22371.
(cherry picked from commit 6c35865d949a8b46f654cd53c7e5f3288def18d0)
Signed-off-by: Wenchen Fan <>
(commit: 895c95e5bde86099c0ed01edce00d5d87b192cbb)
The file was modifiedcore/src/main/scala/org/apache/spark/util/AccumulatorV2.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/AccumulatorSuite.scala (diff)
Commit d88f3e474063010231a5a5d95d192c7796af01e4 by vanzin
[SPARK-23850][SQL] Add separate config for SQL options redaction.
The old code was relying on a core configuration and extended its
default value to include things that redact desired things in the app's
environment. Instead, add a SQL-specific option for which options to
redact, and apply both the core and SQL-specific rules when redacting
the options in the save command.
This is a little sub-optimal since it adds another config, but it
retains the current default behavior.
While there I also fixed a typo and a couple of minor config API usage
issues in the related redaction option that SQL already had.
Tested with existing unit tests, plus checking the env page on a shell
Author: Marcelo Vanzin <>
Closes #21158 from vanzin/SPARK-23850.
(cherry picked from commit ed7ba7db8fa344ff182b72d23ae458e711f63432)
Signed-off-by: Marcelo Vanzin <>
(commit: d88f3e474063010231a5a5d95d192c7796af01e4)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommandSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)