1. [SPARK-23850][SQL] Add separate config for SQL options redaction. (details)
Commit d88f3e474063010231a5a5d95d192c7796af01e4 by vanzin
[SPARK-23850][SQL] Add separate config for SQL options redaction.
The old code was relying on a core configuration and extended its
default value to include things that redact desired things in the app's
environment. Instead, add a SQL-specific option for which options to
redact, and apply both the core and SQL-specific rules when redacting
the options in the save command.
This is a little sub-optimal since it adds another config, but it
retains the current default behavior.
While there I also fixed a typo and a couple of minor config API usage
issues in the related redaction option that SQL already had.
Tested with existing unit tests, plus checking the env page on a shell
Author: Marcelo Vanzin <>
Closes #21158 from vanzin/SPARK-23850.
(cherry picked from commit ed7ba7db8fa344ff182b72d23ae458e711f63432)
Signed-off-by: Marcelo Vanzin <>
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommandSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala (diff)