1. [SPARK-30416][SQL] Log a warning for deprecated SQL config in `set()` (details)
  2. [SPARK-30439][SQL] Support non-nullable column in CREATE TABLE, ADD (details)
Commit 1ffa627ffb93dc1027cb4b72f36ec9b7319f48e4 by gurwls223
[SPARK-30416][SQL] Log a warning for deprecated SQL config in `set()`
and `unset()`
### What changes were proposed in this pull request? 1. Put all
deprecated SQL configs the map `SQLConf.deprecatedSQLConfigs` with extra
info about when configs were deprecated and additional comments that
explain why a config was deprecated, what an user can use instead of it.
Here is the list of already deprecated configs:
   - spark.sql.hive.verifyPartitionPath
   - spark.sql.execution.pandas.respectSessionTimeZone
   - spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName
   - spark.sql.parquet.int64AsTimestampMillis
   - spark.sql.variable.substitute.depth
   - spark.sql.execution.arrow.enabled
   - spark.sql.execution.arrow.fallback.enabled
2. Output warning in `set()` and `unset()` about deprecated SQL configs
### Why are the changes needed? This should improve UX with Spark SQL
and notify users about already deprecated SQL configs.
### Does this PR introduce any user-facing change? Yes, before:
``` spark-sql> set spark.sql.hive.verifyPartitionPath=true;
spark.sql.hive.verifyPartitionPath true
``` After:
``` spark-sql> set spark.sql.hive.verifyPartitionPath=true; 20/01/03
21:28:17 WARN RuntimeConfig: The SQL config
'spark.sql.hive.verifyPartitionPath' has been deprecated in Spark v3.0.0
and may be removed in the future. This config is replaced by
spark.files.ignoreMissingFiles. spark.sql.hive.verifyPartitionPath true
### How was this patch tested? Add new test which registers new log
appender and catches all logging to check that `set()` and `unset()` log
any warning.
Closes #27092 from MaxGekk/group-deprecated-sql-configs.
Authored-by: Maxim Gekk <> Signed-off-by: HyukjinKwon
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
Commit 0ec0355611e7ce79599f86862a90611f7cde6227 by gurwls223
[SPARK-30439][SQL] Support non-nullable column in CREATE TABLE, ADD
### What changes were proposed in this pull request?
Allow users to specify NOT NULL in CREATE TABLE and ADD COLUMN column
definition, and add a new SQL syntax to alter column nullability: ALTER
TABLE ... ALTER COLUMN SET/DROP NOT NULL. This is a SQL standard syntax:
<alter column definition> ::=
ALTER [ COLUMN ] <column name> <alter column action>
<alter column action> ::=
   <set column default clause>
| <drop column default clause>
| <set column not null clause>
| <drop column not null clause>
| ...
<set column not null clause> ::=
<drop column not null clause> ::=
### Why are the changes needed?
Previously we don't support it because the table schema in hive catalog
are always nullable. Since we have catalog plugin now, it makes more
sense to support NOT NULL at spark side, and let catalog implementations
to decide if they support it or not.
### Does this PR introduce any user-facing change?
Yes, this is a new feature
### How was this patch tested?
new tests
Closes #27110 from cloud-fan/nullable.
Authored-by: Wenchen Fan <> Signed-off-by:
HyukjinKwon <>
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala (diff)
The file was modifiedsql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalogSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/TableCatalogSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala (diff)