1. [SPARK-27403][SQL] Fix `updateTableStats` to update table stats always (details)
Commit 2d276c0f0707ab6e84bd49dae6629c4073849951 by dhyun
[SPARK-27403][SQL] Fix `updateTableStats` to update table stats always
with new stats or None
## What changes were proposed in this pull request?
System shall update the table stats automatically if user set
spark.sql.statistics.size.autoUpdate.enabled as true, currently this
property is not having any significance even if it is enabled or
disabled. This feature is similar to Hives auto-gather feature where
statistics are automatically computed by default if this feature is
enabled. Reference:
As part of fix , autoSizeUpdateEnabled  validation is been done
initially so that system will calculate the table size for the user
automatically and record it in metastore as per user expectation.
## How was this patch tested? UT is written and manually verified in
cluster. Tested with unit tests + some internal tests on real cluster.
Before fix:
After fix
Closes #24315 from sujith71955/master_autoupdate.
Authored-by: s71955 <> Signed-off-by:
Dongjoon Hyun <>
(cherry picked from commit 239082d9667a4fa4198bd9524d63c739df147e0e)
Signed-off-by: Dongjoon Hyun <>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala (diff)