1. [SPARK-25674][FOLLOW-UP] Update the stats for each ColumnarBatch (details)
Commit 4cee191c04f14d7272347e4b29201763c6cfb6bf by wenchen
[SPARK-25674][FOLLOW-UP] Update the stats for each ColumnarBatch
## What changes were proposed in this pull request? This PR is a
follow-up of . This
alternative can avoid the unneeded computation in the hot code path.
- For row-based scan, we keep the original way.
- For the columnar scan, we just need to update the stats after each
## How was this patch tested? N/A
Closes #22731 from gatorsmile/udpateStatsFileScanRDD.
Authored-by: gatorsmile <> Signed-off-by: Wenchen
Fan <>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala (diff)