1. [SPARK-27504][SQL] File source V2: support refreshing metadata cache (details)
Commit 31488e1ca506efd34459e6bc9a08b6d0956c8d44 by wenchen
[SPARK-27504][SQL] File source V2: support refreshing metadata cache
## What changes were proposed in this pull request?
In file source V1, if some file is deleted manually, reading the
DataFrame/Table will throws an exception with suggestion message
``` It is possible the underlying files have been updated. You can
explicitly invalidate the cache in Spark by running 'REFRESH TABLE
tableName' command in SQL or by recreating the Dataset/DataFrame
``` After refreshing the table/DataFrame, the reads should return
correct results.
We should follow it in file source V2 as well.
## How was this patch tested? Unit test
Closes #24401 from gengliangwang/refreshFileTable.
Authored-by: Gengliang Wang <>
Signed-off-by: Wenchen Fan <>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/MetadataCacheSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala (diff)