1. [SPARK-24948][SHS][BACKPORT-2.3] Delegate check access permissions to (commit: 9fb70f4586c1b5460ea261f22eb335be902753dc) (details)
Commit 9fb70f4586c1b5460ea261f22eb335be902753dc by sshao
[SPARK-24948][SHS][BACKPORT-2.3] Delegate check access permissions to
the file system
## What changes were proposed in this pull request?
In `SparkHadoopUtil. checkAccessPermission`,  we consider only basic
permissions in order to check whether a user can access a file or not.
This is not a complete check, as it ignores ACLs and other policies a
file system may apply in its internal. So this can result in returning
wrongly that a user cannot access a file (despite he actually can).
The PR proposes to delegate to the filesystem the check whether a file
is accessible or not, in order to return the right result. A caching
layer is added for performance reasons.
## How was this patch tested?
added UT
Author: Marco Gaido <>
Closes #22021 from mgaido91/SPARK-24948_2.3.
(commit: 9fb70f4586c1b5460ea261f22eb335be902753dc)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala (diff)
The file was removedcore/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala (diff)