SuccessChanges

Summary

  1. [SPARK-23434][SQL] Spark should not warn `metadata directory` for a HDFS (details)
Commit 3e48f3b9ee7645e4218ad3ff7559e578d4bd9667 by zsxwing
[SPARK-23434][SQL] Spark should not warn `metadata directory` for a HDFS
file path
## What changes were proposed in this pull request?
In a kerberized cluster, when Spark reads a file path (e.g.
`people.json`), it warns with a wrong warning message during looking up
`people.json/_spark_metadata`. The root cause of this situation is the
difference between `LocalFileSystem` and `DistributedFileSystem`.
`LocalFileSystem.exists()` returns `false`, but
`DistributedFileSystem.exists` raises
`org.apache.hadoop.security.AccessControlException`.
```scala scala> spark.version res0: String = 2.4.0-SNAPSHOT
scala>
spark.read.json("file:///usr/hdp/current/spark-client/examples/src/main/resources/people.json").show
+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+
scala> spark.read.json("hdfs:///tmp/people.json") 18/02/15 05:00:48 WARN
streaming.FileStreamSink: Error while looking for metadata directory.
18/02/15 05:00:48 WARN streaming.FileStreamSink: Error while looking for
metadata directory.
```
After this PR,
```scala scala> spark.read.json("hdfs:///tmp/people.json").show
+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+
```
## How was this patch tested?
Manual.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #20616 from dongjoon-hyun/SPARK-23434.
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala (diff)