SuccessChanges

Summary

  1. [SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui (details)
  2. [SPARK-26551][SQL] Fix schema pruning error when selecting one complex (details)
  3. [SPARK-26586][SS] Fix race condition that causes streams to run with (details)
Commit d9e4cf67c06b2d6daa4cd24b056e33dfb5eb35f5 by vanzin
[SPARK-26482][CORE] Use ConfigEntry for hardcoded configs for ui
categories
## What changes were proposed in this pull request?
The PR makes hardcoded configs below to use `ConfigEntry`.
* spark.ui
* spark.ssl
* spark.authenticate
* spark.master.rest
* spark.master.ui
* spark.metrics
* spark.admin
* spark.modify.acl
This patch doesn't change configs which are not relevant to SparkConf
(e.g. system properties).
## How was this patch tested?
Existing tests.
Closes #23423 from HeartSaVioR/SPARK-26466.
Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com>
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
The file was modifiedcore/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala (diff)
The file was modifiedresource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/JettyUtils.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/SparkContext.scala (diff)
The file was modifiedresource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/metrics/MetricsSystemSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/metrics/MetricsConfigSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ui/HttpSecurityFilterSuite.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala (diff)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/SparkContextSuite.scala (diff)
The file was addedcore/src/main/scala/org/apache/spark/internal/config/UI.scala
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/SecurityManagerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/ConsoleProgressBar.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/worker/Worker.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/SparkUI.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/Utils.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/Master.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/SecurityManager.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/CheckpointSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferSecuritySuite.scala (diff)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala (diff)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ui/UISuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/History.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ConcurrentHiveSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala (diff)
Commit 50ebf3a43b84c8538ec60437189221c2c527990b by d_tsai
[SPARK-26551][SQL] Fix schema pruning error when selecting one complex
field and having is not null predicate on another one
## What changes were proposed in this pull request?
Schema pruning has errors when selecting one complex field and having is
not null predicate on another one:
```scala val query = sql("select * from contacts")
.where("name.middle is not null")
.select(
   "id",
   "name.first",
   "name.middle",
   "name.last"
)
.where("last = 'Jones'")
.select(count("id"))
```
``` java.lang.IllegalArgumentException: middle does not exist.
Available: last
[info]   at
org.apache.spark.sql.types.StructType.$anonfun$fieldIndex$1(StructType.scala:303)
[info]   at scala.collection.immutable.Map$Map1.getOrElse(Map.scala:119)
[info]   at
org.apache.spark.sql.types.StructType.fieldIndex(StructType.scala:302)
[info]   at
org.apache.spark.sql.execution.ProjectionOverSchema.$anonfun$getProjection$6(ProjectionOverSchema.scala:58)
[info]   at scala.Option.map(Option.scala:163)
[info]   at
org.apache.spark.sql.execution.ProjectionOverSchema.getProjection(ProjectionOverSchema.scala:56)
[info]   at
org.apache.spark.sql.execution.ProjectionOverSchema.unapply(ProjectionOverSchema.scala:32)
[info]   at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaPruning$$anonfun$$nestedInanonfun$buildNewProjection$1$1.applyOrElse(Parque
tSchemaPruning.scala:153)
```
## How was this patch tested?
Added tests.
Closes #23474 from viirya/SPARK-26551.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: DB Tsai
<d_tsai@apple.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala (diff)
Commit ae382c94dd10ff494dde4de44e66182bf6dbe8f8 by zsxwing
[SPARK-26586][SS] Fix race condition that causes streams to run with
unexpected confs
## What changes were proposed in this pull request?
Fix race condition where streams can have unexpected conf values.
New streaming queries should run with isolated SparkSessions so that
they aren't affected by conf updates after they are started. In
StreamExecution, the parent SparkSession is cloned and used to run each
batch, but this cloning happens in a separate thread and may happen
after DataStreamWriter.start() returns. If a stream is started and a
conf key is set immediately after, the stream is likely to have the new
value.
## How was this patch tested?
New unit test that fails prior to the production change and passes with
it.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Closes #23513 from mukulmurthy/26586.
Authored-by: Mukul Murthy <mukul.murthy@gmail.com> Signed-off-by:
Shixiong Zhu <zsxwing@gmail.com>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala (diff)