FailedChanges

Summary

  1. [SPARK-31831][SQL][TESTS][FOLLOWUP] Put mocks for HiveSessionImplSuite (commit: 9747e8fc9d1c67a4f27f7e4ff94b5f0aca394919) (details)
  2. [SPARK-32094][PYTHON] Update cloudpickle to v1.5.0 (commit: ea9e8f365a6670972b64da9ff2342568268067e3) (details)
  3. [SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation (commit: efa70b87555840932c5ac22ce3a63e2c461a0081) (details)
  4. [SPARK-32215] Expose a (protected) /workers/kill endpoint on the (commit: ffdbbae1d465fe2c710d020de62ca1a6b0b924d9) (details)
  5. [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache (commit: 34baed8139bf2d1b0a332d123c44434f626aaa18) (details)
  6. [SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to (commit: 5daf244d0f89668007f7168809b8b83decb979eb) (details)
  7. [SPARK-32298][ML] tree models prediction optimization (commit: 3a60b4194980e95c461021b38912e4f08e6761be) (details)
  8. [SPARK-32353][TEST] Update docker/spark-test and clean up unused stuff (commit: 7dc1d8917dd01b5d5808460a5eb6e846795ab4bd) (details)
  9. [SPARK-21040][CORE] Speculate tasks which are running on decommission (commit: 0678afe393b1e4f65b70470483fe0cdb1fe139dc) (details)
Commit 9747e8fc9d1c67a4f27f7e4ff94b5f0aca394919 by kabhwan.opensource
[SPARK-31831][SQL][TESTS][FOLLOWUP] Put mocks for HiveSessionImplSuite
in hive version related subdirectories
### What changes were proposed in this pull request?
This patch fixes the build issue on Hive 1.2 profile brought by #29069,
via putting mocks for HiveSessionImplSuite in hive version related
subdirectories, so that maven build will pick up the proper source code
according to the profile.
### Why are the changes needed?
#29069 fixed the flakiness of HiveSessionImplSuite, but given the patch
relied on the default profile (Hive 2.3) it broke the build with Hive
1.2 profile. This patch addresses both Hive versions.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually confirmed the test suite via below command:
> Hive 1.2
``` build/mvn -Dtest=none
-DwildcardSuites=org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite
test -Phive-1.2 -Phadoop-2.7 -Phive-thriftserver
```
> Hive 2.3
``` build/mvn -Dtest=none
-DwildcardSuites=org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite
test -Phive-2.3 -Phadoop-3.2 -Phive-thriftserver
```
Closes #29129 from frankyin-factual/hive-tests.
Authored-by: Frank Yin <frank@factual.com> Signed-off-by: Jungtaek Lim
(HeartSaVioR) <kabhwan.opensource@gmail.com>
(commit: 9747e8fc9d1c67a4f27f7e4ff94b5f0aca394919)
The file was addedsql/hive-thriftserver/v2.3/src/test/scala/ org/apache/spark/sql/hive/thriftserver/GetCatalogsOperationMock.scala
The file was modifiedsql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveSessionImplSuite.scala (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was addedsql/hive-thriftserver/v1.2/src/test/scala/ org/apache/spark/sql/hive/thriftserver/GetCatalogsOperationMock.scala
Commit ea9e8f365a6670972b64da9ff2342568268067e3 by gurwls223
[SPARK-32094][PYTHON] Update cloudpickle to v1.5.0
### What changes were proposed in this pull request?
This PR aims to upgrade PySpark's embedded cloudpickle to the latest
cloudpickle v1.5.0 (See
https://github.com/cloudpipe/cloudpickle/blob/v1.5.0/cloudpickle/cloudpickle.py)
### Why are the changes needed?
There are many bug fixes. For example, the bug described in the JIRA:
dill unpickling fails because they define `types.ClassType`, which is
undefined in dill. This results in the following error:
``` Traceback (most recent call last):
File
"/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py",
line 279, in loads
   return dill.loads(s)
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317,
in loads
   return load(file, ignore)
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305,
in load
   obj = pik.load()
File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577,
in _load_type
   return _reverse_typemap[name] KeyError: 'ClassType'
```
See also https://github.com/cloudpipe/cloudpickle/issues/82. This was
fixed for cloudpickle 1.3.0+
(https://github.com/cloudpipe/cloudpickle/pull/337), but PySpark's
cloudpickle.py doesn't have this change yet.
More notably, now it supports C pickle implementation with Python 3.8
which hugely improve performance. This is already adopted in another
project such as Ray.
### Does this PR introduce _any_ user-facing change?
Yes, as described above, the bug fixes. Internally, users also could
leverage the fast cloudpickle backed by C pickle.
### How was this patch tested?
Jenkins will test it out.
Closes #29114 from HyukjinKwon/SPARK-32094.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
(commit: ea9e8f365a6670972b64da9ff2342568268067e3)
The file was removedpython/pyspark/cloudpickle.py
The file was modifiedLICENSE (diff)
The file was addedpython/pyspark/cloudpickle/__init__.py
The file was modifieddev/tox.ini (diff)
The file was addedpython/pyspark/cloudpickle/cloudpickle.py
The file was addedpython/pyspark/cloudpickle/compat.py
The file was modifiedpython/setup.py (diff)
The file was addedpython/pyspark/cloudpickle/cloudpickle_fast.py
The file was modifieddev/.rat-excludes (diff)
Commit efa70b87555840932c5ac22ce3a63e2c461a0081 by wenchen
[SPARK-32145][SQL][FOLLOWUP] Fix type in the error log of SparkOperation
### What changes were proposed in this pull request?
Fix typo error in the error log of SparkOperation trait, reported by
https://github.com/apache/spark/pull/28963#discussion_r454954542
### Why are the changes needed?
fix error in thrift server driver log
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Passing GitHub actions
Closes #29140 from yaooqinn/SPARK-32145-F.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(commit: efa70b87555840932c5ac22ce3a63e2c461a0081)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkOperation.scala (diff)
Commit ffdbbae1d465fe2c710d020de62ca1a6b0b924d9 by wenchen
[SPARK-32215] Expose a (protected) /workers/kill endpoint on the
MasterWebUI
### What changes were proposed in this pull request?
This PR allows an external agent to inform the Master that certain hosts
are being decommissioned.
### Why are the changes needed?
The current decommissioning is triggered by the Worker getting getting a
SIGPWR
(out of band possibly by some cleanup hook), which then informs the
Master about it. This approach may not be feasible in some environments
that cannot trigger a clean up hook on the Worker. In addition, when a
large number of worker nodes are being decommissioned then the master
will get a flood of messages.
So we add a new post endpoint `/workers/kill` on the MasterWebUI that
allows an external agent to inform the master about all the nodes being
decommissioned in bulk. The list of nodes is specified by providing a
list of hostnames. All workers on those hosts will be decommissioned.
This API is merely a new entry point into the existing decommissioning
logic. It does not change how the decommissioning request is handled in
its core.
### Does this PR introduce _any_ user-facing change?
Yes, a new endpoint `/workers/kill` is added to the MasterWebUI. By
default only requests originating from an IP address local to the
MasterWebUI are allowed.
### How was this patch tested?
Added unit tests
Closes #29015 from agrawaldevesh/master_decom_endpoint.
Authored-by: Devesh Agrawal <devesh.agrawal@gmail.com> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
(commit: ffdbbae1d465fe2c710d020de62ca1a6b0b924d9)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/Master.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/UI.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/DeployMessage.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/master/ui/MasterWebUISuite.scala (diff)
Commit 34baed8139bf2d1b0a332d123c44434f626aaa18 by wenchen
[SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache
### What changes were proposed in this pull request? New
`spark.sql.metadataCacheTTLSeconds` option that adds time-to-live cache
behaviour to the existing caches in `FileStatusCache` and
`SessionCatalog`.
### Why are the changes needed? Currently Spark [caches file listing for
tables](https://spark.apache.org/docs/2.4.4/sql-data-sources-parquet.html#metadata-refreshing)
and requires issuing `REFRESH TABLE` any time the file listing has
changed outside of Spark. Unfortunately, simply submitting `REFRESH
TABLE` commands could be very cumbersome. Assuming frequently added
files, hundreds of tables and dozens of users querying the data (and
expecting up-to-date results), manually refreshing metadata for each
table is not a solution.
This is a pretty common use-case for streaming ingestion of data, which
can be done outside of Spark (with tools like Kafka Connect, etc.).
A similar feature exists in Presto: `hive.file-status-cache-expire-time`
can be found
[here](https://prestosql.io/docs/current/connector/hive.html#hive-configuration-properties).
### Does this PR introduce _any_ user-facing change? Yes, it's
controlled with the new `spark.sql.metadataCacheTTLSeconds` option.
When it's set to `-1` (by default), the behaviour of caches doesn't
change, so it stays _backwards-compatible_.
Otherwise, you can specify a value in seconds, for example
`spark.sql.metadataCacheTTLSeconds: 60` means 1-minute cache TTL.
### How was this patch tested?
Added new tests in:
- FileIndexSuite
- SessionCatalogSuite
Closes #28852 from sap1ens/SPARK-30616-metadata-cache-ttl.
Authored-by: Yaroslav Tkachenko <sapiensy@gmail.com> Signed-off-by:
Wenchen Fan <wenchen@databricks.com>
(commit: 34baed8139bf2d1b0a332d123c44434f626aaa18)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala (diff)
Commit 5daf244d0f89668007f7168809b8b83decb979eb by srowen
[SPARK-32329][TESTS] Rename HADOOP2_MODULE_PROFILES to
HADOOP_MODULE_PROFILES
### What changes were proposed in this pull request?
This PR aims to rename `HADOOP2_MODULE_PROFILES` to
`HADOOP_MODULE_PROFILES` because Hadoop 3 is now the default.
### Why are the changes needed?
Hadoop 3 is now the default.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GitHub Action dependency test.
Closes #29128 from williamhyun/williamhyun-patch-3.
Authored-by: williamhyun <62487364+williamhyun@users.noreply.github.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(commit: 5daf244d0f89668007f7168809b8b83decb979eb)
The file was modifieddev/test-dependencies.sh (diff)
Commit 3a60b4194980e95c461021b38912e4f08e6761be by srowen
[SPARK-32298][ML] tree models prediction optimization
### What changes were proposed in this pull request? use while-loop
instead of the recursive way
### Why are the changes needed? 3% ~ 10% faster
### Does this PR introduce _any_ user-facing change? No
### How was this patch tested? existing testsuites
Closes #29095 from zhengruifeng/tree_pred_opt.
Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by: Sean
Owen <srowen@gmail.com>
(commit: 3a60b4194980e95c461021b38912e4f08e6761be)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/tree/Node.scala (diff)
Commit 7dc1d8917dd01b5d5808460a5eb6e846795ab4bd by dongjoon
[SPARK-32353][TEST] Update docker/spark-test and clean up unused stuff
### What changes were proposed in this pull request? This PR aims to
update the docker/spark-test and clean up unused stuff.
### Why are the changes needed? Since Spark 3.0.0, Java 11 is supported.
We had better use the latest Java and OS.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested? Manually do the following as described in
https://github.com/apache/spark/blob/master/external/docker/spark-test/README.md
.
``` docker run -v $SPARK_HOME:/opt/spark spark-test-master docker run -v
$SPARK_HOME:/opt/spark spark-test-worker spark://<master_ip>:7077
```
Closes #29150 from williamhyun/docker.
Authored-by: William Hyun <williamhyun3@gmail.com> Signed-off-by:
Dongjoon Hyun <dongjoon@apache.org>
(commit: 7dc1d8917dd01b5d5808460a5eb6e846795ab4bd)
The file was modifiedexternal/docker/spark-test/master/default_cmd (diff)
The file was modifiedexternal/docker/spark-test/base/Dockerfile (diff)
The file was modifiedexternal/docker/spark-test/worker/default_cmd (diff)
Commit 0678afe393b1e4f65b70470483fe0cdb1fe139dc by hkarau
[SPARK-21040][CORE] Speculate tasks which are running on decommission
executors
### What changes were proposed in this pull request? This PR adds
functionality to consider the running tasks on decommission executors
based on some config. In spark-on-cloud , we sometimes already know that
an executor won't be alive for more than fix amount of time. Ex- In AWS
Spot nodes, once we get the notification, we know that a node will be
gone in 120 seconds. So if the running tasks on the decommissioning
executors may run beyond currentTime+120 seconds, then they are
candidate for speculation.
### Why are the changes needed? Currently when an executor is
decommission, we stop scheduling new tasks on those executors but the
already running tasks keeps on running on them. Based on the cloud, we
might know beforehand that an executor won't be alive for more than a
preconfigured time. Different cloud providers gives different timeouts
before they take away the nodes. For Ex- In case of AWS spot nodes, an
executor won't be alive for more than 120 seconds. We can utilize this
information in cloud environments and take better decisions about
speculating the already running tasks on decommission executors.
### Does this PR introduce _any_ user-facing change? Yes. This PR adds a
new config "spark.executor.decommission.killInterval" which they can
explicitly set based on the cloud environment where they are running.
### How was this patch tested? Added UT.
Closes #28619 from
prakharjain09/SPARK-21040-speculate-decommission-exec-tasks.
Authored-by: Prakhar Jain <prakharjain09@gmail.com> Signed-off-by:
Holden Karau <hkarau@apple.com>
(commit: 0678afe393b1e4f65b70470483fe0cdb1fe139dc)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala (diff)