SuccessChanges

Summary

  1. [MINOR][DOCS] Fix a broken link for Arrow's supported types in the (commit: 235ec9ee74f80900c9bbfbe966856e96c947564e) (details)
  2. [SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark tests only for (commit: 52a420ff62fabcc649d7acf85a72a3c0cd6437e8) (details)
  3. [SPARK-23941][MESOS] Mesos task failed on specific spark app name (commit: 682f05d638a67b869d0e170f6c99683445e2449d) (details)
Commit 235ec9ee74f80900c9bbfbe966856e96c947564e by hyukjinkwon
[MINOR][DOCS] Fix a broken link for Arrow's supported types in the
programming guide
## What changes were proposed in this pull request?
This PR fixes a broken link for Arrow's supported types in the
programming guide.
## How was this patch tested?
Manually tested via `SKIP_API=1 jekyll watch`.
"Supported SQL Types" here in
https://spark.apache.org/docs/latest/sql-programming-guide.html#enabling-for-conversion-tofrom-pandas
is broken. It should be
https://spark.apache.org/docs/latest/sql-programming-guide.html#supported-sql-types
Author: hyukjinkwon <gurwls223@apache.org>
Closes #21191 from HyukjinKwon/minor-arrow-link.
(cherry picked from commit 56f501e1c0cec3be7d13008bd2c0182ec83ed2a2)
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
(commit: 235ec9ee74f80900c9bbfbe966856e96c947564e)
The file was modifieddocs/sql-programming-guide.md (diff)
Commit 52a420ff62fabcc649d7acf85a72a3c0cd6437e8 by hyukjinkwon
[SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark tests only for
`-Phive`
## What changes were proposed in this pull request?
When `PyArrow` or `Pandas` are not available, the corresponding PySpark
tests are skipped automatically. Currently, PySpark tests fail when we
are not using `-Phive`. This PR aims to skip Hive related PySpark tests
when `-Phive` is not given.
**BEFORE**
```bash
$ build/mvn -DskipTests clean package
$ python/run-tests.py --python-executables python2.7 --modules
pyspark-sql File
"/Users/dongjoon/spark/python/pyspark/sql/readwriter.py", line 295, in
pyspark.sql.readwriter.DataFrameReader.table
... IllegalArgumentException: u"Error while instantiating
'org.apache.spark.sql.hive.HiveExternalCatalog':"
**********************************************************************
  1 of   3 in pyspark.sql.readwriter.DataFrameReader.table
***Test Failed*** 1 failures.
```
**AFTER**
```bash
$ build/mvn -DskipTests clean package
$ python/run-tests.py --python-executables python2.7 --modules
pyspark-sql
... Tests passed in 138 seconds
Skipped tests in pyspark.sql.tests with python2.7:
...
   test_hivecontext (pyspark.sql.tests.HiveSparkSubmitTests) ... skipped
'Hive is not available.'
```
## How was this patch tested?
This is a test-only change. First, this should pass the Jenkins. Then,
manually do the following.
```bash build/mvn -DskipTests clean package python/run-tests.py
--python-executables python2.7 --modules pyspark-sql
```
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #21141 from dongjoon-hyun/SPARK-23853.
(cherry picked from commit b857fb549f3bf4e6f289ba11f3903db0a3696dec)
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
(commit: 52a420ff62fabcc649d7acf85a72a3c0cd6437e8)
The file was modifiedpython/pyspark/sql/tests.py (diff)
The file was modifiedpython/pyspark/sql/readwriter.py (diff)
Commit 682f05d638a67b869d0e170f6c99683445e2449d by vanzin
[SPARK-23941][MESOS] Mesos task failed on specific spark app name
## What changes were proposed in this pull request? Shell escaped the
name passed to spark-submit and change how conf attributes are shell
escaped.
## How was this patch tested? This test has been tested manually with
Hive-on-spark with mesos or with the use case described in the issue
with the sparkPi application with a custom name which contains illegal
shell characters.
With this PR, hive-on-spark on mesos works like a charm with hive
3.0.0-SNAPSHOT.
I state that this contribution is my original work and that I license
the work to the project under the project’s open source license
Author: Bounkong Khamphousone <bounkong.khamphousone@ebiznext.com>
Closes #21014 from tiboun/fix/SPARK-23941.
(cherry picked from commit 6782359a04356e4cde32940861bf2410ef37f445)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 682f05d638a67b869d0e170f6c99683445e2449d)
The file was modifiedresource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala (diff)