SuccessChanges

Summary

  1. [SPARK-24530][PYTHON] Add a control to force Python version in Sphinx (commit: 19542f5de390f6096745e478dd9339958db899e8) (details)
  2. Preparing Spark release v2.3.2-rc2 (commit: 307499e1a99c6ad3ce0b978626894ea2c1e3807e) (details)
  3. Preparing development version 2.3.3-SNAPSHOT (commit: 86457a16de20eb126d0569d12f51b2e427fd03c3) (details)
  4. [SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas (commit: 32429256f3e659c648462e5b2740747645740c97) (details)
Commit 19542f5de390f6096745e478dd9339958db899e8 by hyukjinkwon
[SPARK-24530][PYTHON] Add a control to force Python version in Sphinx
via environment variable, SPHINXPYTHON
## What changes were proposed in this pull request?
This PR proposes to add `SPHINXPYTHON` environment variable to control
the Python version used by Sphinx.
The motivation of this environment variable is, it seems not properly
rendering some signatures in the Python documentation when Python 2 is
used by Sphinx. See the JIRA's case. It should be encouraged to use
Python 3, but looks we will probably live with this problem for a long
while in any event.
For the default case of `make html`, it keeps previous behaviour and use
`SPHINXBUILD` as it was. If `SPHINXPYTHON` is set, then it forces Sphinx
to use the specific Python version.
```
$ SPHINXPYTHON=python3 make html python3 -msphinx -b html -d
_build/doctrees   . _build/html Running Sphinx v1.7.5
...
```
1. if `SPHINXPYTHON` is set, use Python. If `SPHINXBUILD` is set, use
sphinx-build. 2. If both are set, `SPHINXBUILD` has a higher priority
over `SPHINXPYTHON` 3. By default, `SPHINXBUILD` is used as
'sphinx-build'.
Probably, we can somehow work around this via explicitly setting
`SPHINXBUILD` but `sphinx-build` can't be easily distinguished since it
(at least in my environment and up to my knowledge) doesn't replace
`sphinx-build` when newer Sphinx is installed in different Python
version. It confuses and doesn't warn for its Python version.
## How was this patch tested?
Manually tested:
**`python` (Python 2.7) in the path with Sphinx:**
```
$ make html sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v1.7.5
...
```
**`python` (Python 2.7) in the path without Sphinx:**
```
$ make html Makefile:8: *** The 'sphinx-build' command was not found.
Make sure you have Sphinx installed, then set the SPHINXBUILD
environment variable to point to the full path of the 'sphinx-build'
executable. Alternatively you can add the directory with the executable
to your PATH. If you don't have Sphinx installed, grab it from
http://sphinx-doc.org/.  Stop.
```
**`SPHINXPYTHON` set `python` (Python 2.7)  with Sphinx:**
```
$ SPHINXPYTHON=python make html Makefile:35: *** Note that Python 3 is
required to generate PySpark documentation correctly for now. Current
Python executable was less than Python 3. See SPARK-24530. To force
Sphinx to use a specific Python executable, please set SPHINXPYTHON to
point to the Python 3 executable..  Stop.
```
**`SPHINXPYTHON` set `python` (Python 2.7)  without Sphinx:**
```
$ SPHINXPYTHON=python make html Makefile:35: *** Note that Python 3 is
required to generate PySpark documentation correctly for now. Current
Python executable was less than Python 3. See SPARK-24530. To force
Sphinx to use a specific Python executable, please set SPHINXPYTHON to
point to the Python 3 executable..  Stop.
```
**`SPHINXPYTHON` set `python3` with Sphinx:**
```
$ SPHINXPYTHON=python3 make html python3 -msphinx -b html -d
_build/doctrees   . _build/html Running Sphinx v1.7.5
...
```
**`SPHINXPYTHON` set `python3` without Sphinx:**
```
$ SPHINXPYTHON=python3 make html Makefile:39: *** Python executable
'python3' did not have Sphinx installed. Make sure you have Sphinx
installed, then set the SPHINXPYTHON environment variable to point to
the Python executable having Sphinx installed. If you don't have Sphinx
installed, grab it from http://sphinx-doc.org/.  Stop.
```
**`SPHINXBUILD` set:**
```
$ SPHINXBUILD=sphinx-build make html sphinx-build -b html -d
_build/doctrees   . _build/html Running Sphinx v1.7.5
...
```
**Both `SPHINXPYTHON` and `SPHINXBUILD` are set:**
```
$ SPHINXBUILD=sphinx-build SPHINXPYTHON=python make html sphinx-build -b
html -d _build/doctrees   . _build/html Running Sphinx v1.7.5
...
```
Author: hyukjinkwon <gurwls223@apache.org>
Closes #21659 from HyukjinKwon/SPARK-24530.
(cherry picked from commit 1f94bf492c3bce3b61f7fec6132b50e06dea94a8)
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
(commit: 19542f5de390f6096745e478dd9339958db899e8)
The file was modifiedpython/docs/Makefile (diff)
The file was modifieddocs/_config.yml (diff)
The file was modifiedexternal/kinesis-asl-assembly/pom.xml (diff)
The file was modifiedsql/core/pom.xml (diff)
The file was modifiedexamples/pom.xml (diff)
The file was modifiedcommon/sketch/pom.xml (diff)
The file was modifiedmllib-local/pom.xml (diff)
The file was modifiedpython/pyspark/version.py (diff)
The file was modifiedcommon/network-shuffle/pom.xml (diff)
The file was modifiedhadoop-cloud/pom.xml (diff)
The file was modifiedrepl/pom.xml (diff)
The file was modifiedgraphx/pom.xml (diff)
The file was modifiedcommon/unsafe/pom.xml (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was modifiedcommon/network-common/pom.xml (diff)
The file was modifiedcore/pom.xml (diff)
The file was modifiedexternal/flume-sink/pom.xml (diff)
The file was modifiedassembly/pom.xml (diff)
The file was modifiedsql/hive/pom.xml (diff)
The file was modifiedexternal/spark-ganglia-lgpl/pom.xml (diff)
The file was modifiedtools/pom.xml (diff)
The file was modifiedexternal/kafka-0-8-assembly/pom.xml (diff)
The file was modifiedexternal/kafka-0-10/pom.xml (diff)
The file was modifiedmllib/pom.xml (diff)
The file was modifiedlauncher/pom.xml (diff)
The file was modifiedcommon/kvstore/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-assembly/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-sql/pom.xml (diff)
The file was modifiedcommon/network-yarn/pom.xml (diff)
The file was modifiedexternal/kafka-0-8/pom.xml (diff)
The file was modifiedR/pkg/DESCRIPTION (diff)
The file was modifiedresource-managers/yarn/pom.xml (diff)
The file was modifiedresource-managers/mesos/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/pom.xml (diff)
The file was modifiedexternal/kinesis-asl/pom.xml (diff)
The file was modifiedexternal/docker-integration-tests/pom.xml (diff)
The file was modifiedexternal/flume-assembly/pom.xml (diff)
The file was modifiedsql/catalyst/pom.xml (diff)
The file was modifiedcommon/tags/pom.xml (diff)
The file was modifiedexternal/flume/pom.xml (diff)
The file was modifiedstreaming/pom.xml (diff)
The file was modifiedpom.xml (diff)
The file was modifiedlauncher/pom.xml (diff)
The file was modifiedhadoop-cloud/pom.xml (diff)
The file was modifiedassembly/pom.xml (diff)
The file was modifiedexternal/docker-integration-tests/pom.xml (diff)
The file was modifiedcommon/network-shuffle/pom.xml (diff)
The file was modifiedrepl/pom.xml (diff)
The file was modifiedresource-managers/mesos/pom.xml (diff)
The file was modifiedexternal/kinesis-asl/pom.xml (diff)
The file was modifiedgraphx/pom.xml (diff)
The file was modifiedsql/catalyst/pom.xml (diff)
The file was modifiedexternal/kinesis-asl-assembly/pom.xml (diff)
The file was modifiedpython/pyspark/version.py (diff)
The file was modifiedcommon/tags/pom.xml (diff)
The file was modifiedR/pkg/DESCRIPTION (diff)
The file was modifieddocs/_config.yml (diff)
The file was modifiedexamples/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-sql/pom.xml (diff)
The file was modifiedexternal/flume-assembly/pom.xml (diff)
The file was modifiedexternal/flume/pom.xml (diff)
The file was modifiedmllib/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/pom.xml (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was modifiedsql/core/pom.xml (diff)
The file was modifiedresource-managers/yarn/pom.xml (diff)
The file was modifiedexternal/spark-ganglia-lgpl/pom.xml (diff)
The file was modifiedmllib-local/pom.xml (diff)
The file was modifiedcommon/unsafe/pom.xml (diff)
The file was modifiedcommon/network-yarn/pom.xml (diff)
The file was modifiedcore/pom.xml (diff)
The file was modifiedcommon/kvstore/pom.xml (diff)
The file was modifiedcommon/network-common/pom.xml (diff)
The file was modifiedpom.xml (diff)
The file was modifiedstreaming/pom.xml (diff)
The file was modifiedexternal/flume-sink/pom.xml (diff)
The file was modifiedexternal/kafka-0-10/pom.xml (diff)
The file was modifiedsql/hive/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-assembly/pom.xml (diff)
The file was modifiedcommon/sketch/pom.xml (diff)
The file was modifiedtools/pom.xml (diff)
The file was modifiedexternal/kafka-0-8-assembly/pom.xml (diff)
The file was modifiedexternal/kafka-0-8/pom.xml (diff)
Commit 32429256f3e659c648462e5b2740747645740c97 by gatorsmile
[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas
A self-join on a dataset which contains a `FlatMapGroupsInPandas` fails
because of duplicate attributes. This happens because we are not dealing
with this specific case in our `dedupAttr` rules.
The PR fix the issue by adding the management of the specific case
added UT + manual tests
Author: Marco Gaido <marcogaido91@gmail.com> Author: Marco Gaido
<mgaido@hortonworks.com>
Closes #21737 from mgaido91/SPARK-24208.
(cherry picked from commit ebf4bfb966389342bfd9bdb8e3b612828c18730c)
Signed-off-by: Xiao Li <gatorsmile@gmail.com>
(commit: 32429256f3e659c648462e5b2740747645740c97)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/GroupedDatasetSuite.scala (diff)
The file was modifiedpython/pyspark/sql/tests.py (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)