SuccessChanges

Summary

  1. [SPARK-19019][PYTHON][BRANCH-1.6] Fix hijacked `collections.namedtuple` (details)
Commit 6b315f3d5fe0aea1fdd65f1d56571ead930a54c3 by holden
[SPARK-19019][PYTHON][BRANCH-1.6] Fix hijacked `collections.namedtuple`
and port cloudpickle changes for PySpark to work with Python 3.6.0
## What changes were proposed in this pull request?
This PR proposes to backports https://github.com/apache/spark/pull/16429
to branch-1.6 so that Python 3.6.0 works with Spark 1.6.x.
## How was this patch tested?
Manually, via
```
./run-tests --python-executables=python3.6
```
``` Finished test(python3.6): pyspark.conf (5s) Finished
test(python3.6): pyspark.broadcast (7s) Finished test(python3.6):
pyspark.accumulators (9s) Finished test(python3.6): pyspark.rdd (16s)
Finished test(python3.6): pyspark.shuffle (0s) Finished test(python3.6):
pyspark.serializers (11s) Finished test(python3.6): pyspark.profiler
(5s) Finished test(python3.6): pyspark.context (21s) Finished
test(python3.6): pyspark.ml.clustering (12s) Finished test(python3.6):
pyspark.ml.feature (16s) Finished test(python3.6):
pyspark.ml.classification (16s) Finished test(python3.6):
pyspark.ml.recommendation (16s) Finished test(python3.6):
pyspark.ml.tuning (14s) Finished test(python3.6): pyspark.ml.regression
(16s) Finished test(python3.6): pyspark.ml.evaluation (12s) Finished
test(python3.6): pyspark.ml.tests (17s) Finished test(python3.6):
pyspark.mllib.classification (18s) Finished test(python3.6):
pyspark.mllib.evaluation (12s) Finished test(python3.6):
pyspark.mllib.feature (19s) Finished test(python3.6):
pyspark.mllib.linalg.__init__ (0s) Finished test(python3.6):
pyspark.mllib.fpm (12s) Finished test(python3.6):
pyspark.mllib.clustering (31s) Finished test(python3.6):
pyspark.mllib.random (8s) Finished test(python3.6):
pyspark.mllib.linalg.distributed (17s) Finished test(python3.6):
pyspark.mllib.recommendation (23s) Finished test(python3.6):
pyspark.mllib.stat.KernelDensity (0s) Finished test(python3.6):
pyspark.mllib.stat._statistics (13s) Finished test(python3.6):
pyspark.mllib.regression (22s) Finished test(python3.6):
pyspark.mllib.util (9s) Finished test(python3.6): pyspark.mllib.tree
(14s) Finished test(python3.6): pyspark.sql.types (9s) Finished
test(python3.6): pyspark.sql.context (16s) Finished test(python3.6):
pyspark.sql.column (14s) Finished test(python3.6): pyspark.sql.group
(16s) Finished test(python3.6): pyspark.sql.dataframe (25s) Finished
test(python3.6): pyspark.tests (164s) Finished test(python3.6):
pyspark.sql.window (6s) Finished test(python3.6): pyspark.sql.functions
(19s) Finished test(python3.6): pyspark.streaming.util (0s) Finished
test(python3.6): pyspark.sql.readwriter (24s) Finished test(python3.6):
pyspark.sql.tests (38s) Finished test(python3.6): pyspark.mllib.tests
(133s) Finished test(python3.6): pyspark.streaming.tests (189s) Tests
passed in 380 seconds
```
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #17375 from HyukjinKwon/SPARK-19019-backport-1.6.
The file was modifiedpython/pyspark/serializers.py (diff)
The file was modifiedpython/pyspark/cloudpickle.py (diff)