SuccessChanges

Summary

  1. [SPARK-25253][PYSPARK] Refactor local connection & auth code (details)
  2. [PYSPARK] Updates to pyspark broadcast (details)
  3. [PYSPARK][SQL] Updates to RowQueue (details)
  4. [CORE] Updates to remote cache reads (details)
Commit a2a54a5f49364a1825932c9f04eb0ff82dd7d465 by irashid
[SPARK-25253][PYSPARK] Refactor local connection & auth code
This eliminates some duplication in the code to connect to a server on
localhost to talk directly to the jvm.  Also it gives consistent ipv6
and error handling.  Two other incidental changes, that shouldn't
matter: 1) python barrier tasks perform authentication immediately
(rather than waiting for the BARRIER_FUNCTION indicator) 2) for
`rdd._load_from_socket`, the timeout is only increased after
authentication.
Closes #22247 from squito/py_connection_refactor.
Authored-by: Imran Rashid <irashid@cloudera.com> Signed-off-by:
hyukjinkwon <gurwls223@apache.org>
(cherry picked from commit 38391c9aa8a88fcebb337934f30298a32d91596b)
The file was modifiedpython/pyspark/java_gateway.py (diff)
The file was modifiedpython/pyspark/worker.py (diff)
The file was modifiedpython/pyspark/rdd.py (diff)
Commit 09dd34cb1706f2477a89174d6a1a0f17ed5b0a65 by irashid
[PYSPARK] Updates to pyspark broadcast
The file was modifiedpython/pyspark/context.py (diff)
The file was modifiedpython/pyspark/broadcast.py (diff)
The file was addedpython/pyspark/test_serializers.py
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRunner.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRDD.scala (diff)
The file was addedpython/pyspark/test_broadcast.py
The file was modifiedpython/pyspark/serializers.py (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/api/python/PythonRDDSuite.scala (diff)
The file was modifieddev/sparktestsupport/modules.py (diff)
The file was modifiedpython/pyspark/worker.py (diff)
Commit 6d742d1bd71aa3803dce91a830b37284cb18cf70 by irashid
[PYSPARK][SQL] Updates to RowQueue
Tested with updates to RowQueueSuite
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/python/RowQueue.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/python/RowQueueSuite.scala (diff)
Commit 575fea120e25249716e3f680396580c5f9e26b5b by irashid
[CORE] Updates to remote cache reads
Covered by tests in DistributedSuite
The file was modifiedcore/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala (diff)
The file was addedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/SimpleDownloadFile.java
The file was modifiedcore/src/main/scala/org/apache/spark/network/BlockTransferService.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/DiskStore.scala (diff)
The file was addedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/DownloadFile.java
The file was modifiedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleClient.java (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala (diff)
The file was modifiedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManager.scala (diff)
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/buffer/ManagedBuffer.java (diff)
The file was removedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/TempFileManager.java
The file was addedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/DownloadFileWritableChannel.java
The file was modifiedcore/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala (diff)
The file was modifiedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java (diff)
The file was addedcommon/network-shuffle/src/main/java/org/apache/spark/network/shuffle/DownloadFileManager.java
The file was modifiedcore/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala (diff)