SuccessChanges

Summary

  1. [SPARK-26201] Fix python broadcast with encryption (commit: 4ee463ac96f343afb6f61072a427cc0d30dedbfd) (details)
  2. [MINOR][DOC] Correct some document description errors (commit: 0058986de08fcf9da6480e153912dd62da92bfae) (details)
Commit 4ee463ac96f343afb6f61072a427cc0d30dedbfd by tgraves
[SPARK-26201] Fix python broadcast with encryption
## What changes were proposed in this pull request? Python with rpc and
disk encryption enabled along with a python broadcast variable and just
read the value back on the driver side the job failed with:
Traceback (most recent call last): File "broadcast.py", line 37, in
<module> words_new.value File "/pyspark.zip/pyspark/broadcast.py", line
137, in value File "pyspark.zip/pyspark/broadcast.py", line 122, in
load_from_path File "pyspark.zip/pyspark/broadcast.py", line 128, in
load EOFError: Ran out of input
To reproduce use configs: --conf spark.network.crypto.enabled=true
--conf spark.io.encryption.enabled=true
Code:
words_new = sc.broadcast(["scala", "java", "hadoop", "spark", "akka"])
words_new.value print(words_new.value)
## How was this patch tested? words_new = sc.broadcast([“scala”, “java”,
“hadoop”, “spark”, “akka”]) textFile = sc.textFile(“README.md”)
wordCounts = textFile.flatMap(lambda line: line.split()).map(lambda
word: (word + words_new.value[1], 1)).reduceByKey(lambda a, b: a+b)
count = wordCounts.count()
print(count)
words_new.value
print(words_new.value)
Closes #23166 from redsanket/SPARK-26201.
Authored-by: schintap <schintap@oath.com> Signed-off-by: Thomas Graves
<tgraves@apache.org>
(cherry picked from commit 9b23be2e95fec756066ca0ed3188c3db2602b757)
Signed-off-by: Thomas Graves <tgraves@apache.org>
(commit: 4ee463ac96f343afb6f61072a427cc0d30dedbfd)
The file was modifiedcore/src/main/scala/org/apache/spark/api/python/PythonRDD.scala (diff)
The file was modifiedpython/pyspark/broadcast.py (diff)
The file was modifiedpython/pyspark/test_broadcast.py (diff)
Commit 0058986de08fcf9da6480e153912dd62da92bfae by sean.owen
[MINOR][DOC] Correct some document description errors
Correct some document description errors.
N/A
Closes #23162 from 10110346/docerror.
Authored-by: liuxian <liu.xian3@zte.com.cn> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(cherry picked from commit 60e4239a1e3506d342099981b6e3b3b8431a203e)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 0058986de08fcf9da6480e153912dd62da92bfae)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)