SuccessChanges

Summary

  1. [SPARK-23806] Broadcast.unpersist can cause fatal exception when used… (commit: 38c0bd7db86ca2b7e167b89338028863bcc26906) (details)
  2. [SPARK-23785][LAUNCHER] LauncherBackend doesn't check state of (commit: 0bfbcaf6696570b74923047266b00ba4dc2ba97c) (details)
  3. [SPARK-23639][SQL] Obtain token before init metastore client in SparkSQL (commit: 516304521a9bd2632bd1a42c9161b1216a8ead92) (details)
Commit 38c0bd7db86ca2b7e167b89338028863bcc26906 by wenchen
[SPARK-23806] Broadcast.unpersist can cause fatal exception when used…
… with dynamic allocation
## What changes were proposed in this pull request?
ignore errors when you are waiting for a broadcast.unpersist. This is
handling it the same way as doing rdd.unpersist in
https://issues.apache.org/jira/browse/SPARK-22618
## How was this patch tested?
Patch was tested manually against a couple jobs that exhibit this
behavior, with the change the application no longer dies due to this and
just prints the warning.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: Thomas Graves <tgraves@unharmedunarmed.corp.ne1.yahoo.com>
Closes #20924 from tgravescs/SPARK-23806.
(cherry picked from commit 641aec68e8167546dbb922874c086c9b90198f08)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: 38c0bd7db86ca2b7e167b89338028863bcc26906)
The file was modifiedcore/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala (diff)
Commit 0bfbcaf6696570b74923047266b00ba4dc2ba97c by vanzin
[SPARK-23785][LAUNCHER] LauncherBackend doesn't check state of
connection before setting state
## What changes were proposed in this pull request?
Changed `LauncherBackend` `set` method so that it checks if the
connection is open or not before writing to it (uses `isConnected`).
## How was this patch tested?
None
Author: Sahil Takiar <stakiar@cloudera.com>
Closes #20893 from sahilTakiar/master.
(cherry picked from commit 491ec114fd3886ebd9fa29a482e3d112fb5a088c)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 0bfbcaf6696570b74923047266b00ba4dc2ba97c)
The file was modifiedcore/src/main/scala/org/apache/spark/launcher/LauncherBackend.scala (diff)
The file was modifiedlauncher/src/test/java/org/apache/spark/launcher/LauncherServerSuite.java (diff)
Commit 516304521a9bd2632bd1a42c9161b1216a8ead92 by vanzin
[SPARK-23639][SQL] Obtain token before init metastore client in SparkSQL
CLI
## What changes were proposed in this pull request?
In SparkSQLCLI, SessionState generates before SparkContext
instantiating. When we use --proxy-user to impersonate, it's unable to
initializing a metastore client to talk to the secured metastore for no
kerberos ticket.
This PR use real user ugi to obtain token for owner before talking to
kerberized metastore.
## How was this patch tested?
Manually verified with kerberized hive metasotre / hdfs.
Author: Kent Yao <yaooqinn@hotmail.com>
Closes #20784 from yaooqinn/SPARK-23639.
(cherry picked from commit a7755fd8ce2f022118b9827aaac7d5d59f0f297a)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 516304521a9bd2632bd1a42c9161b1216a8ead92)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/security/HiveDelegationTokenProvider.scala (diff)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala (diff)