1. [SPARK-24062][THRIFT SERVER] Fix SASL encryption cannot enabled issue in (commit: 07ec75ca00a8ad6608ec577e771fa4d2651a9661) (details)
Commit 07ec75ca00a8ad6608ec577e771fa4d2651a9661 by sshao
[SPARK-24062][THRIFT SERVER] Fix SASL encryption cannot enabled issue in
thrift server
## What changes were proposed in this pull request?
For the details of the exception please see
The issue is:
Spark on Yarn stores SASL secret in current UGI's credentials, this
credentials will be distributed to AM and executors, so that executors
and drive share the same secret to communicate. But STS/Hive library
code will refresh the current UGI by UGI's loginFromKeytab() after Spark
application is started, this will create a new UGI in the current
driver's context with empty tokens and secret keys, so secret key is
lost in the current context's UGI, that's why Spark driver throws secret
key not found exception.
In Spark 2.2 code, Spark also stores this secret key in
SecurityManager's class variable, so even UGI is refreshed, the secret
is still existed in the object, so STS with SASL can still be worked in
Spark 2.2. But in Spark 2.3, we always search key from current UGI,
which makes it fail to work in Spark 2.3.
To fix this issue, there're two possible solutions:
1. Fix in STS/Hive library, when a new UGI is refreshed, copy the secret
key from original UGI to the new one. The difficulty is that some codes
to refresh the UGI is existed in Hive library, which makes us hard to
change the code. 2. Roll back the logics in SecurityManager to match
Spark 2.2, so that this issue can be fixed.
2nd solution seems a simple one. So I will propose a PR with 2nd
## How was this patch tested?
Verified in local cluster.
CC vanzin  tgravescs  please help to review. Thanks!
Author: jerryshao <>
Closes #21138 from jerryshao/SPARK-24062.
(cherry picked from commit ffaf0f9fd407aeba7006f3d785ea8a0e51187357)
Signed-off-by: jerryshao <>
(commit: 07ec75ca00a8ad6608ec577e771fa4d2651a9661)
The file was modifiedcore/src/main/scala/org/apache/spark/SecurityManager.scala (diff)