SuccessChanges

Summary

  1. [SPARK-27275][CORE] Fix potential corruption in (commit: 530ec5247ccbe049cea8747195fc2011e71ad0f9) (details)
  2. [SPARK-27244][CORE] Redact Passwords While Using Option logConf=true (commit: 0975fe9f7e6ed4154988b8df1487ee90976eef49) (details)
  3. [SPARK-27301][DSTREAM] Shorten the FileSystem cached life cycle to the (commit: b57fef9f0cb29fb4bc4bb905a84fecb3a1ed5ddd) (details)
Commit 530ec5247ccbe049cea8747195fc2011e71ad0f9 by wenchen
[SPARK-27275][CORE] Fix potential corruption in
EncryptedMessage.transferTo (2.4)
## What changes were proposed in this pull request?
Backport https://github.com/apache/spark/pull/24211 to 2.4
## How was this patch tested?
Jenkins
Closes #24229 from zsxwing/SPARK-27275-2.4.
Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(cherry picked from commit 298e4fa6f8054c54e246f91b70d62174ccdb9413)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: 530ec5247ccbe049cea8747195fc2011e71ad0f9)
The file was modifiedcommon/network-common/src/test/java/org/apache/spark/network/crypto/AuthEngineSuite.java (diff)
The file was modifiedcommon/network-common/src/main/java/org/apache/spark/network/crypto/TransportCipher.java (diff)
The file was modifiedcommon/network-common/src/test/java/org/apache/spark/network/crypto/AuthIntegrationSuite.java (diff)
Commit 0975fe9f7e6ed4154988b8df1487ee90976eef49 by sean.owen
[SPARK-27244][CORE] Redact Passwords While Using Option logConf=true
When logConf is set to true, config keys that contain password were
printed in cleartext in driver log. This change uses the already present
redact method in Utils, to redact all the passwords based on redact
pattern in SparkConf and then print the conf to driver log thus ensuring
that sensitive information like passwords is not printed in clear text.
This patch was tested through `SparkConfSuite` & then entire unit test
through sbt
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Closes #24196 from ninadingole/SPARK-27244.
Authored-by: Ninad Ingole <robert.wallis@example.com> Signed-off-by:
Sean Owen <sean.owen@databricks.com>
(cherry picked from commit dbc7ce18b934fbfd0743b1348fc1265778f07027)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 0975fe9f7e6ed4154988b8df1487ee90976eef49)
The file was modifiedcore/src/main/scala/org/apache/spark/SparkConf.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/SparkConfSuite.scala (diff)
Commit b57fef9f0cb29fb4bc4bb905a84fecb3a1ed5ddd by sean.owen
[SPARK-27301][DSTREAM] Shorten the FileSystem cached life cycle to the
cleanup method inner scope
## What changes were proposed in this pull request?
The cached FileSystem's token will expire if no tokens explicitly are
add into it.
```scala 19/03/28 13:40:16 INFO storage.BlockManager: Removing RDD 83189
19/03/28 13:40:16 INFO rdd.MapPartitionsRDD: Removing RDD 82860 from
persistence list 19/03/28 13:40:16 INFO spark.ContextCleaner: Cleaned
shuffle 6005 19/03/28 13:40:16 INFO storage.BlockManager: Removing RDD
82860 19/03/28 13:40:16 INFO scheduler.ReceivedBlockTracker: Deleting
batches: 19/03/28 13:40:16 INFO scheduler.InputInfoTracker: remove old
batch metadata: 1553750250000 ms 19/03/28 13:40:17 WARN
security.UserGroupInformation: PriviledgedActionException
as:ursHADOOP.HZ.NETEASE.COM (auth:KERBEROS)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 53240500 for urs) is expired, current
time: 2019-03-28 13:40:17,010+0800 expected renewal time: 2019-03-28
13:39:48,523+0800 19/03/28 13:40:17 WARN ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 53240500 for urs) is expired, current
time: 2019-03-28 13:40:17,010+0800 expected renewal time: 2019-03-28
13:39:48,523+0800 19/03/28 13:40:17 WARN security.UserGroupInformation:
PriviledgedActionException as:ursHADOOP.HZ.NETEASE.COM (auth:KERBEROS)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 53240500 for urs) is expired, current
time: 2019-03-28 13:40:17,010+0800 expected renewal time: 2019-03-28
13:39:48,523+0800 19/03/28 13:40:17 WARN hdfs.LeaseRenewer: Failed to
renew lease for [DFSClient_NONMAPREDUCE_-1396157959_1] for 53 seconds.
Will retry shortly ...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 53240500 for urs) is expired, current
time: 2019-03-28 13:40:17,010+0800 expected renewal time: 2019-03-28
13:39:48,523+0800 at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399) at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy11.renewLease(Unknown Source) at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.renewLease(Unknown Source) at
org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878) at
org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at
org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at
org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at
java.lang.Thread.run(Thread.java:748)
```
This PR shorten the FileSystem cached life cycle to the cleanup method
inner scope in case of token expiry.
## How was this patch tested?
existing ut
Closes #24235 from yaooqinn/SPARK-27301.
Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(cherry picked from commit f4c73b7c685b901dd69950e4929c65e3b8dd3a55)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: b57fef9f0cb29fb4bc4bb905a84fecb3a1ed5ddd)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala (diff)