SuccessChanges

Summary

  1. [SPARK-19613][SS][TEST] Random.nextString is not safe for directory (commit: fec43fe1bcd7553eff077805a13bb301445c62a3) (details)
  2. [SPARK-23991][DSTREAMS] Fix data loss when WAL write fails in (commit: 49a6c2b915f37682781efba708a103e709c54cf7) (details)
Commit fec43fe1bcd7553eff077805a13bb301445c62a3 by hyukjinkwon
[SPARK-19613][SS][TEST] Random.nextString is not safe for directory
namePrefix
## What changes were proposed in this pull request?
`Random.nextString` is good for generating random string data, but it's
not proper for directory name prefix in `Utils.createDirectory(tempDir,
Random.nextString(10))`. This PR uses more safe directory namePrefix.
```scala scala> scala.util.Random.nextString(10) res0: String =
馨쭔ᎰႻ穚䃈兩㻞藑並
```
```scala StateStoreRDDSuite:
- versioning and immutability
- recovering from files
- usage with iterators - only gets and only puts
- preferred locations using StateStoreCoordinator *** FAILED ***
java.io.IOException: Failed to create a temp directory (under
/.../spark/sql/core/target/tmp/StateStoreRDDSuite8712796397908632676)
after 10 attempts!
at org.apache.spark.util.Utils$.createDirectory(Utils.scala:295)
at
org.apache.spark.sql.execution.streaming.state.StateStoreRDDSuite$$anonfun$13$$anonfun$apply$6.apply(StateStoreRDDSuite.scala:152)
at
org.apache.spark.sql.execution.streaming.state.StateStoreRDDSuite$$anonfun$13$$anonfun$apply$6.apply(StateStoreRDDSuite.scala:149)
at
org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:42)
at
org.apache.spark.sql.execution.streaming.state.StateStoreRDDSuite$$anonfun$13.apply(StateStoreRDDSuite.scala:149)
at
org.apache.spark.sql.execution.streaming.state.StateStoreRDDSuite$$anonfun$13.apply(StateStoreRDDSuite.scala:149)
...
- distributed test *** FAILED ***
java.io.IOException: Failed to create a temp directory (under
/.../spark/sql/core/target/tmp/StateStoreRDDSuite8712796397908632676)
after 10 attempts!
at org.apache.spark.util.Utils$.createDirectory(Utils.scala:295)
```
## How was this patch tested?
Pass the existing tests.StateStoreRDDSuite:
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #21446 from dongjoon-hyun/SPARK-19613.
(cherry picked from commit b31b587cd091010337378cf448fd598c37757053)
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
(commit: fec43fe1bcd7553eff077805a13bb301445c62a3)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDDSuite.scala (diff)
Commit 49a6c2b915f37682781efba708a103e709c54cf7 by sshao
[SPARK-23991][DSTREAMS] Fix data loss when WAL write fails in
allocateBlocksToBatch
When blocks tried to get allocated to a batch and WAL write fails then
the blocks will be removed from the received block queue. This fact
simply produces data loss because the next allocation will not find the
mentioned blocks in the queue.
In this PR blocks will be removed from the received queue only if WAL
write succeded.
Additional unit test.
Author: Gabor Somogyi <gabor.g.somogyi@gmail.com>
Closes #21430 from gaborgsomogyi/SPARK-23991.
Change-Id: I5ead84f0233f0c95e6d9f2854ac2ff6906f6b341
(cherry picked from commit aca65c63cb12073eb193fe08998994c60acb8b58)
Signed-off-by: jerryshao <sshao@hortonworks.com>
(commit: 49a6c2b915f37682781efba708a103e709c54cf7)
The file was modifiedstreaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala (diff)
The file was modifiedstreaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala (diff)