SuccessChanges

Summary

  1. [SPARK-23788][SS] Fix race in StreamingQuerySuite (commit: 523fcafc5c4a79cf3455f3ceab6d886679399495) (details)
  2. [SPARK-23599][SQL] Add a UUID generator from Pseudo-Random Numbers (commit: 57026a1851aca9fe028cb39e8059f0bf133f3e0c) (details)
  3. [HOT-FIX] Fix SparkOutOfMemoryError: Unable to acquire 262144 bytes of (commit: 2fd7acabf8d55789662d52d94bd30f84b05a577a) (details)
Commit 523fcafc5c4a79cf3455f3ceab6d886679399495 by zsxwing
[SPARK-23788][SS] Fix race in StreamingQuerySuite
## What changes were proposed in this pull request?
The serializability test uses the same MemoryStream instance for 3
different queries. If any of those queries ask it to commit before the
others have run, the rest will see empty dataframes. This can fail the
test if q3 is affected.
We should use one instance per query instead.
## How was this patch tested?
Existing unit test. If I move q2.processAllAvailable() before starting
q3, the test always fails without the fix.
Author: Jose Torres <torres.joseph.f+github@gmail.com>
Closes #20896 from jose-torres/fixrace.
(cherry picked from commit 816a5496ba4caac438f70400f72bb10bfcc02418)
Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
(commit: 523fcafc5c4a79cf3455f3ceab6d886679399495)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala (diff)
Commit 57026a1851aca9fe028cb39e8059f0bf133f3e0c by hvanhovell
[SPARK-23599][SQL] Add a UUID generator from Pseudo-Random Numbers
## What changes were proposed in this pull request?
This patch adds a UUID generator from Pseudo-Random Numbers. We can use
it later to have deterministic `UUID()` expression.
## How was this patch tested?
Added unit tests.
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes #20817 from viirya/SPARK-23599.
(cherry picked from commit 4de638c1976dea74761bbe5c30da808178ee885d)
Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>
(commit: 57026a1851aca9fe028cb39e8059f0bf133f3e0c)
The file was addedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGeneratorSuite.scala
The file was addedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGenerator.scala
Commit 2fd7acabf8d55789662d52d94bd30f84b05a577a by hvanhovell
[HOT-FIX] Fix SparkOutOfMemoryError: Unable to acquire 262144 bytes of
memory, got 224631
## What changes were proposed in this pull request?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88263/testReport
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88260/testReport
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88257/testReport
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88224/testReport
These tests all failed:
``` org.apache.spark.memory.SparkOutOfMemoryError:  Unable to acquire
262144 bytes of memory, got 224631 at
org.apache.spark.memory.MemoryConsumer.throwOom(MemoryConsumer.java:157)
at
org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:98)
at
org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:787)
at
org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:204)
at
org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:219)
...
```
This PR ignore this test.
## How was this patch tested?
N/A
Author: Yuming Wang <yumwang@ebay.com>
Closes #20835 from wangyum/SPARK-23598.
(cherry picked from commit 15c3c983008557165cc91713ddaf2dbd6d5a506c)
Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>
(commit: 2fd7acabf8d55789662d52d94bd30f84b05a577a)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala (diff)