1. [R][BACKPORT-2.3] update package description (commit: 36db45d5b90ddc3ce54febff2ed41cd29c0a8a04) (details)
  2. [SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or (commit: ae1b44c2dd7f25695269acc6bff7c0483c4a7e4d) (details)
Commit 36db45d5b90ddc3ce54febff2ed41cd29c0a8a04 by gurwls223
[R][BACKPORT-2.3] update package description
## What changes were proposed in this pull request?
doesn't port cleanly to 2.3. we need this in branch-2.4 and branch-2.3
Closes #23861 from felixcheung/2.3rdesc.
Authored-by: Felix Cheung <> Signed-off-by:
Hyukjin Kwon <>
(commit: 36db45d5b90ddc3ce54febff2ed41cd29c0a8a04)
The file was modifiedR/pkg/DESCRIPTION (diff)
Commit ae1b44c2dd7f25695269acc6bff7c0483c4a7e4d by dhyun
[SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or
Double.NaN for all NaN values
## What changes were proposed in this pull request?
Apache Spark uses the predefined `Float.NaN` and `Double.NaN` for NaN
values, but there exists more NaN values with different binary
```scala scala>
java.nio.ByteBuffer.allocate(4).putFloat(Float.NaN).array res1:
Array[Byte] = Array(127, -64, 0, 0)
scala> val x = java.lang.Float.intBitsToFloat(-6966608) x: Float = NaN
scala> java.nio.ByteBuffer.allocate(4).putFloat(x).array res2:
Array[Byte] = Array(-1, -107, -78, -80)
Since users can have these values, `RandomDataGenerator` generates these
NaN values. However, this causes `checkEvaluationWithUnsafeProjection`
failures due to the difference between `UnsafeRow` binary presentation.
The following is the UT failure instance. This PR aims to fix this UT
## How was this patch tested?
Pass the Jenkins with the newly added test cases.
Closes #23851 from dongjoon-hyun/SPARK-26950.
Authored-by: Dongjoon Hyun <> Signed-off-by: Wenchen
Fan <>
(cherry picked from commit ffef3d40741b0be321421aa52a6e17a26d89f541)
Signed-off-by: Wenchen Fan <>
(cherry picked from commit ef67be363be6d6b6954b55ef1c243a0672b84abb)
Signed-off-by: Dongjoon Hyun <>
(commit: ae1b44c2dd7f25695269acc6bff7c0483c4a7e4d)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala (diff)