1. [SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or (details)
Commit ae1b44c2dd7f25695269acc6bff7c0483c4a7e4d by dhyun
[SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or
Double.NaN for all NaN values
## What changes were proposed in this pull request?
Apache Spark uses the predefined `Float.NaN` and `Double.NaN` for NaN
values, but there exists more NaN values with different binary
```scala scala>
java.nio.ByteBuffer.allocate(4).putFloat(Float.NaN).array res1:
Array[Byte] = Array(127, -64, 0, 0)
scala> val x = java.lang.Float.intBitsToFloat(-6966608) x: Float = NaN
scala> java.nio.ByteBuffer.allocate(4).putFloat(x).array res2:
Array[Byte] = Array(-1, -107, -78, -80)
Since users can have these values, `RandomDataGenerator` generates these
NaN values. However, this causes `checkEvaluationWithUnsafeProjection`
failures due to the difference between `UnsafeRow` binary presentation.
The following is the UT failure instance. This PR aims to fix this UT
## How was this patch tested?
Pass the Jenkins with the newly added test cases.
Closes #23851 from dongjoon-hyun/SPARK-26950.
Authored-by: Dongjoon Hyun <> Signed-off-by: Wenchen
Fan <>
(cherry picked from commit ffef3d40741b0be321421aa52a6e17a26d89f541)
Signed-off-by: Wenchen Fan <>
(cherry picked from commit ef67be363be6d6b6954b55ef1c243a0672b84abb)
Signed-off-by: Dongjoon Hyun <>
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala (diff)