1. [SPARK-26806][SS] EventTimeStats.merge should handle zeros correctly (commit: a5d22da1888b8110b490d52d2c36b3fc907254f6) (details)
Commit a5d22da1888b8110b490d52d2c36b3fc907254f6 by zsxwing
[SPARK-26806][SS] EventTimeStats.merge should handle zeros correctly
## What changes were proposed in this pull request?
Right now, EventTimeStats.merge doesn't handle `zero.merge(zero)`
correctly. This will make `avg` become `NaN`. And whatever gets merged
with the result of `zero.merge(zero)`, `avg` will still be `NaN`. Then
finally, we call `NaN.toLong` and get `0`, and the user will see the
following incorrect report:
"eventTime" : {
   "avg" : "1970-01-01T00:00:00.000Z",
   "max" : "2019-01-31T12:57:00.000Z",
   "min" : "2019-01-30T18:44:04.000Z",
   "watermark" : "1970-01-01T00:00:00.000Z"
This issue was reported by liancheng .
This PR fixes the above issue.
## How was this patch tested?
The new unit tests.
Closes #23718 from zsxwing/merge-zero.
Authored-by: Shixiong Zhu <> Signed-off-by: Shixiong
Zhu <>
(cherry picked from commit 03a928cbecaf38bbbab3e6b957fcbb542771cfbd)
Signed-off-by: Shixiong Zhu <>
(commit: a5d22da1888b8110b490d52d2c36b3fc907254f6)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/EventTimeWatermarkExec.scala (diff)