SuccessChanges

Summary

  1. [SPARK-27624][CORE] Fix CalenderInterval to show an empty interval (commit: 8eac465059f95d02df9f0df4c992c28dac4d4f1e) (details)
  2. [MINOR][DOCS] Fix invalid documentation for StreamingQueryManager Class (commit: 564dbf61b9e3febd623a08bec9506505fd337bc3) (details)
Commit 8eac465059f95d02df9f0df4c992c28dac4d4f1e by dhyun
[SPARK-27624][CORE] Fix CalenderInterval to show an empty interval
correctly
## What changes were proposed in this pull request?
If the interval is `0`, it doesn't show both the value `0` and the unit
at all. For example, this happens in the explain plans and Spark Web UI
on `EventTimeWatermark` diagram.
**BEFORE**
```scala scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "1
microsecond").explain
== Physical Plan == EventTimeWatermark ts#0: timestamp, interval 1
microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#0]
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "0
microsecond").explain
== Physical Plan == EventTimeWatermark ts#3: timestamp, interval
+- StreamingRelation FileSource[/tmp/t], [ts#3]
```
**AFTER**
```scala scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "1
microsecond").explain
== Physical Plan == EventTimeWatermark ts#0: timestamp, interval 1
microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#0]
scala> spark.readStream.schema("ts
timestamp").parquet("/tmp/t").withWatermark("ts", "0
microsecond").explain
== Physical Plan == EventTimeWatermark ts#3: timestamp, interval 0
microseconds
+- StreamingRelation FileSource[/tmp/t], [ts#3]
```
## How was this patch tested?
Pass the Jenkins with the updated test case.
Closes #24516 from dongjoon-hyun/SPARK-27624.
Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon
Hyun <dhyun@apple.com>
(cherry picked from commit 614a5cc600b0ac01c5d03b1dc5fdf996ef18ac0e)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(commit: 8eac465059f95d02df9f0df4c992c28dac4d4f1e)
The file was modifiedcommon/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java (diff)
The file was modifiedcommon/unsafe/src/test/java/org/apache/spark/unsafe/types/CalendarIntervalSuite.java (diff)
Commit 564dbf61b9e3febd623a08bec9506505fd337bc3 by gurwls223
[MINOR][DOCS] Fix invalid documentation for StreamingQueryManager Class
## What changes were proposed in this pull request?
When following the example for using
`spark.streams().awaitAnyTermination()` a valid pyspark code will output
the following error:
``` Traceback (most recent call last):
File "pyspark_app.py", line 182, in <module>
   spark.streams().awaitAnyTermination() TypeError:
'StreamingQueryManager' object is not callable
```
Docs URL:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries
This changes the documentation line to properly call the method under
the StreamingQueryManager Class
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager
## How was this patch tested?
After changing the syntax, error no longer occurs and pyspark
application works
This is only docs change
Closes #24547 from asaf400/patch-1.
Authored-by: Asaf Levy <asaf400@gmail.com> Signed-off-by: HyukjinKwon
<gurwls223@apache.org>
(cherry picked from commit 09422f5139cc13abaf506453819c2bb91e174ae3)
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
(commit: 564dbf61b9e3febd623a08bec9506505fd337bc3)
The file was modifieddocs/structured-streaming-programming-guide.md (diff)