SuccessChanges

Summary

  1. [SPARK-25630][TEST] Reduce test time of HadoopFsRelationTest (details)
  2. [SPARK-25639][DOCS] Added docs for foreachBatch, python foreach and (details)
Commit 6a60fb0aad62e98c8a0e1c365819f31b1fc0132e by gatorsmile
[SPARK-25630][TEST] Reduce test time of HadoopFsRelationTest
## What changes were proposed in this pull request? There was 5 suites
extends `HadoopFsRelationTest`,  for testing
"orc"/"parquet"/"text"/"json" data sources. This PR refactor the base
trait `HadoopFsRelationTest`: 1. Rename unnecessary loop for setting
parquet conf 2. The test case `SPARK-8406: Avoids name collision while
writing files` takes about 14 to 20 seconds. As now all the file format
data source are using common code, for creating result files, we can
test one data source(Parquet) only to reduce test time.
To run related 5 suites:
```
./build/sbt "hive/testOnly *HadoopFsRelationSuite"
``` The total test run time is reduced from 5 minutes 40 seconds to 3
minutes 50 seconds.
## How was this patch tested?
Unit test
Closes #22643 from gengliangwang/refactorHadoopFsRelationTest.
Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/sources/HadoopFsRelationTest.scala (diff)
Commit f9935a3f85f46deef2cb7b213c1c02c8ff627a8c by tathagata.das1565
[SPARK-25639][DOCS] Added docs for foreachBatch, python foreach and
multiple watermarks
## What changes were proposed in this pull request?
Added
- Python foreach
- Scala, Java and Python foreachBatch
- Multiple watermark policy
- The semantics of what changes are allowed to the streaming between
restarts.
## How was this patch tested? No tests
Closes #22627 from tdas/SPARK-25639.
Authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by:
Tathagata Das <tathagata.das1565@gmail.com>
The file was modifieddocs/structured-streaming-programming-guide.md (diff)