1. [SPARK-23808][SQL] Set default Spark session in test-only spark (commit: 1365d739d9d4d9cdc81b8b0c6b80ddad15227cba) (details)
  2. Revert "[SPARK-23785][LAUNCHER] LauncherBackend doesn't check state of (commit: 3f5955aa0a16707a31fa9089d4babd64b5eed6ff) (details)
  3. [SPARK-23827][SS] StreamingJoinExec should ensure that input data is (commit: 507cff246cd9e15a418d67b66bf762be4ae71c67) (details)
Commit 1365d739d9d4d9cdc81b8b0c6b80ddad15227cba by gatorsmile
[SPARK-23808][SQL] Set default Spark session in test-only spark
## What changes were proposed in this pull request?
Set default Spark session in the TestSparkSession and
TestHiveSparkSession constructors.
## How was this patch tested?
new unit tests
Author: Jose Torres <>
Closes #20926 from jose-torres/test3.
(cherry picked from commit b348901192b231153b58fe5720253168c87963d4)
Signed-off-by: gatorsmile <>
(commit: 1365d739d9d4d9cdc81b8b0c6b80ddad15227cba)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala (diff)
The file was addedsql/core/src/test/scala/org/apache/spark/sql/test/TestSparkSessionSuite.scala
Commit 3f5955aa0a16707a31fa9089d4babd64b5eed6ff by vanzin
Revert "[SPARK-23785][LAUNCHER] LauncherBackend doesn't check state of
connection before setting state"
This reverts commit 0bfbcaf6696570b74923047266b00ba4dc2ba97c.
(commit: 3f5955aa0a16707a31fa9089d4babd64b5eed6ff)
The file was modifiedcore/src/main/scala/org/apache/spark/launcher/LauncherBackend.scala (diff)
The file was modifiedlauncher/src/test/java/org/apache/spark/launcher/ (diff)
Commit 507cff246cd9e15a418d67b66bf762be4ae71c67 by tathagata.das1565
[SPARK-23827][SS] StreamingJoinExec should ensure that input data is
partitioned into specific number of partitions
## What changes were proposed in this pull request?
Currently, the requiredChildDistribution does not specify the
partitions. This can cause the weird corner cases where the child's
distribution is `SinglePartition` which satisfies the required
distribution of `ClusterDistribution(no-num-partition-requirement)`,
thus eliminating the shuffle needed to repartition input data into the
required number of partitions (i.e. same as state stores). That can lead
to "file not found" errors on the state store delta files as the
micro-batch-with-no-shuffle will not run certain tasks and therefore not
generate the expected state store delta files.
This PR adds the required constraint on the number of partitions.
## How was this patch tested? Modified test harness to always check that
ANY stateful operator should have a constraint on the number of
partitions. As part of that, the existing opt-in checks on child output
partitioning were removed, as they are redundant.
Author: Tathagata Das <>
Closes #20941 from tdas/SPARK-23827.
(cherry picked from commit 15298b99ac8944e781328423289586176cf824d7)
Signed-off-by: Tathagata Das <>
(commit: 507cff246cd9e15a418d67b66bf762be4ae71c67)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala (diff)
The file was removedsql/core/src/test/scala/org/apache/spark/sql/streaming/StatefulOperatorTest.scala
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/streaming/DeduplicateSuite.scala (diff)