SuccessChanges

Summary

  1. [SPARK-24107][CORE] ChunkedByteBuffer.writeFully method has not reset (commit: 88abf7b9bff7162648265490356bc3f999e86dd3) (details)
  2. [SPARK-23971][BACKPORT-2.3] Should not leak Spark sessions across test (commit: b3adb5300031b191411b84c9ab1d203949bf342c) (details)
Commit 88abf7b9bff7162648265490356bc3f999e86dd3 by wenchen
[SPARK-24107][CORE] ChunkedByteBuffer.writeFully method has not reset
the limit value
JIRA Issue:
https://issues.apache.org/jira/browse/SPARK-24107?jql=text%20~%20%22ChunkedByteBuffer%22
ChunkedByteBuffer.writeFully method has not reset the limit value. When
chunks larger than bufferWriteChunkSize, such as 80 * 1024 * 1024 larger
than config.BUFFER_WRITE_CHUNK_SIZE(64 * 1024 * 1024),only while once,
will lost 16 * 1024 * 1024 byte
Author: WangJinhai02 <jinhai.wang02@ele.me>
Closes #21175 from manbuyun/bugfix-ChunkedByteBuffer.
(cherry picked from commit 152eaf6ae698cd0df7f5a5be3f17ee46e0be929d)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(commit: 88abf7b9bff7162648265490356bc3f999e86dd3)
The file was modifiedcore/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala (diff)
Commit b3adb5300031b191411b84c9ab1d203949bf342c by gatorsmile
[SPARK-23971][BACKPORT-2.3] Should not leak Spark sessions across test
suites
This PR is to backport the PR https://github.com/apache/spark/pull/21058
to Apache 2.3. This should be the cause why we saw the test regressions
in Apache 2.3 branches:
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.6/317/testReport/org.apache.spark.sql.execution.datasources.parquet/ParquetQuerySuite/SPARK_15678__not_use_cache_on_overwrite/history/
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.7/318/testReport/junit/org.apache.spark.sql/DataFrameSuite/inputFiles/history/
---
## What changes were proposed in this pull request?
Many suites currently leak Spark sessions (sometimes with stopped
SparkContexts) via the thread-local active Spark session and default
Spark session. We should attempt to clean these up and detect when this
happens to improve the reproducibility of tests.
## How was this patch tested?
Existing tests
Author: Eric Liang <ekl@databricks.com>
Closes #21197 from gatorsmile/backportSPARK-23971.
(commit: b3adb5300031b191411b84c9ab1d203949bf342c)
The file was modifiedmllib/src/test/java/org/apache/spark/SharedSparkSession.java (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/test/SharedSparkSession.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SessionStateSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala (diff)