1. [SPARK-24107][CORE] ChunkedByteBuffer.writeFully method has not reset (commit: 88abf7b9bff7162648265490356bc3f999e86dd3) (details)
  2. [SPARK-23971][BACKPORT-2.3] Should not leak Spark sessions across test (commit: b3adb5300031b191411b84c9ab1d203949bf342c) (details)
Commit 88abf7b9bff7162648265490356bc3f999e86dd3 by wenchen
[SPARK-24107][CORE] ChunkedByteBuffer.writeFully method has not reset
the limit value
JIRA Issue:
ChunkedByteBuffer.writeFully method has not reset the limit value. When
chunks larger than bufferWriteChunkSize, such as 80 * 1024 * 1024 larger
than config.BUFFER_WRITE_CHUNK_SIZE(64 * 1024 * 1024),only while once,
will lost 16 * 1024 * 1024 byte
Author: WangJinhai02 <>
Closes #21175 from manbuyun/bugfix-ChunkedByteBuffer.
(cherry picked from commit 152eaf6ae698cd0df7f5a5be3f17ee46e0be929d)
Signed-off-by: Wenchen Fan <>
(commit: 88abf7b9bff7162648265490356bc3f999e86dd3)
The file was modifiedcore/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala (diff)
Commit b3adb5300031b191411b84c9ab1d203949bf342c by gatorsmile
[SPARK-23971][BACKPORT-2.3] Should not leak Spark sessions across test
This PR is to backport the PR
to Apache 2.3. This should be the cause why we saw the test regressions
in Apache 2.3 branches:
## What changes were proposed in this pull request?
Many suites currently leak Spark sessions (sometimes with stopped
SparkContexts) via the thread-local active Spark session and default
Spark session. We should attempt to clean these up and detect when this
happens to improve the reproducibility of tests.
## How was this patch tested?
Existing tests
Author: Eric Liang <>
Closes #21197 from gatorsmile/backportSPARK-23971.
(commit: b3adb5300031b191411b84c9ab1d203949bf342c)
The file was modifiedmllib/src/test/java/org/apache/spark/ (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/test/SharedSparkSession.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SessionStateSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala (diff)