SuccessChanges

Summary

  1. [SPARK-26751][SQL] Fix memory leak when statement run in background and (commit: 4d6ea2cbb83d365434d0650a9e5ecaefc939b97d) (details)
  2. Preparing Spark release v2.3.3-rc2 (commit: 66fd9c34bf406a4b5f86605d06c9607752bd637a) (details)
  3. Preparing development version 2.3.4-SNAPSHOT (commit: 7845807d76c617fcb738dd9916caca80460db27e) (details)
Commit 4d6ea2cbb83d365434d0650a9e5ecaefc939b97d by sean.owen
[SPARK-26751][SQL] Fix memory leak when statement run in background and
throw exception which is not HiveSQLException
## What changes were proposed in this pull request? When we run in
background and we get exception which is not HiveSQLException, we may
encounter memory leak since handleToOperation will not removed
correctly. The reason is below: 1. When calling operation.run() in
HiveSessionImpl#executeStatementInternal we throw an exception which is
not HiveSQLException 2. Then the opHandle generated by
SparkSQLOperationManager will not be added into opHandleSet of
HiveSessionImpl , and operationManager.closeOperation(opHandle) will not
be called 3. When we close the session we will also call
operationManager.closeOperation(opHandle),since we did not add this
opHandle into the opHandleSet.
For the reasons above,the opHandled will always in
SparkSQLOperationManager#handleToOperation,which will cause memory leak.
More details and a case has attached on
https://issues.apache.org/jira/browse/SPARK-26751 This patch will always
throw HiveSQLException when running in background
## How was this patch tested? Exist UT
Closes #23673 from caneGuy/zhoukang/fix-hivesessionimpl-leak.
Authored-by: zhoukang <zhoukang199191@gmail.com> Signed-off-by: Sean
Owen <sean.owen@databricks.com>
(cherry picked from commit 255faaf3436e1f41838062ed460f801bb0be40ec)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 4d6ea2cbb83d365434d0650a9e5ecaefc939b97d)
The file was modifiedsql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala (diff)
The file was modifiedhadoop-cloud/pom.xml (diff)
The file was modifiedrepl/pom.xml (diff)
The file was modifiedexternal/kinesis-asl-assembly/pom.xml (diff)
The file was modifiedmllib/pom.xml (diff)
The file was modifiedassembly/pom.xml (diff)
The file was modifiedcommon/unsafe/pom.xml (diff)
The file was modifiedresource-managers/mesos/pom.xml (diff)
The file was modifiedstreaming/pom.xml (diff)
The file was modifiedmllib-local/pom.xml (diff)
The file was modifiedexternal/flume-assembly/pom.xml (diff)
The file was modifiedexternal/spark-ganglia-lgpl/pom.xml (diff)
The file was modifiedexternal/kafka-0-10/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/pom.xml (diff)
The file was modifiedexternal/docker-integration-tests/pom.xml (diff)
The file was modifiedresource-managers/yarn/pom.xml (diff)
The file was modifiedcommon/network-yarn/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-assembly/pom.xml (diff)
The file was modifiedcommon/sketch/pom.xml (diff)
The file was modifiedcommon/network-common/pom.xml (diff)
The file was modifiedcommon/kvstore/pom.xml (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-sql/pom.xml (diff)
The file was modifiedcommon/network-shuffle/pom.xml (diff)
The file was modifiedexternal/kafka-0-8-assembly/pom.xml (diff)
The file was modifiedR/pkg/DESCRIPTION (diff)
The file was modifiedexternal/flume-sink/pom.xml (diff)
The file was modifiedsql/catalyst/pom.xml (diff)
The file was modifiedcore/pom.xml (diff)
The file was modifiedcommon/tags/pom.xml (diff)
The file was modifiedlauncher/pom.xml (diff)
The file was modifiedpom.xml (diff)
The file was modifiedexternal/kafka-0-8/pom.xml (diff)
The file was modifiedsql/hive/pom.xml (diff)
The file was modifieddocs/_config.yml (diff)
The file was modifiedexternal/flume/pom.xml (diff)
The file was modifiedpython/pyspark/version.py (diff)
The file was modifiedexamples/pom.xml (diff)
The file was modifiedsql/core/pom.xml (diff)
The file was modifiedexternal/kinesis-asl/pom.xml (diff)
The file was modifiedgraphx/pom.xml (diff)
The file was modifiedtools/pom.xml (diff)
The file was modifiedsql/core/pom.xml (diff)
The file was modifiedexternal/kinesis-asl-assembly/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-sql/pom.xml (diff)
The file was modifiedmllib/pom.xml (diff)
The file was modifiedcommon/sketch/pom.xml (diff)
The file was modifiedresource-managers/mesos/pom.xml (diff)
The file was modifiedexternal/flume-assembly/pom.xml (diff)
The file was modifiedassembly/pom.xml (diff)
The file was modifiedR/pkg/DESCRIPTION (diff)
The file was modifieddocs/_config.yml (diff)
The file was modifiedexternal/kafka-0-8-assembly/pom.xml (diff)
The file was modifiedexternal/kinesis-asl/pom.xml (diff)
The file was modifiedexternal/spark-ganglia-lgpl/pom.xml (diff)
The file was modifiedexamples/pom.xml (diff)
The file was modifiedexternal/kafka-0-10/pom.xml (diff)
The file was modifiedmllib-local/pom.xml (diff)
The file was modifiedresource-managers/kubernetes/core/pom.xml (diff)
The file was modifiedcommon/network-common/pom.xml (diff)
The file was modifiedgraphx/pom.xml (diff)
The file was modifiedsql/hive-thriftserver/pom.xml (diff)
The file was modifiedexternal/flume-sink/pom.xml (diff)
The file was modifiedrepl/pom.xml (diff)
The file was modifiedsql/catalyst/pom.xml (diff)
The file was modifiedpom.xml (diff)
The file was modifiedcommon/network-shuffle/pom.xml (diff)
The file was modifiedsql/hive/pom.xml (diff)
The file was modifiedtools/pom.xml (diff)
The file was modifiedexternal/docker-integration-tests/pom.xml (diff)
The file was modifiedcommon/unsafe/pom.xml (diff)
The file was modifiedcommon/tags/pom.xml (diff)
The file was modifiedhadoop-cloud/pom.xml (diff)
The file was modifiedresource-managers/yarn/pom.xml (diff)
The file was modifiedpython/pyspark/version.py (diff)
The file was modifiedexternal/kafka-0-8/pom.xml (diff)
The file was modifiedcommon/network-yarn/pom.xml (diff)
The file was modifiedcore/pom.xml (diff)
The file was modifiedstreaming/pom.xml (diff)
The file was modifiedlauncher/pom.xml (diff)
The file was modifiedexternal/flume/pom.xml (diff)
The file was modifiedexternal/kafka-0-10-assembly/pom.xml (diff)
The file was modifiedcommon/kvstore/pom.xml (diff)