1. [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIntoHiveDirCommand (commit: 60e56bcabf29addcada2c19ce368a0e32c99e412) (details)
  2. [SPARK-24889][CORE] Update block info when unpersist rdds (commit: 18688d370399dcf92f4228db6c7e3cb186804c18) (details)
Commit 60e56bcabf29addcada2c19ce368a0e32c99e412 by dongjoon
[SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIntoHiveDirCommand
output schema in Parquet issue
## What changes were proposed in this pull request?
Backport to branch-2.3.
## How was this patch tested?
unit tests
Closes #22387 from wangyum/SPARK-25313-FOLLOW-UP-branch-2.3.
Authored-by: Yuming Wang <> Signed-off-by: Dongjoon Hyun
(commit: 60e56bcabf29addcada2c19ce368a0e32c99e412)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveDirCommand.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala (diff)
Commit 18688d370399dcf92f4228db6c7e3cb186804c18 by vanzin
[SPARK-24889][CORE] Update block info when unpersist rdds
We will update block info coming from executors, at the timing like
caching a RDD. However, when removing RDDs with unpersisting, we don't
ask to update block info. So the block info is not updated.
We can fix this with few options:
1. Ask to update block info when unpersisting
This is simplest but changes driver-executor communication a bit.
2. Update block info when processing the event of unpersisting RDD
We send a `SparkListenerUnpersistRDD` event when unpersisting RDD. When
processing this event, we can update block info of the RDD. This only
changes event processing code so the risk seems to be lower.
Currently this patch takes option 2 for lower risk. If we agree first
option has no risk, we can change to it.
Unit tests.
Closes #22341 from viirya/SPARK-24889.
Authored-by: Liang-Chi Hsieh <> Signed-off-by: Marcelo
Vanzin <>
(cherry picked from commit 14f3ad20932535fe952428bf255e7eddd8fa1b58)
Signed-off-by: Marcelo Vanzin <>
(commit: 18688d370399dcf92f4228db6c7e3cb186804c18)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/LiveEntity.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala (diff)