SuccessChanges

Summary

  1. [SPARK-25357][SQL] Add metadata to SparkPlanInfo to dump more (details)
Commit 9ac9f36c48391e4f2c1c32747bd2ad94a1b21c08 by wenchen
[SPARK-25357][SQL] Add metadata to SparkPlanInfo to dump more
information like file path to event log
## What changes were proposed in this pull request?
Field metadata removed from SparkPlanInfo in #18600 . Corresponding,
many meta data was also removed from event
SparkListenerSQLExecutionStart in Spark event log. If we want to analyze
event log to get all input paths, we couldn't get them. Instead,
simpleString of SparkPlanInfo JSON only display 100 characters, it won't
help.
Before 2.3, the fragment of SparkListenerSQLExecutionStart in event log
looks like below (It contains the metadata field which has the intact
information):
>{"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
Location:
InMemoryFileIndex[hdfs://cluster1/sys/edw/test1/test2/test3/test4...,
"metadata": {"Location":
"InMemoryFileIndex[hdfs://cluster1/sys/edw/test1/test2/test3/test4/test5/snapshot/dt=20180904]","ReadSchema":"struct<snpsht_start_dt:date,snpsht_end_dt:date,am_ntlogin_name:string,am_first_name:string,am_last_name:string,isg_name:string,CRE_DATE:date,CRE_USER:string,UPD_DATE:timestamp,UPD_USER:string>"}
After #18600, metadata field was removed.
>{"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
Location:
InMemoryFileIndex[hdfs://cluster1/sys/edw/test1/test2/test3/test4...,
So I add this field back to SparkPlanInfo class. Then it will log out
the meta data to event log. Intact information in event log is very
useful for offline job analysis.
## How was this patch tested? Unit test
Closes #22353 from LantaoJin/SPARK-25357.
Authored-by: LantaoJin <jinlantao@gmail.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
(cherry picked from commit 6dc5921e66d56885b95c07e56e687f9f6c1eaca7)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SQLJsonProtocolSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala (diff)