1. [SPARK-23823][SQL] Keep origin in transformExpression (commit: f93667f84b33537a69ee20a7d4dee664828803f6) (details)
  2. [SPARK-23822][SQL] Improve error message for Parquet schema mismatches (commit: ccc4a20453bbbaf1f3e5e46fb7c0277f1e6c65b9) (details)
Commit f93667f84b33537a69ee20a7d4dee664828803f6 by gatorsmile
[SPARK-23823][SQL] Keep origin in transformExpression
Keep origin for all the methods using transformExpression
## What changes were proposed in this pull request?
Keep origin in transformExpression
## How was this patch tested?
Manually tested that this fixes and columns have
correct origins after Analyzer.analyze
Author: JiahuiJiang <> Author: Jiahui Jiang
Closes #20961 from JiahuiJiang/jj/keep-origin.
(cherry picked from commit d65e531b44a388fed25d3cbf28fdce5a2d0598e6)
Signed-off-by: gatorsmile <>
(commit: f93667f84b33537a69ee20a7d4dee664828803f6)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala (diff)
The file was addedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala
Commit ccc4a20453bbbaf1f3e5e46fb7c0277f1e6c65b9 by gatorsmile
[SPARK-23822][SQL] Improve error message for Parquet schema mismatches
## What changes were proposed in this pull request?
This pull request tries to improve the error message for spark while
reading parquet files with different schemas, e.g. One with a STRING
column and the other with a INT column. A new
ParquetSchemaColumnConvertNotSupportedException is added to replace the
old UnsupportedOperationException. The Exception is again wrapped in
FileScanRdd.scala to throw a more a general QueryExecutionException with
the actual parquet file name which trigger the exception.
## How was this patch tested?
Unit tests added to check the new exception and verify the error
Also manually tested with two parquet with different schema to check the
error message.
<img width="1125" alt="screen shot 2018-03-30 at 4 03 04 pm"
Author: Yuchen Huo <>
Closes #20953 from yuchenhuo/SPARK-23822.
(cherry picked from commit 94524019315ad463f9bc13c107131091d17c6af9)
Signed-off-by: gatorsmile <>
(commit: ccc4a20453bbbaf1f3e5e46fb7c0277f1e6c65b9)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala (diff)
The file was addedsql/core/src/main/java/org/apache/spark/sql/execution/datasources/
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecutionException.scala (diff)
The file was modifiedsql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ (diff)