1. [SPARK-30826][SQL] Respect reference case in `StringStartsWith` pushed (details)
Commit 6294cb8af552e7e62ad7cafa05eb33763db02dc4 by wenchen
[SPARK-30826][SQL] Respect reference case in `StringStartsWith` pushed
down to parquet
### What changes were proposed in this pull request? In the PR, I
propose to convert the attribute name of `StringStartsWith` pushed down
to the Parquet datasource to column reference via the
`nameToParquetField` map. Similar conversions are performed for other
source filters pushed down to parquet.
### Why are the changes needed? This fixes the bug described in
[SPARK-30826]( The
query from an external table:
```sql CREATE TABLE t1 (col STRING) USING parquet OPTIONS (path '$path')
``` created on top of written parquet files by
`Seq("42").toDF("COL").write.parquet(path)` returns wrong empty result:
```scala spark.sql("SELECT * FROM t1 WHERE col LIKE '4%'").show
### Does this PR introduce any user-facing change? Yes. After the
changes the result is correct for the example above:
```scala spark.sql("SELECT * FROM t1 WHERE col LIKE '4%'").show
| 42|
### How was this patch tested? Added a test to `ParquetFilterSuite`
Closes #27574 from MaxGekk/parquet-StringStartsWith-case-sens.
Authored-by: Maxim Gekk <> Signed-off-by: Wenchen Fan
(cherry picked from commit 8b73b92aadd685b29ef3d9b33366f5e1fd3dae99)
Signed-off-by: Wenchen Fan <>
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala (diff)