1. [SPARK-24867][SQL] Add AnalysisBarrier to DataFrameWriter (commit: fa552c3c1102404fe98c72a5b83cffbc5ba41df3) (details)
Commit fa552c3c1102404fe98c72a5b83cffbc5ba41df3 by gatorsmile
[SPARK-24867][SQL] Add AnalysisBarrier to DataFrameWriter
     val udf1 = udf({(x: Int, y: Int) => x + y})
     val df = spark.range(0, 3).toDF("a")
       .withColumn("b", udf1($"a", udf1($"a", lit(10))))
``` Cache is not being used because the plans do not match with the
cached plan. This is a regression caused by the changes we made in
AnalysisBarrier, since not all the Analyzer rules are idempotent.
Added a test.
Also found a bug in the DSV1 write path. This is not a regression. Thus,
opened a separate JIRA
Author: Xiao Li <>
Closes #21821 from gatorsmile/testMaster22.
(cherry picked from commit d2e7deb59f641e93778b763d5396f73d38f9a785)
Signed-off-by: Xiao Li <>
(commit: fa552c3c1102404fe98c72a5b83cffbc5ba41df3)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala (diff)