1. [SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas (details)
Commit 32429256f3e659c648462e5b2740747645740c97 by gatorsmile
[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas
A self-join on a dataset which contains a `FlatMapGroupsInPandas` fails
because of duplicate attributes. This happens because we are not dealing
with this specific case in our `dedupAttr` rules.
The PR fix the issue by adding the management of the specific case
added UT + manual tests
Author: Marco Gaido <> Author: Marco Gaido
Closes #21737 from mgaido91/SPARK-24208.
(cherry picked from commit ebf4bfb966389342bfd9bdb8e3b612828c18730c)
Signed-off-by: Xiao Li <>
The file was modifiedpython/pyspark/sql/ (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/GroupedDatasetSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala (diff)