1. [SPARK-24552][CORE][SQL][BRANCH-2.3] Use unique id instead of attempt (commit: db538b25ae9016c624ed7c70a34dee2036d80d3a) (details)
Commit db538b25ae9016c624ed7c70a34dee2036d80d3a by vanzin
[SPARK-24552][CORE][SQL][BRANCH-2.3] Use unique id instead of attempt
number for writes .
This passes a unique attempt id instead of attempt number to v2 data
sources and hadoop APIs, because attempt number is reused when stages
are retried. When attempt numbers are reused, sources that track data by
partition id and attempt number may incorrectly clean up data because
the same attempt number can be both committed and aborted.
Author: Marcelo Vanzin <>
Closes #21615 from vanzin/SPARK-24552-2.3.
(commit: db538b25ae9016c624ed7c70a34dee2036d80d3a)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala (diff)