1. [SPARK-25273][DOC] How to install testthat 1.0.2 (commit: b072717b3f6178e728c0bf855aca243c275e58f0) (details)
  2. [SPARK-24909][CORE] Always unregister pending partition on task (commit: dbf0b934024dfa562775967e3e3114de5a163443) (details)
Commit b072717b3f6178e728c0bf855aca243c275e58f0 by hyukjinkwon
[SPARK-25273][DOC] How to install testthat 1.0.2
## What changes were proposed in this pull request?
R tests require `testthat` v1.0.2. In the PR, I described how to install
the version in the section
Closes #22272 from MaxGekk/r-testthat-doc.
Authored-by: Maxim Gekk <> Signed-off-by:
hyukjinkwon <>
(commit: b072717b3f6178e728c0bf855aca243c275e58f0)
The file was modifieddocs/ (diff)
The file was modifieddocs/ (diff)
Commit dbf0b934024dfa562775967e3e3114de5a163443 by tgraves
[SPARK-24909][CORE] Always unregister pending partition on task
Spark scheduler can hang when fetch failures, executor lost, task
running on lost executor, and multiple stage attempts. To fix this we
change to always unregister the pending partition on task completion.
this PR is actually reverting the change in SPARK-19263, so that it
always does shuffleStage.pendingPartitions -= task.partitionId.   The
change in SPARK-23433, should fix the issue originally from SPARK-19263.
Unit tests.  The condition happens on a race which I haven't reproduced
on a real customer, just see it sometimes on customers jobs in a real
cluster. I am also working on adding spark scheduler integration tests.
Closes #21976 from tgravescs/SPARK-24909.
Authored-by: Thomas Graves <>
Signed-off-by: Marcelo Vanzin <>
(cherry picked from commit ec3e9986385880adce1648eae30007eccff862ba)
Signed-off-by: Thomas Graves <>
(commit: dbf0b934024dfa562775967e3e3114de5a163443)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala (diff)