SuccessChanges

Summary

  1. [SPARK-27112][CORE] : Create a resource ordering between threads to r… (commit: 7bb2b42de9a52e75107469f028c1e707d6a174b7) (details)
Commit 7bb2b42de9a52e75107469f028c1e707d6a174b7 by irashid
[SPARK-27112][CORE] : Create a resource ordering between threads to r…
…esolve the deadlocks encountered when trying to kill executors either
due to dynamic allocation or blacklisting
Closes #24072 from pgandhi999/SPARK-27112-2.
Authored-by: pgandhi <pgandhiverizonmedia.com> Signed-off-by: Imran
Rashid <irashidcloudera.com>
## What changes were proposed in this pull request?
There are two deadlocks as a result of the interplay between three
different threads:
**task-result-getter thread**
**spark-dynamic-executor-allocation thread**
**dispatcher-event-loop thread(makeOffers())**
The fix ensures ordering synchronization constraint by acquiring lock on
`TaskSchedulerImpl` before acquiring lock on
`CoarseGrainedSchedulerBackend` in `makeOffers()` as well as
killExecutors() method. This ensures resource ordering between the
threads and thus, fixes the deadlocks.
## How was this patch tested?
Manual Tests
Closes #24134 from pgandhi999/branch-2.4-SPARK-27112.
Authored-by: pgandhi <pgandhi@verizonmedia.com> Signed-off-by: Imran
Rashid <irashid@cloudera.com>
(cherry picked from commit 95e73b328ac883be2ced9099f20c8878e498e297)
Signed-off-by: Imran Rashid <irashid@cloudera.com>
(commit: 7bb2b42de9a52e75107469f028c1e707d6a174b7)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala (diff)