1. [SPARK-23365][CORE] Do not adjust num executors when killing idle (commit: 30242b664a05120c01b54ba9ac0ebed114f0d54e) (details)
Commit 30242b664a05120c01b54ba9ac0ebed114f0d54e by vanzin
[SPARK-23365][CORE] Do not adjust num executors when killing idle
The ExecutorAllocationManager should not adjust the target number of
executors when killing idle executors, as it has already adjusted the
target number down based on the task backlog.
The name `replace` was misleading with DynamicAllocation on, as the
target number of executors is changed outside of the call to
`killExecutors`, so I adjusted that name.  Also separated out the logic
of `countFailures` as you don't always want that tied to `replace`.
While I was there I made two changes that weren't directly related to
this: 1) Fixed `countFailures` in a couple cases where it was getting an
incorrect value since it used to be tied to `replace`, eg. when killing
executors on a blacklisted node. 2) hard error if you call
`sc.killExecutors` with dynamic allocation on, since that's another way
the ExecutorAllocationManager and the CoarseGrainedSchedulerBackend
would get out of sync.
Added a unit test case which verifies that the calls to
ExecutorAllocationClient do not adjust the number of executors.
Author: Imran Rashid <>
Closes #20604 from squito/SPARK-23365.
(cherry picked from commit ecb8b383af1cf1b67f3111c148229e00c9c17c40)
Signed-off-by: Marcelo Vanzin <>
(commit: 30242b664a05120c01b54ba9ac0ebed114f0d54e)
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/SparkContext.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala (diff)