1. [SPARK-30417][CORE] Task speculation numTaskThreshold should be greater (details)
Commit c49abf820d53fac5d443ded3ef4206935a6b3fd9 by xingbo.jiang
[SPARK-30417][CORE] Task speculation numTaskThreshold should be greater
than 0 even EXECUTOR_CORES is not set under Standalone mode
### What changes were proposed in this pull request?
Previously in,
we compare the number of tasks with `(conf.get(EXECUTOR_CORES) /
sched.CPUS_PER_TASK)`. In standalone mode if the value is not explicitly
set by default, the conf value would be 1 but the executor would
actually use all the cores of the worker. So it is allowed to have
`CPUS_PER_TASK` greater than `EXECUTOR_CORES`. To handle this case, we
change the condition to be `numTasks <=
Math.max(conf.get(EXECUTOR_CORES) / sched.CPUS_PER_TASK, 1)`
### Why are the changes needed?
For standalone mode if the user set the `spark.task.cpus` to be greater
than 1 but didn't set the `spark.executor.cores`. Even though there is
only 1 task in the stage it would not be speculative run.
### Does this PR introduce any user-facing change?
Solve the problem above by allowing speculative run when there is only 1
task in the stage.
### How was this patch tested?
Existing tests and one more test in TaskSetManagerSuite
Closes #27126 from yuchenhuo/SPARK-30417.
Authored-by: Yuchen Huo <> Signed-off-by:
Xingbo Jiang <>
The file was modifiedcore/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala (diff)
The file was modifieddocs/ (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/internal/config/package.scala (diff)