1. [SPARK-26927][CORE] Ensure executor is active when processing events in (commit: d0290eaf240cfddfb51e490721484afaaba43176) (details)
  2. [MINOR][CORE] Use https for bintray spark-packages repository (commit: 959a7ecb9a1d433bed18f82c4c3354806f1329c7) (details)
Commit d0290eaf240cfddfb51e490721484afaaba43176 by vanzin
[SPARK-26927][CORE] Ensure executor is active when processing events in
dynamic allocation manager.
There is a race condition in the `ExecutorAllocationManager` that the
`SparkListenerExecutorRemoved` event is posted before the
`SparkListenerTaskStart` event, which will cause the incorrect result of
`executorIds`. Then, when some executor idles, the real executors will
be removed even actual executor number is equal to `minNumExecutors` due
to the incorrect computation of `newExecutorTotal`(may greater than the
`minNumExecutors`), thus may finally causing zero available executors
but a wrong positive number of executorIds was kept in memory.
What's more, even the `SparkListenerTaskEnd` event can not make the fake
`executorIds` released, because later idle event for the fake executors
can not cause the real removal of these executors, as they are already
removed and they are not exist in the `executorDataMap`  of
`CoaseGrainedSchedulerBackend`, so that the `onExecutorRemoved` method
will never be called again.
For details see
This PR is to fix this problem.
existUT and added UT
Closes #23842 from
Lead-authored-by: Liupengcheng <> Co-authored-by:
liupengcheng <> Signed-off-by: Marcelo Vanzin
(cherry picked from commit d5cfe08fdc7ad07e948f329c0bdeeca5c2574a18)
Signed-off-by: Marcelo Vanzin <>
(commit: d0290eaf240cfddfb51e490721484afaaba43176)
The file was modifiedcore/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala (diff)
Commit 959a7ecb9a1d433bed18f82c4c3354806f1329c7 by sean.owen
[MINOR][CORE] Use https for bintray spark-packages repository
## What changes were proposed in this pull request?
This patch changes the schema of url from http to https for bintray
spark-packages repository. Looks like we already changed the schema of
repository url for pom.xml but missed inside the code.
## How was this patch tested?
Manually ran the `--package` via `./bin/spark-shell --verbose
--packages "RedisLabs:spark-redis:0.3.2"`
... Ivy Default Cache set to: /Users/jlim/.ivy2/cache The jars for the
packages stored in: /Users/jlim/.ivy2/jars
:: loading settings :: url =
RedisLabs#spark-redis added as a dependency
:: resolving dependencies ::
confs: [default]
found RedisLabs#spark-redis;0.3.2 in spark-packages
found redis.clients#jedis;2.7.2 in central
found org.apache.commons#commons-pool2;2.3 in central downloading
[SUCCESSFUL ] RedisLabs#spark-redis;0.3.2!spark-redis.jar (824ms)
[SUCCESSFUL ] redis.clients#jedis;2.7.2!jedis.jar (576ms) downloading
[SUCCESSFUL ] org.apache.commons#commons-pool2;2.3!commons-pool2.jar
:: resolution report :: resolve 4586ms :: artifacts dl 1555ms
:: modules in use:
RedisLabs#spark-redis;0.3.2 from spark-packages in [default]
org.apache.commons#commons-pool2;2.3 from central in [default]
redis.clients#jedis;2.7.2 from central in [default]
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
|      default     |   3   |   3   |   3   |   0   ||   3   |   3   |
Closes #24061 from HeartSaVioR/MINOR-use-https-to-bintray-repository.
Authored-by: Jungtaek Lim (HeartSaVioR) <>
Signed-off-by: Sean Owen <>
(cherry picked from commit f57af2286f85bf67706e14fecfbfd9ef034c2927)
Signed-off-by: Sean Owen <>
(commit: 959a7ecb9a1d433bed18f82c4c3354806f1329c7)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala (diff)