SuccessChanges

Summary

  1. [SPARK-26927][CORE] Ensure executor is active when processing events in (commit: d0290eaf240cfddfb51e490721484afaaba43176) (details)
  2. [MINOR][CORE] Use https for bintray spark-packages repository (commit: 959a7ecb9a1d433bed18f82c4c3354806f1329c7) (details)
Commit d0290eaf240cfddfb51e490721484afaaba43176 by vanzin
[SPARK-26927][CORE] Ensure executor is active when processing events in
dynamic allocation manager.
There is a race condition in the `ExecutorAllocationManager` that the
`SparkListenerExecutorRemoved` event is posted before the
`SparkListenerTaskStart` event, which will cause the incorrect result of
`executorIds`. Then, when some executor idles, the real executors will
be removed even actual executor number is equal to `minNumExecutors` due
to the incorrect computation of `newExecutorTotal`(may greater than the
`minNumExecutors`), thus may finally causing zero available executors
but a wrong positive number of executorIds was kept in memory.
What's more, even the `SparkListenerTaskEnd` event can not make the fake
`executorIds` released, because later idle event for the fake executors
can not cause the real removal of these executors, as they are already
removed and they are not exist in the `executorDataMap`  of
`CoaseGrainedSchedulerBackend`, so that the `onExecutorRemoved` method
will never be called again.
For details see https://issues.apache.org/jira/browse/SPARK-26927
This PR is to fix this problem.
existUT and added UT
Closes #23842 from
liupc/Fix-race-condition-that-casues-dyanmic-allocation-not-working.
Lead-authored-by: Liupengcheng <liupengcheng@xiaomi.com> Co-authored-by:
liupengcheng <liupengcheng@xiaomi.com> Signed-off-by: Marcelo Vanzin
<vanzin@cloudera.com>
(cherry picked from commit d5cfe08fdc7ad07e948f329c0bdeeca5c2574a18)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: d0290eaf240cfddfb51e490721484afaaba43176)
The file was modifiedcore/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala (diff)
Commit 959a7ecb9a1d433bed18f82c4c3354806f1329c7 by sean.owen
[MINOR][CORE] Use https for bintray spark-packages repository
## What changes were proposed in this pull request?
This patch changes the schema of url from http to https for bintray
spark-packages repository. Looks like we already changed the schema of
repository url for pom.xml but missed inside the code.
## How was this patch tested?
Manually ran the `--package` via `./bin/spark-shell --verbose
--packages "RedisLabs:spark-redis:0.3.2"`
```
... Ivy Default Cache set to: /Users/jlim/.ivy2/cache The jars for the
packages stored in: /Users/jlim/.ivy2/jars
:: loading settings :: url =
jar:file:/Users/jlim/WorkArea/ScalaProjects/spark/dist/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
RedisLabs#spark-redis added as a dependency
:: resolving dependencies ::
org.apache.spark#spark-submit-parent-2fee2e18-7832-4a4d-9e97-7b3d0fef766d;1.0
confs: [default]
found RedisLabs#spark-redis;0.3.2 in spark-packages
found redis.clients#jedis;2.7.2 in central
found org.apache.commons#commons-pool2;2.3 in central downloading
https://dl.bintray.com/spark-packages/maven/RedisLabs/spark-redis/0.3.2/spark-redis-0.3.2.jar
...
[SUCCESSFUL ] RedisLabs#spark-redis;0.3.2!spark-redis.jar (824ms)
downloading
https://repo1.maven.org/maven2/redis/clients/jedis/2.7.2/jedis-2.7.2.jar
...
[SUCCESSFUL ] redis.clients#jedis;2.7.2!jedis.jar (576ms) downloading
https://repo1.maven.org/maven2/org/apache/commons/commons-pool2/2.3/commons-pool2-2.3.jar
...
[SUCCESSFUL ] org.apache.commons#commons-pool2;2.3!commons-pool2.jar
(150ms)
:: resolution report :: resolve 4586ms :: artifacts dl 1555ms
:: modules in use:
RedisLabs#spark-redis;0.3.2 from spark-packages in [default]
org.apache.commons#commons-pool2;2.3 from central in [default]
redis.clients#jedis;2.7.2 from central in [default]
---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |   3   |   3   |   3   |   0   ||   3   |   3   |
---------------------------------------------------------------------
```
Closes #24061 from HeartSaVioR/MINOR-use-https-to-bintray-repository.
Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(cherry picked from commit f57af2286f85bf67706e14fecfbfd9ef034c2927)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 959a7ecb9a1d433bed18f82c4c3354806f1329c7)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala (diff)