FailedChanges

Summary

  1. [SPARK-25394][CORE] Add an application status metrics source (details)
Commit bd2c4471311cd7e948c80b4927a903636ce0ce7e by vanzin
[SPARK-25394][CORE] Add an application status metrics source
- Exposes several metrics regarding application status as a source,
useful to scrape them via jmx instead of mining the metrics rest api.
Example use case: prometheus + jmx exporter.
- Metrics are gathered when a job ends at the AppStatusListener side,
could be more fine-grained but most metrics like tasks completed are
also counted by executors. More metrics could be exposed in the future
to avoid scraping executors in some scenarios.
- a config option `spark.app.status.metrics.enabled` is added to
disable/enable these metrics, by default they are disabled.
This was manually tested with jmx source enabled and prometheus server
on k8s:
![metrics](https://user-images.githubusercontent.com/7945591/45300945-63064d00-b518-11e8-812a-d9b4155ba0c0.png)
In the next pic the job delay is shown for repeated pi calculation
(Spark action).
![pi](https://user-images.githubusercontent.com/7945591/45329927-89a1a380-b56b-11e8-9cc1-5e76cb83969f.png)
Closes #22381 from skonto/add_app_status_metrics.
Authored-by: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
The file was modifiedcore/src/main/scala/org/apache/spark/SparkContext.scala (diff)
The file was addedcore/src/main/scala/org/apache/spark/status/AppStatusSource.scala
The file was modifiedcore/src/main/scala/org/apache/spark/status/LiveEntity.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusStore.scala (diff)