1. [SPARK-25394][CORE] Add an application status metrics source (details)
Commit bd2c4471311cd7e948c80b4927a903636ce0ce7e by vanzin
[SPARK-25394][CORE] Add an application status metrics source
- Exposes several metrics regarding application status as a source,
useful to scrape them via jmx instead of mining the metrics rest api.
Example use case: prometheus + jmx exporter.
- Metrics are gathered when a job ends at the AppStatusListener side,
could be more fine-grained but most metrics like tasks completed are
also counted by executors. More metrics could be exposed in the future
to avoid scraping executors in some scenarios.
- a config option `` is added to
disable/enable these metrics, by default they are disabled.
This was manually tested with jmx source enabled and prometheus server
on k8s:
In the next pic the job delay is shown for repeated pi calculation
(Spark action).
Closes #22381 from skonto/add_app_status_metrics.
Authored-by: Stavros Kontopoulos <>
Signed-off-by: Marcelo Vanzin <>
The file was modifiedcore/src/main/scala/org/apache/spark/SparkContext.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/LiveEntity.scala (diff)
The file was addedcore/src/main/scala/org/apache/spark/status/AppStatusSource.scala
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusStore.scala (diff)