1. [SPARK-25697][CORE] When zstd compression enabled, InProgress (details)
Commit 8e039a75548e91b0a8799d9d72c6797b066ddd62 by sean.owen
[SPARK-25697][CORE] When zstd compression enabled, InProgress
application is throwing Error in the history webui
## What changes were proposed in this pull request? When we enable event
log compression and compression codec as 'zstd', we are unable to open
the webui of the running application from the history server page. The
reason is that, Replay listener was unable to read from the zstd
compressed eventlog due to the zstd frame was not finished yet. This
causes truncated error while reading the eventLog.
So, when we try to open the WebUI from the History server page, it
throws "truncated error ", and we never able to open running application
in the webui, when we enable zstd compression.
In this PR, when the IO excpetion happens, and if it is a running
application, we  log the error,
"Failed to read Spark event log: evetLogDirAppName.inprogress", instead
of throwing exception.
## How was this patch tested? Test steps: 1)spark.eventLog.compress =
true 2) = zstd 3)restart history server 4)
launch bin/spark-shell 5) run some queries 6) Open history server page
7) click on the application
**Before fix:**
![screenshot from 2018-10-10
![screenshot from 2018-10-10
**After fix:**
![screenshot from 2018-10-10
![screenshot from 2018-10-10
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot;
otherwise, remove this)
Please review before opening a
pull request.
Closes #22689 from shahidki31/SPARK-25697.
Authored-by: Shahid <> Signed-off-by: Sean Owen
The file was modifiedcore/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala (diff)