SuccessChanges

Summary

  1. [SPARK-23470][UI] Use first attempt of last stage to define job (commit: a1ee6f1fc543120763f1b373bb31bc6d84004318) (details)
  2. [SPARK-23468][CORE] Stringify auth secret before storing it in (commit: 1d78f03ae3037f1ddbe6533a6733b7805a6705bf) (details)
  3. [SPARK-23454][SS][DOCS] Added trigger information to the Structured (commit: 3e7269eb904b591883300d7433e5c99be0b3b5b3) (details)
  4. [SPARK-23484][SS] Fix possible race condition in KafkaContinuousReader (commit: 373ac642fc145527bdfccae046d0e98f105ee7b3) (details)
  5. [SPARK-23481][WEBUI] lastStageAttempt should fail when a stage doesn't (commit: 23ba4416e1bbbaa818876d7a837f7a5e260aa048) (details)
Commit a1ee6f1fc543120763f1b373bb31bc6d84004318 by sameerag
[SPARK-23470][UI] Use first attempt of last stage to define job
description.
This is much faster than finding out what the last attempt is, and the
data should be the same.
There's room for improvement in this page (like only loading data for
the jobs being shown, instead of loading all available jobs and sorting
them), but this should bring performance on par with the 2.2 version.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #20644 from vanzin/SPARK-23470.
(cherry picked from commit 2ba77ed9e51922303e3c3533e368b95788bd7de5)
Signed-off-by: Sameer Agarwal <sameerag@apache.org>
(commit: a1ee6f1fc543120763f1b373bb31bc6d84004318)
The file was modifiedcore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala (diff)
Commit 1d78f03ae3037f1ddbe6533a6733b7805a6705bf by vanzin
[SPARK-23468][CORE] Stringify auth secret before storing it in
credentials.
The secret is used as a string in many parts of the code, so it has to
be turned into a hex string to avoid issues such as the random byte
sequence not containing a valid UTF8 sequence.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #20643 from vanzin/SPARK-23468.
(cherry picked from commit 6d398c05cbad69aa9093429e04ae44c73b81cd5a)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 1d78f03ae3037f1ddbe6533a6733b7805a6705bf)
The file was modifiedcore/src/main/scala/org/apache/spark/SecurityManager.scala (diff)
Commit 3e7269eb904b591883300d7433e5c99be0b3b5b3 by tathagata.das1565
[SPARK-23454][SS][DOCS] Added trigger information to the Structured
Streaming programming guide
## What changes were proposed in this pull request?
- Added clear information about triggers
- Made the semantics guarantees of watermarks more clear for streaming
aggregations and stream-stream joins.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot;
otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #20631 from tdas/SPARK-23454.
(cherry picked from commit 601d653bff9160db8477f86d961e609fc2190237)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
(commit: 3e7269eb904b591883300d7433e5c99be0b3b5b3)
The file was modifieddocs/structured-streaming-programming-guide.md (diff)
Commit 373ac642fc145527bdfccae046d0e98f105ee7b3 by tathagata.das1565
[SPARK-23484][SS] Fix possible race condition in KafkaContinuousReader
## What changes were proposed in this pull request?
var `KafkaContinuousReader.knownPartitions` should be threadsafe as it
is accessed from multiple threads - the query thread at the time of
reader factory creation, and the epoch tracking thread at the time of
`needsReconfiguration`.
## How was this patch tested?
Existing tests.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #20655 from tdas/SPARK-23484.
(cherry picked from commit 3fd0ccb13fea44727d970479af1682ef00592147)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
(commit: 373ac642fc145527bdfccae046d0e98f105ee7b3)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousReader.scala (diff)
Commit 23ba4416e1bbbaa818876d7a837f7a5e260aa048 by zsxwing
[SPARK-23481][WEBUI] lastStageAttempt should fail when a stage doesn't
exist
## What changes were proposed in this pull request?
The issue here is `AppStatusStore.lastStageAttempt` will return the next
available stage in the store when a stage doesn't exist.
This PR adds `last(stageId)` to ensure it returns a correct `StageData`
## How was this patch tested?
The new unit test.
Author: Shixiong Zhu <zsxwing@gmail.com>
Closes #20654 from zsxwing/SPARK-23481.
(cherry picked from commit 744d5af652ee8cece361cbca31e5201134e0fb42)
Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
(commit: 23ba4416e1bbbaa818876d7a837f7a5e260aa048)
The file was modifiedcore/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusStore.scala (diff)