SuccessChanges

Summary

  1. [SPARK-23881][CORE][TEST] Fix flaky test (commit: bf1dabede512ec89fec507f85bec0a8bae5753f0) (details)
  2. [SPARK-23816][CORE] Killed tasks should ignore FetchFailures. (commit: 0f2aabc6bc64d2b5d46e59525111bd95fcd73610) (details)
Commit bf1dabede512ec89fec507f85bec0a8bae5753f0 by gatorsmile
[SPARK-23881][CORE][TEST] Fix flaky test
JobCancellationSuite."interruptible iterator of shuffle reader"
## What changes were proposed in this pull request?
The test case JobCancellationSuite."interruptible iterator of shuffle
reader" has been flaky because `KillTask` event is handled
asynchronously, so it can happen that the semaphore is released but the
task is still running. Actually we only have to check if the total
number of processed elements is less than the input elements number, so
we know the task get cancelled.
## How was this patch tested?
The new test case still fails without the purposed patch, and succeeded
in current master.
Author: Xingbo Jiang <xingbo.jiang@databricks.com>
Closes #20993 from jiangxb1987/JobCancellationSuite.
(cherry picked from commit d81f29ecafe8fc9816e36087e3b8acdc93d6cc1b)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
(commit: bf1dabede512ec89fec507f85bec0a8bae5753f0)
The file was modifiedcore/src/test/scala/org/apache/spark/JobCancellationSuite.scala (diff)
Commit 0f2aabc6bc64d2b5d46e59525111bd95fcd73610 by vanzin
[SPARK-23816][CORE] Killed tasks should ignore FetchFailures.
SPARK-19276 ensured that FetchFailures do not get swallowed by other
layers of exception handling, but it also meant that a killed task could
look like a fetch failure.  This is particularly a problem with
speculative execution, where we expect to kill tasks as they are reading
shuffle data.  The fix is to ensure that we always check for killed
tasks first.
Added a new unit test which fails before the fix, ran it 1k times to
check for flakiness.  Full suite of tests on jenkins.
Author: Imran Rashid <irashid@cloudera.com>
Closes #20987 from squito/SPARK-23816.
(cherry picked from commit 10f45bb8233e6ac838dd4f053052c8556f5b54bd)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 0f2aabc6bc64d2b5d46e59525111bd95fcd73610)
The file was modifiedcore/src/main/scala/org/apache/spark/executor/Executor.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala (diff)