SuccessChanges

Summary

  1. [SPARK-25454][SQL] add a new config for picking minimum precision for (commit: 26d893a4f64de18222942568f7735114447a6ab7) (details)
  2. [SPARK-25536][CORE] metric value for METRIC_OUTPUT_RECORDS_WRITTEN is (commit: f40e4c71cdb46392648c35a2f2cb0de140f3c5a8) (details)
  3. [SPARK-25533][CORE][WEBUI] AppSummary should hold the information about (commit: f13565b6ec2de2e3304b42de3a2e61da6a8ff3b0) (details)
Commit 26d893a4f64de18222942568f7735114447a6ab7 by gatorsmile
[SPARK-25454][SQL] add a new config for picking minimum precision for
integral literals
## What changes were proposed in this pull request?
https://github.com/apache/spark/pull/20023 proposed to allow precision
lose during decimal operations, to reduce the possibilities of overflow.
This is a behavior change and is protected by the
DECIMAL_OPERATIONS_ALLOW_PREC_LOSS config. However, that PR introduced
another behavior change: pick a minimum precision for integral literals,
which is not protected by a config. This PR add a new config for it:
`spark.sql.literal.pickMinimumPrecision`.
This can allow users to work around issue in SPARK-25454, which is
caused by a long-standing bug of negative scale.
## How was this patch tested?
a new test
Closes #22494 from cloud-fan/decimal.
Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by:
gatorsmile <gatorsmile@gmail.com>
(cherry picked from commit d0990e3dfee752a6460a6360e1a773138364d774)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
(commit: 26d893a4f64de18222942568f7735114447a6ab7)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala (diff)
Commit f40e4c71cdb46392648c35a2f2cb0de140f3c5a8 by dongjoon
[SPARK-25536][CORE] metric value for METRIC_OUTPUT_RECORDS_WRITTEN is
incorrect
## What changes were proposed in this pull request? changed metric value
of METRIC_OUTPUT_RECORDS_WRITTEN from
'task.metrics.inputMetrics.recordsRead' to
'task.metrics.outputMetrics.recordsWritten'. This bug was introduced in
SPARK-22190. https://github.com/apache/spark/pull/19426
## How was this patch tested? Existing tests
Closes #22555 from shahidki31/SPARK-25536.
Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Dongjoon Hyun
<dongjoon@apache.org>
(cherry picked from commit 5def10e61e49dba85f4d8b39c92bda15137990a2)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(commit: f40e4c71cdb46392648c35a2f2cb0de140f3c5a8)
The file was modifiedcore/src/main/scala/org/apache/spark/executor/Executor.scala (diff)
Commit f13565b6ec2de2e3304b42de3a2e61da6a8ff3b0 by vanzin
[SPARK-25533][CORE][WEBUI] AppSummary should hold the information about
succeeded Jobs and completed stages only
Currently, In the spark UI, when there are failed jobs or failed stages,
display message for the completed jobs and completed stages are not
consistent with the previous versions of spark. Reason is because,
AppSummary holds the information about all the jobs and stages. But, In
the below code, it checks against the completedJobs and completedStages.
So, AppSummary should hold only successful jobs and stages.
https://github.com/apache/spark/blob/66d29870c09e6050dd846336e596faaa8b0d14ad/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala#L306
https://github.com/apache/spark/blob/66d29870c09e6050dd846336e596faaa8b0d14ad/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala#L119
So, we should  keep only completed jobs and stage information in the
AppSummary, to make it consistent with Spark2.2
Test steps:
bin/spark-shell
``` sc.parallelize(1 to 5, 5).collect() sc.parallelize(1 to 5, 2).map{ x
=> throw new RuntimeException("Fail")}.collect()
```
**Before fix:**
![screenshot from 2018-09-26
03-24-53](https://user-images.githubusercontent.com/23054875/46045669-f60bcd80-c13b-11e8-9aa6-a2e5a2038dba.png)
![screenshot from 2018-09-26
03-25-08](https://user-images.githubusercontent.com/23054875/46045699-0ae86100-c13c-11e8-94e5-ad35944c7615.png)
**After fix:**
![screenshot from 2018-09-26
03-16-14](https://user-images.githubusercontent.com/23054875/46045636-d83e6880-c13b-11e8-98df-f49d15c18958.png)
![screenshot from 2018-09-26
03-16-28](https://user-images.githubusercontent.com/23054875/46045645-e1c7d080-c13b-11e8-8c9c-d32e1f663356.png)
Closes #22549 from shahidki31/SPARK-25533.
Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin
<vanzin@cloudera.com>
(cherry picked from commit 5ee21661834e837d414bc20591982a092c0aece3)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: f13565b6ec2de2e3304b42de3a2e61da6a8ff3b0)
The file was modifiedcore/src/main/scala/org/apache/spark/status/AppStatusListener.scala (diff)