SuccessChanges

Summary

  1. [SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in (commit: 28c9adbd6537c1545cc2b448c8908444ca858c44) (details)
  2. [SPARK-23838][WEBUI] Running SQL query is displayed as "completed" in (commit: a81e20314ddd1c5078ae8b25b00e1efc3cb9652d) (details)
  3. [SPARK-23637][YARN] Yarn might allocate more resource if a same executor (commit: 0b7b8cceda4ce7791d78259451b8c51b49fb2786) (details)
Commit 28c9adbd6537c1545cc2b448c8908444ca858c44 by gatorsmile
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in
unresolved state
## What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases
they're part of coalesce they will not break its type checking rules
## How was this patch tested?
Added unit test
Author: Robert Kruszewski <robertk@palantir.com>
Closes #20914 from robert3005/rk/propagate-empty-fix.
(cherry picked from commit 5cfd5fabcdbd77a806b98a6dd59b02772d2f6dee)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
(commit: 28c9adbd6537c1545cc2b448c8908444ca858c44)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala (diff)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala (diff)
Commit a81e20314ddd1c5078ae8b25b00e1efc3cb9652d by vanzin
[SPARK-23838][WEBUI] Running SQL query is displayed as "completed" in
SQL tab
## What changes were proposed in this pull request?
A running SQL query would appear as completed in the Spark UI:
![image1](https://user-images.githubusercontent.com/1097932/38170733-3d7cb00c-35bf-11e8-994c-43f2d4fa285d.png)
We can see the query in "Completed queries", while in in the job page we
see it's still running Job 132.
![image2](https://user-images.githubusercontent.com/1097932/38170735-48f2c714-35bf-11e8-8a41-6fae23543c46.png)
After some time in the query still appears in "Completed queries" (while
it's still running), but the "Duration" gets increased.
![image3](https://user-images.githubusercontent.com/1097932/38170737-50f87ea4-35bf-11e8-8b60-000f6f918964.png)
To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6.
The reason is that updates from executions are written into kvstore
periodically, and the job start event may be missed.
## How was this patch tested? Manually run the job again and check the
SQL Tab. The fix is pretty simple.
Author: Gengliang Wang <gengliang.wang@databricks.com>
Closes #20955 from gengliangwang/jobCompleted.
(cherry picked from commit d8379e5bc3629f4e8233ad42831bdaf68c24cfeb)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: a81e20314ddd1c5078ae8b25b00e1efc3cb9652d)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala (diff)
Commit 0b7b8cceda4ce7791d78259451b8c51b49fb2786 by vanzin
[SPARK-23637][YARN] Yarn might allocate more resource if a same executor
is killed multiple times.
## What changes were proposed in this pull request?
`YarnAllocator` uses `numExecutorsRunning` to track the number of
running executor. `numExecutorsRunning` is used to check if there're
executors missing and need to allocate more.
In current code, `numExecutorsRunning` can be negative when driver asks
to kill a same idle executor multiple times.
## How was this patch tested? UT added
Author: jinxing <jinxing6042@126.com>
Closes #20781 from jinxing64/SPARK-23637.
(cherry picked from commit d3bd0435ee4ff3d414f32cce3f58b6b9f67e68bc)
Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
(commit: 0b7b8cceda4ce7791d78259451b8c51b49fb2786)
The file was modifiedresource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala (diff)
The file was modifiedresource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala (diff)