FailedChanges

Summary

  1. [SPARK-25062][SQL] Clean up BlockLocations in InMemoryFileIndex (details)
  2. [SPARK-25575][WEBUI][FOLLOWUP] SQL tab in the spark UI support hide (details)
  3. [SPARK-25671] Build external/spark-ganglia-lgpl in Jenkins Test (details)
Commit b0cee9605e7c71cfd020aa917319478f9ac61bdb by dongjoon
[SPARK-25062][SQL] Clean up BlockLocations in InMemoryFileIndex
## What changes were proposed in this pull request?
`InMemoryFileIndex` contains a cache of `LocatedFileStatus` objects.
Each `LocatedFileStatus` object can contain several `BlockLocation`s or
some subclass of it. Filling up this cache by listing files happens
recursively either on the driver or on the executors, depending on the
parallel discovery threshold
(`spark.sql.sources.parallelPartitionDiscovery.threshold`). If the
listing happens on the executors block location objects are converted to
simple `BlockLocation` objects to ensure serialization requirements. If
it happens on the driver then there is no conversion and depending on
the file system a `BlockLocation` object can be a subclass like
`HdfsBlockLocation` and consume more memory. This PR adds the conversion
to the latter case and decreases memory consumption.
## How was this patch tested?
Added unit test.
Closes #22603 from peter-toth/SPARK-25062.
Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon
Hyun <dongjoon@apache.org>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala (diff)
Commit 756a3ab18c2032f3e84e5591725ec713e3e22726 by sean.owen
[SPARK-25575][WEBUI][FOLLOWUP] SQL tab in the spark UI support hide
tables
## What changes were proposed in this pull request? After the PR,
https://github.com/apache/spark/pull/22592, SQL tab supports collapsing
table. However, after refreshing the page, it doesn't store it previous
state. This was due to a typo in the argument list in the
collapseTablePageLoadCommand().
## How was this patch tested? bin/spark-shell
``` sql("create table a (id int)") for(i <- 1 to 100) sql(s"insert into
a values ($i)")
```
![screenshot from 2018-10-06
10-19-30](https://user-images.githubusercontent.com/23054875/46567490-59bea380-c951-11e8-9484-9aa2ee84b816.png)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
Closes #22650 from shahidki31/SPARK-25575-followUp.
Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
The file was modifiedcore/src/main/resources/org/apache/spark/ui/static/webui.js (diff)
Commit 8bb242902760535d12c6c40c5d8481a98fdc11e0 by gatorsmile
[SPARK-25671] Build external/spark-ganglia-lgpl in Jenkins Test
## What changes were proposed in this pull request? Currently, we do not
build external/spark-ganglia-lgpl in Jenkins tests when the code is
changed.
## How was this patch tested? N/A
Closes #22658 from gatorsmile/buildGanglia.
Authored-by: gatorsmile <gatorsmile@gmail.com> Signed-off-by: gatorsmile
<gatorsmile@gmail.com>
The file was modifieddev/sparktestsupport/modules.py (diff)