1. [SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map when task ends (commit: ea01e362f7427c6f16445db95982923f19c07171) (details)
  2. [DOCS] Fixed NDCG formula issues (commit: 9702bb637d5ac665fefaa96cc69c5f92553f613a) (details)
Commit ea01e362f7427c6f16445db95982923f19c07171 by hyukjinkwon
[SPARK-25144][SQL][TEST][BRANCH-2.3] Free aggregate map when task ends
## What changes were proposed in this pull request?
[SPARK-25144]( reports
memory leaks on Apache Spark 2.0.2 ~ 2.3.2-RC5.
```scala scala> case class Foo(bar: Option[String]) scala> val ds =
List(Foo(Some("bar"))).toDS scala> val result =
ds.flatMap( scala> result.rdd.isEmpty 18/08/19 23:01:54
WARN Executor: Managed memory leak detected; size = 8650752 bytes, TID =
125 res0: Boolean = false
This is a backport of cloud-fan 's which is a single commit
among 3 commits of SPARK-21743. In addition, I added a test case to
prevent regressions in branch-2.3 and branch-2.2. Although SPARK-21743
is reverted due to regression, this subpatch can go to branch-2.3 and
branch-2.2. This will be merged as cloud-fan 's commit.
## How was this patch tested?
Pass the jenkins with a newly added test case.
Closes #22150 from dongjoon-hyun/SPARK-25144.
Lead-authored-by: Wenchen Fan <> Co-authored-by:
Dongjoon Hyun <> Signed-off-by: hyukjinkwon
(commit: ea01e362f7427c6f16445db95982923f19c07171)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala (diff)
The file was modifiedsql/core/src/main/java/org/apache/spark/sql/execution/ (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala (diff)
Commit 9702bb637d5ac665fefaa96cc69c5f92553f613a by sean.owen
[DOCS] Fixed NDCG formula issues
When j is 0, log(j+1) will be 0, and this leads to division by 0 issue.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot;
otherwise, remove this)
Please review before opening a
pull request.
Closes #22090 from yueguoguo/patch-1.
Authored-by: Zhang Le <>
Signed-off-by: Sean Owen <>
(cherry picked from commit 219ed7b487c2dfb5007247f77ebf1b3cc73cecb5)
Signed-off-by: Sean Owen <>
(commit: 9702bb637d5ac665fefaa96cc69c5f92553f613a)
The file was modifieddocs/ (diff)