1. [SPARK-24216][SQL] Spark TypedAggregateExpression uses getSimpleName (commit: 9d63e540e00bc655faf6d8fe1d0035bc0b9a9192) (details)
Commit 9d63e540e00bc655faf6d8fe1d0035bc0b9a9192 by wenchen
[SPARK-24216][SQL] Spark TypedAggregateExpression uses getSimpleName
that is not safe in scala
When user create a aggregator object in scala and pass the aggregator to
Spark Dataset's agg() method, Spark's will initialize
TypedAggregateExpression with the nodeName field as
aggregator.getClass.getSimpleName. However, getSimpleName is not safe in
scala environment, depending on how user creates the aggregator object.
For example, if the aggregator class full qualified name is
"$myAgg$2$", the getSimpleName will throw
java.lang.InternalError "Malformed class name". This has been reported
in scalatest and
discussed in many scala upstream jiras such as SI-8110, SI-5425.
To fix this issue, we follow the solution in to add safer version of
getSimpleName as a util method, and TypedAggregateExpression will invoke
this util method rather than getClass.getSimpleName.
added unit test
Author: Fangshi Li <>
Closes #21276 from fangshil/SPARK-24216.
(cherry picked from commit cc88d7fad16e8b5cbf7b6b9bfe412908782b4a45)
Signed-off-by: Wenchen Fan <>
(commit: 9d63e540e00bc655faf6d8fe1d0035bc0b9a9192)
The file was modifiedcore/src/main/scala/org/apache/spark/util/AccumulatorV2.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala (diff)
The file was modifiedcore/src/main/scala/org/apache/spark/util/Utils.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala (diff)
The file was modifiedcore/src/test/scala/org/apache/spark/util/UtilsSuite.scala (diff)