1. [SPARK-26233][SQL][BACKPORT-2.3] CheckOverflow when encoding a decimal (commit: 1899dd24427fc25dbedd257860075782855f138c) (details)
Commit 1899dd24427fc25dbedd257860075782855f138c by dongjoon
[SPARK-26233][SQL][BACKPORT-2.3] CheckOverflow when encoding a decimal
## What changes were proposed in this pull request?
When we encode a Decimal from external source we don't check for
overflow. That method is useful not only in order to enforce that we can
represent the correct value in the specified range, but it also changes
the underlying data to the right precision/scale. Since in our code
generation we assume that a decimal has exactly the same precision and
scale of its data type, missing to enforce it can lead to corrupted
output/results when there are subsequent transformations.
## How was this patch tested?
added UT
Closes #23233 from mgaido91/SPARK-26233_2.3.
Authored-by: Marco Gaido <> Signed-off-by:
Dongjoon Hyun <>
(commit: 1899dd24427fc25dbedd257860075782855f138c)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala (diff)