1. [SPARK-26706][SQL] Fix `illegalNumericPrecedence` for ByteType (commit: de3b5c459869e9ff0979e579828e822e6b01f0e3) (details)
  2. [SPARK-26706][SQL][FOLLOWUP] Fix `illegalNumericPrecedence` for ByteType (commit: 23e35d4f6cd75d3cc2a8054f8872fdf61c1a8bcc) (details)
Commit de3b5c459869e9ff0979e579828e822e6b01f0e3 by d_tsai
[SPARK-26706][SQL] Fix `illegalNumericPrecedence` for ByteType
This PR contains a minor change in `Cast$mayTruncate` that fixes its
logic for bytes.
Right now, `mayTruncate(ByteType, LongType)` returns `false` while
`mayTruncate(ShortType, LongType)` returns `true`. Consequently,
`spark.range(1, 3).as[Byte]` and `spark.range(1, 3).as[Short]` behave
Potentially, this bug can silently corrupt someone's data.
// executes silently even though Long is converted into Byte
spark.range(Long.MaxValue - 10, Long.MaxValue).as[Byte]
.map(b => b - 1)
|  -12|
|  -11|
|  -10|
|   -9|
|   -8|
|   -7|
|   -6|
|   -5|
|   -4|
|   -3|
// throws an AnalysisException: Cannot up cast `id` from bigint to
smallint as it may truncate spark.range(Long.MaxValue - 10,
.map(s => s - 1)
This PR comes with a set of unit tests.
Closes #23632 from aokolnychyi/cast-fix.
Authored-by: Anton Okolnychyi <> Signed-off-by: DB
Tsai <>
(commit: de3b5c459869e9ff0979e579828e822e6b01f0e3)
The file was modifiedsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala (diff)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala (diff)
Commit 23e35d4f6cd75d3cc2a8054f8872fdf61c1a8bcc by d_tsai
[SPARK-26706][SQL][FOLLOWUP] Fix `illegalNumericPrecedence` for ByteType
Removed automatically generated tests in
``` test("canSafeCast and mayTruncate must be consistent for numeric
since `canSafeCast` doesn't exit in 2.3 branch. We have enough test
coverages in the explict casting tests.
Authored-by: DB Tsai <> Signed-off-by: DB Tsai
(commit: 23e35d4f6cd75d3cc2a8054f8872fdf61c1a8bcc)
The file was modifiedsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala (diff)