1. [SPARK-23446][PYTHON] Explicitly check supported types in toPandas (commit: ccb0a59d7383db451b86aee67423eb6e28f1f982) (details)
  2. [SPARK-23381][CORE] Murmur3 hash generates a different value from other (commit: 8360da07110d847a01b243e6d786922a5057ad9f) (details)
Commit ccb0a59d7383db451b86aee67423eb6e28f1f982 by gatorsmile
[SPARK-23446][PYTHON] Explicitly check supported types in toPandas
## What changes were proposed in this pull request?
This PR explicitly specifies and checks the types we supported in
`toPandas`. This was a hole. For example, we haven't finished the binary
type support in Python side yet but now it allows as below:
```python spark.conf.set("spark.sql.execution.arrow.enabled", "false")
df = spark.createDataFrame([[bytearray("a")]]) df.toPandas()
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
    _1 0  [97]
_1 0  a
This should be disallowed. I think the same things also apply to nested
timestamps too.
I also added some nicer message about
`spark.sql.execution.arrow.enabled` in the error message.
## How was this patch tested?
Manually tested and tests added in `python/pyspark/sql/`.
Author: hyukjinkwon <>
Closes #20625 from HyukjinKwon/pandas_convertion_supported_type.
(cherry picked from commit c5857e496ff0d170ed0339f14afc7d36b192da6d)
Signed-off-by: gatorsmile <>
(commit: ccb0a59d7383db451b86aee67423eb6e28f1f982)
The file was modifiedpython/pyspark/sql/ (diff)
The file was modifiedpython/pyspark/sql/ (diff)
Commit 8360da07110d847a01b243e6d786922a5057ad9f by gatorsmile
[SPARK-23381][CORE] Murmur3 hash generates a different value from other
## What changes were proposed in this pull request? Murmur3 hash
generates a different value from the original and other implementations
(like Scala standard library and Guava or so) when the length of a bytes
array is not multiple of 4.
## How was this patch tested? Added a unit test.
**Note: When we merge this PR, please give all the credits to Shintaro
Author: Shintaro Murakami <>
Author: gatorsmile <> Author: Shintaro Murakami
Closes #20630 from gatorsmile/pr-20568.
(cherry picked from commit d5ed2108d32e1d95b26ee7fed39e8a733e935e2c)
Signed-off-by: gatorsmile <>
(commit: 8360da07110d847a01b243e6d786922a5057ad9f)
The file was modifiedcommon/sketch/src/main/java/org/apache/spark/util/sketch/ (diff)
The file was modifiedcommon/unsafe/src/test/java/org/apache/spark/unsafe/hash/ (diff)
The file was modifiedcommon/unsafe/src/main/java/org/apache/spark/unsafe/hash/ (diff)
The file was modifiedpython/pyspark/ml/ (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/feature/FeatureHasherSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala (diff)