1. [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark (details)
Commit c3eaee776509b0a23d0ba7a575575516bab4aa4e by hyukjinkwon
[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
## What changes were proposed in this pull request?
Previously Pyspark used the private constructor for SparkSession when
building that object. This resulted in a SparkSession without checking
the sql.extensions parameter for additional session extensions. To fix
this we instead use the Session.builder() path as SparkR uses, this
loads the extensions and allows their use in PySpark.
## How was this patch tested?
An integration test was added which mimics the Scala test for the same
Please review before opening a
pull request.
Closes #21990 from RussellSpitzer/SPARK-25003-master.
Authored-by: Russell Spitzer <> Signed-off-by:
hyukjinkwon <>
The file was modifiedpython/pyspark/sql/ (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala (diff)