1. [SPARK-26560][SQL] Spark should be able to run Hive UDF using jar (details)
Commit 5d870ef0bc70527fd1bc99a4ad17e4941c923351 by wenchen
[SPARK-26560][SQL] Spark should be able to run Hive UDF using jar
regardless of current thread context classloader
### What changes were proposed in this pull request?
This patch is based on #23921 but revised to be simpler, as well as adds
UT to test the behavior.
(This patch contains the commit from #23921 to retain credit.)
Spark loads new JARs for `ADD JAR` and `CREATE FUNCTION ... USING JAR`
into jar classloader in shared state, and changes current thread's
context classloader to jar classloader as many parts of remaining codes
rely on current thread's context classloader.
This would work if the further queries will run in same thread and
there's no change on context classloader for the thread, but once the
context classloader of current thread is switched back by various
reason, Spark fails to create instance of class for the function.
This bug mostly affects spark-shell, as spark-shell will roll back
current thread's context classloader at every prompt. But it may also
affects the case of job-server, where the queries may be running in
multiple threads.
This patch fixes the issue via switching the context classloader to the
classloader which loads the class. Hopefully FunctionBuilder created by
`makeFunctionBuilder` has the information of Class as a part of closure,
hence the Class itself can be provided regardless of current thread's
context classloader.
### Why are the changes needed?
Without this patch, end users cannot execute Hive UDF using JAR twice in
### Does this PR introduce any user-facing change?
### How was this patch tested?
New UT.
Closes #27025 from HeartSaVioR/SPARK-26560-revised.
Lead-authored-by: Jungtaek Lim (HeartSaVioR)
<> Co-authored-by: nivo091
<> Signed-off-by: Wenchen Fan
The file was addedsql/hive/src/test/noclasspath/TestUDTF-spark-26560.jar
The file was addedsql/hive/src/test/noclasspath/README
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala (diff)
The file was modifiedsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala (diff)