SuccessChanges

Summary

  1. [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main (commit: 7a596187ee72c5131b9c8b26b5996a6e251b52be) (details)
Commit 7a596187ee72c5131b9c8b26b5996a6e251b52be by sean.owen
[SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main
resource does not honor "spark.jars.packages"
SparkSubmit determines pyspark app by the suffix of primary resource but
Livy uses "spark-internal" as the primary resource when calling
spark-submit, therefore args.isPython is set to false in
SparkSubmit.scala.
In Yarn mode, SparkSubmit module is responsible for resolving maven
coordinates and adding them to "spark.submit.pyFiles" so that python's
system path can be set correctly.
The fix is to resolve maven coordinates not only when args.isPython is
true, but also when primary resource is spark-internal.
Tested the patch with Livy submitting pyspark app, spark-submit, pyspark
with or without packages config.
Signed-off-by: Shanyu Zhao <shzhaomicrosoft.com>
Closes #23009 from shanyu/shanyu-26011.
Authored-by: Shanyu Zhao <shzhao@microsoft.com> Signed-off-by: Sean Owen
<sean.owen@databricks.com>
(cherry picked from commit 9a5fda60e532dc7203d21d5fbe385cd561906ccb)
Signed-off-by: Sean Owen <sean.owen@databricks.com>
(commit: 7a596187ee72c5131b9c8b26b5996a6e251b52be)
The file was modifiedcore/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala (diff)