20/08/03 12:30:53.157 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:30:53.256 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:30:53.272 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:30:53.272 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[GetGroups], valueName=Time)
20/08/03 12:30:53.275 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:30:53.391 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:30:53.426 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:30:53.447 main DEBUG Groups:  Creating new Groups object
20/08/03 12:30:53.455 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:30:53.456 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:30:53.456 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:30:53.456 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:30:53.457 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:30:53.458 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:30:53.630 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:30:53.634 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:30:53.635 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:30:53.639 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:30:53.640 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:30:53.640 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:30:53.640 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:30:53.640 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:30:53.642 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:30:53.643 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:30:53.643 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:30:53.644 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:30:53.644 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:31:00.665 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:31:00.746 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
20/08/03 12:31:00.758 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
20/08/03 12:31:00.759 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
20/08/03 12:31:00.760 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:31:00.813 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:31:00.829 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:31:00.842 main DEBUG Groups:  Creating new Groups object
20/08/03 12:31:00.846 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:31:00.847 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:31:00.847 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:31:00.847 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:31:00.848 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:31:00.848 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:31:00.951 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:31:00.954 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:31:00.954 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:31:00.958 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:31:00.958 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:31:00.958 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:31:00.958 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:31:00.959 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:31:00.961 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:31:00.961 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:31:00.962 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:31:00.962 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:31:00.963 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:31:09.756 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:31:09.820 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:363)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:363)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:31:09.862 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:31:10.012 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
20/08/03 12:31:10.035 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
20/08/03 12:31:10.045 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
20/08/03 12:31:10.049 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:31:10.148 main DEBUG Groups:  Creating new Groups object
20/08/03 12:31:10.155 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:31:10.155 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:31:10.156 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:31:10.156 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:31:10.158 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:31:10.158 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:31:10.367 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:31:10.371 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:31:10.371 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:31:10.376 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:31:10.376 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:31:10.376 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:31:10.377 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:31:10.377 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:31:11.218 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:31:11.218 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:31:11.219 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:31:11.220 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:31:11.221 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:31:32.702 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:31:32.736 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:363)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:363)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:31:32.767 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:31:32.851 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
20/08/03 12:31:32.866 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
20/08/03 12:31:32.867 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[GetGroups])
20/08/03 12:31:32.869 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:31:32.940 main DEBUG Groups:  Creating new Groups object
20/08/03 12:31:32.946 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:31:32.946 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:31:32.946 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:31:32.946 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:31:32.947 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:31:32.948 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:31:33.165 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:31:33.169 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:31:33.169 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:31:33.173 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:31:33.174 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:31:33.174 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:31:33.174 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:31:33.174 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:31:33.770 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:31:33.770 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:31:33.771 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:31:33.772 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:31:33.772 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:31:51.675 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:31:51.715 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:363)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:363)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:31:51.748 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:31:51.820 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, about=, sampleName=Ops, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:31:51.832 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, about=, sampleName=Ops, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:31:51.833 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, about=, sampleName=Ops, type=DEFAULT, value=[GetGroups], valueName=Time)
20/08/03 12:31:51.835 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:31:51.891 main DEBUG Groups:  Creating new Groups object
20/08/03 12:31:51.895 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:31:51.895 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:31:51.895 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:31:51.895 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:31:51.896 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:31:51.896 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:31:52.086 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:31:52.090 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:31:52.090 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:31:52.094 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:31:52.095 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:31:52.095 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:31:52.095 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:31:52.096 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:31:52.812 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:31:52.813 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:31:52.813 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:31:52.814 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:31:52.815 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:32:12.876 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:32:12.927 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:191)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:147)
	at org.apache.spark.deploy.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:145)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:145)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:363)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:363)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:32:12.978 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:32:13.091 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
20/08/03 12:32:13.110 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
20/08/03 12:32:13.111 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
20/08/03 12:32:13.113 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:32:13.202 main DEBUG Groups:  Creating new Groups object
20/08/03 12:32:13.210 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:32:13.210 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:32:13.210 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:32:13.210 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:32:13.212 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:32:13.212 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:32:13.411 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:32:13.415 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:32:13.415 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:32:13.421 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:32:13.421 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:32:13.422 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:32:13.422 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:32:13.423 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:32:14.152 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:32:14.152 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:32:14.153 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:32:14.154 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:32:14.154 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:32:16.361 nioEventLoopGroup-2-2 INFO SparkContext: Running Spark version 3.0.1-SNAPSHOT
20/08/03 12:32:16.403 nioEventLoopGroup-2-2 INFO ResourceUtils: ==============================================================
20/08/03 12:32:16.404 nioEventLoopGroup-2-2 INFO ResourceUtils: Resources for spark.driver:

20/08/03 12:32:16.405 nioEventLoopGroup-2-2 INFO ResourceUtils: ==============================================================
20/08/03 12:32:16.406 nioEventLoopGroup-2-2 INFO SparkContext: Submitted application: SparkR
20/08/03 12:32:16.468 nioEventLoopGroup-2-2 INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:32:16.468 nioEventLoopGroup-2-2 INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:32:16.468 nioEventLoopGroup-2-2 INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:32:16.468 nioEventLoopGroup-2-2 INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:32:16.469 nioEventLoopGroup-2-2 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:32:16.612 nioEventLoopGroup-2-2 DEBUG TransportServer: Shuffle server started on port: 37208
20/08/03 12:32:16.621 nioEventLoopGroup-2-2 INFO Utils: Successfully started service 'sparkDriver' on port 37208.
20/08/03 12:32:16.622 nioEventLoopGroup-2-2 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
20/08/03 12:32:16.692 nioEventLoopGroup-2-2 INFO SparkEnv: Registering MapOutputTracker
20/08/03 12:32:16.692 nioEventLoopGroup-2-2 DEBUG MapOutputTrackerMasterEndpoint: init
20/08/03 12:32:16.758 nioEventLoopGroup-2-2 INFO SparkEnv: Registering BlockManagerMaster
20/08/03 12:32:16.765 nioEventLoopGroup-2-2 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/08/03 12:32:16.766 nioEventLoopGroup-2-2 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/08/03 12:32:16.770 nioEventLoopGroup-2-2 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
20/08/03 12:32:16.788 nioEventLoopGroup-2-2 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-0f64468e-ee48-431c-92f1-5031f077528b
20/08/03 12:32:16.789 nioEventLoopGroup-2-2 DEBUG DiskBlockManager: Adding shutdown hook
20/08/03 12:32:16.832 nioEventLoopGroup-2-2 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
20/08/03 12:32:16.895 nioEventLoopGroup-2-2 INFO SparkEnv: Registering OutputCommitCoordinator
20/08/03 12:32:16.897 nioEventLoopGroup-2-2 DEBUG OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
20/08/03 12:32:17.051 nioEventLoopGroup-2-2 INFO SparkContext: Added JAR file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-df565cbb-2295-4dfc-89d2-c0a01154e511/sparkRTestJar-1596483129759.jar at spark://amp-jenkins-worker-06.amp:37208/jars/sparkRTestJar-1596483129759.jar with timestamp 1596483137051
20/08/03 12:32:17.053 nioEventLoopGroup-2-2 INFO SparkContext: Added file file:/home/jenkins/workspace/NewSparkPullRequestBuilder/R/pkg/tests/fulltests/jarTest.R at file:/home/jenkins/workspace/NewSparkPullRequestBuilder/R/pkg/tests/fulltests/jarTest.R with timestamp 1596483137053
20/08/03 12:32:17.056 nioEventLoopGroup-2-2 INFO Utils: Copying /home/jenkins/workspace/NewSparkPullRequestBuilder/R/pkg/tests/fulltests/jarTest.R to /tmp/spark-d5bf3d3a-05d0-4b1e-915f-1fb2b42b2992/userFiles-6d95ead8-f9d0-484c-807c-e07101afd1d7/jarTest.R
20/08/03 12:32:17.352 Finalizer DEBUG NativeAzureFileSystem: finalize() called.
20/08/03 12:32:17.352 Finalizer DEBUG NativeAzureFileSystem: finalize() called.
20/08/03 12:32:17.424 nioEventLoopGroup-2-2 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-06.amp
20/08/03 12:32:17.466 nioEventLoopGroup-2-2 DEBUG TransportServer: Shuffle server started on port: 43071
20/08/03 12:32:17.466 nioEventLoopGroup-2-2 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43071.
20/08/03 12:32:17.467 nioEventLoopGroup-2-2 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-06.amp:43071
20/08/03 12:32:17.469 nioEventLoopGroup-2-2 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/08/03 12:32:17.493 nioEventLoopGroup-2-2 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-06.amp, 43071, None)
20/08/03 12:32:17.496 dispatcher-BlockManagerMaster DEBUG DefaultTopologyMapper: Got a request for amp-jenkins-worker-06.amp
20/08/03 12:32:17.498 dispatcher-BlockManagerMaster INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-06.amp:43071 with 366.3 MiB RAM, BlockManagerId(driver, amp-jenkins-worker-06.amp, 43071, None)
20/08/03 12:32:17.524 nioEventLoopGroup-2-2 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-06.amp, 43071, None)
20/08/03 12:32:17.526 nioEventLoopGroup-2-2 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-06.amp, 43071, None)
20/08/03 12:32:18.202 nioEventLoopGroup-2-2 DEBUG log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
20/08/03 12:32:18.207 nioEventLoopGroup-2-2 INFO log: Logging initialized @8213ms to org.eclipse.jetty.util.log.Slf4jLog
20/08/03 12:32:18.269 nioEventLoopGroup-2-2 DEBUG DecoratedObjectFactory: Adding Decorator: org.eclipse.jetty.util.DeprecationWarning@5b38948c
20/08/03 12:32:18.290 nioEventLoopGroup-2-2 DEBUG ContainerLifeCycle: o.e.j.s.ServletContextHandler@7b20f142{/,null,UNAVAILABLE} added {ServletHandler@465cae93{STOPPED},MANAGED}
20/08/03 12:32:18.410 nioEventLoopGroup-2-2 DEBUG SparkContext: Adding shutdown hook
20/08/03 12:32:18.649 nioEventLoopGroup-2-2 INFO HiveConf: Found configuration file file:/home/jenkins/workspace/NewSparkPullRequestBuilder/sql/core/target/scala-2.12/test-classes/hive-site.xml
20/08/03 12:32:18.795 nioEventLoopGroup-2-2 INFO SharedState: loading hive config file: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/sql/core/target/scala-2.12/test-classes/hive-site.xml
20/08/03 12:32:18.890 nioEventLoopGroup-2-2 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/jenkins/workspace/NewSparkPullRequestBuilder/spark-warehouse').
20/08/03 12:32:18.890 nioEventLoopGroup-2-2 INFO SharedState: Warehouse path is 'file:/home/jenkins/workspace/NewSparkPullRequestBuilder/spark-warehouse'.
20/08/03 12:32:20.043 dispatcher-event-loop-0 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/08/03 12:32:20.069 nioEventLoopGroup-2-2 INFO MemoryStore: MemoryStore cleared
20/08/03 12:32:20.070 nioEventLoopGroup-2-2 INFO BlockManager: BlockManager stopped
20/08/03 12:32:20.081 nioEventLoopGroup-2-2 INFO BlockManagerMaster: BlockManagerMaster stopped
20/08/03 12:32:20.086 dispatcher-event-loop-0 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/08/03 12:32:20.127 nioEventLoopGroup-2-2 INFO SparkContext: Successfully stopped SparkContext
20/08/03 12:32:24.058 Thread-1 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:32:24.059 Thread-1 INFO ShutdownHookManager: Deleting directory /tmp/spark-0e349e6d-d749-4d6f-8f7f-4c8e593fac57
20/08/03 12:32:24.062 Thread-1 INFO ShutdownHookManager: Deleting directory /tmp/spark-d5bf3d3a-05d0-4b1e-915f-1fb2b42b2992
20/08/03 12:32:26.812 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:32:26.907 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:32:26.924 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 12:32:26.925 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[GetGroups], valueName=Time)
20/08/03 12:32:26.926 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:32:26.976 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:32:26.992 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:32:27.006 main DEBUG Groups:  Creating new Groups object
20/08/03 12:32:27.011 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:32:27.011 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:32:27.011 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:32:27.011 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:32:27.012 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:32:27.012 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:32:27.125 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:32:27.128 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:32:27.129 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:32:27.133 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:32:27.133 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:32:27.133 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:32:27.133 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:32:27.134 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:32:27.136 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:32:27.137 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:32:27.138 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:32:27.138 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:32:27.139 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:32:27.354 Thread-1 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:32:27.356 Thread-1 INFO ShutdownHookManager: Deleting directory /tmp/spark-1779d85f-7dc8-4e84-ac73-4729e47a9330
20/08/03 12:35:12.882 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:35:13.012 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
20/08/03 12:35:13.027 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
20/08/03 12:35:13.027 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
20/08/03 12:35:13.029 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 12:35:13.105 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 12:35:13.129 main DEBUG Shell: setsid exited with exit code 0
20/08/03 12:35:13.149 main DEBUG Groups:  Creating new Groups object
20/08/03 12:35:13.155 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 12:35:13.156 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 12:35:13.156 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 12:35:13.156 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 12:35:13.157 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 12:35:13.157 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 12:35:13.305 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 12:35:13.310 main DEBUG UserGroupInformation: hadoop login
20/08/03 12:35:13.310 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 12:35:13.314 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 12:35:13.314 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 12:35:13.314 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 12:35:13.314 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 12:35:13.315 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 12:35:13.317 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 12:35:13.317 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 12:35:13.318 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 12:35:13.318 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 12:35:13.319 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/08/03 12:35:25.824 Thread-1 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:35:25.825 Thread-1 INFO ShutdownHookManager: Deleting directory /tmp/spark-34dd5184-1ae8-4ddd-944e-a55191b91d7c
20/08/03 12:47:54.421 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:47:54.436 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
java.lang.RuntimeException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:65)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:47:54.444 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:47:55.847 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:47:55.862 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
java.lang.RuntimeException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:65)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:47:55.870 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:47:57.362 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:47:57.385 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
java.lang.OutOfMemoryError
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:66)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:47:57.394 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:47:59.010 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:47:59.031 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
java.lang.OutOfMemoryError
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:66)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:47:59.041 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:48:00.474 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:48:00.488 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
org.apache.spark.util.SparkFatalException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:68)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:48:00.496 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:48:01.966 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:48:01.986 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
org.apache.spark.util.SparkFatalException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:68)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:48:01.997 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:48:03.450 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:48:03.464 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
org.apache.spark.util.SparkFatalException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:70)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:48:03.472 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 12:48:05.001 Thread-0 DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 12:48:05.025 Thread-0 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Thread-0,5,main]
org.apache.spark.util.SparkFatalException
	at org.apache.spark.util.ThrowableTypes$.<init>(SparkUncaughtExceptionHandlerSuite.scala:70)
	at org.apache.spark.util.ThrowableTypes$.<clinit>(SparkUncaughtExceptionHandlerSuite.scala)
	at org.apache.spark.util.ThrowableThrower$ThrowerThread.run(SparkUncaughtExceptionHandlerSuite.scala:86)
20/08/03 12:48:05.037 Thread-2 INFO ShutdownHookManager: Shutdown hook called
20/08/03 13:17:42.689 main DEBUG ShutdownHookManager: Adding shutdown hook
20/08/03 13:17:42.784 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 13:17:42.800 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
20/08/03 13:17:42.800 main DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[GetGroups], valueName=Time)
20/08/03 13:17:42.803 main DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
20/08/03 13:17:42.856 main DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:329)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:354)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/08/03 13:17:42.875 main DEBUG Shell: setsid exited with exit code 0
20/08/03 13:17:42.893 main DEBUG Groups:  Creating new Groups object
20/08/03 13:17:42.899 main DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
20/08/03 13:17:42.899 main DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
20/08/03 13:17:42.899 main DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/08/03 13:17:42.899 main WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/08/03 13:17:42.901 main DEBUG PerformanceAdvisory: Falling back to shell based
20/08/03 13:17:42.901 main DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
20/08/03 13:17:43.022 main DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
20/08/03 13:17:43.025 main DEBUG UserGroupInformation: hadoop login
20/08/03 13:17:43.026 main DEBUG UserGroupInformation: hadoop login commit
20/08/03 13:17:43.030 main DEBUG UserGroupInformation: using local user:UnixPrincipal: jenkins
20/08/03 13:17:43.030 main DEBUG UserGroupInformation: Using user: "UnixPrincipal: jenkins" with name jenkins
20/08/03 13:17:43.031 main DEBUG UserGroupInformation: User entry: "jenkins"
20/08/03 13:17:43.031 main DEBUG UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
20/08/03 13:17:43.032 main DEBUG UserGroupInformation: UGI loginUser:jenkins (auth:SIMPLE)
20/08/03 13:17:43.034 main INFO SecurityManager: Changing view acls to: jenkins
20/08/03 13:17:43.035 main INFO SecurityManager: Changing modify acls to: jenkins
20/08/03 13:17:43.035 main INFO SecurityManager: Changing view acls groups to: 
20/08/03 13:17:43.036 main INFO SecurityManager: Changing modify acls groups to: 
20/08/03 13:17:43.037 main INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()