20/07/01 10:38:03.653 pool-1-thread-1 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT 20/07/01 10:38:03.892 pool-1-thread-1 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/07/01 10:38:04.010 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:04.011 pool-1-thread-1 INFO ResourceUtils: No custom resources configured for spark.driver. 20/07/01 10:38:04.011 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:04.012 pool-1-thread-1 INFO SparkContext: Submitted application: test-sql-context 20/07/01 10:38:04.034 pool-1-thread-1 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 20/07/01 10:38:04.056 pool-1-thread-1 INFO ResourceProfile: Limiting resource is cpu 20/07/01 10:38:04.057 pool-1-thread-1 INFO ResourceProfileManager: Added ResourceProfile id: 0 20/07/01 10:38:04.123 pool-1-thread-1 INFO SecurityManager: Changing view acls to: jenkins 20/07/01 10:38:04.124 pool-1-thread-1 INFO SecurityManager: Changing modify acls to: jenkins 20/07/01 10:38:04.124 pool-1-thread-1 INFO SecurityManager: Changing view acls groups to: 20/07/01 10:38:04.124 pool-1-thread-1 INFO SecurityManager: Changing modify acls groups to: 20/07/01 10:38:04.125 pool-1-thread-1 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set() 20/07/01 10:38:04.547 pool-1-thread-1 INFO Utils: Successfully started service 'sparkDriver' on port 43937. 20/07/01 10:38:04.585 pool-1-thread-1 INFO SparkEnv: Registering MapOutputTracker 20/07/01 10:38:04.628 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMaster 20/07/01 10:38:04.634 pool-1-thread-1 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/07/01 10:38:04.635 pool-1-thread-1 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/07/01 10:38:04.641 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/07/01 10:38:04.662 pool-1-thread-1 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/blockmgr-cf3f59b7-be57-4d55-a994-2c15c645ba28 20/07/01 10:38:04.719 pool-1-thread-1 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB 20/07/01 10:38:04.737 pool-1-thread-1 INFO SparkEnv: Registering OutputCommitCoordinator 20/07/01 10:38:04.922 pool-1-thread-1 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-04.amp 20/07/01 10:38:04.949 pool-1-thread-1 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38641. 20/07/01 10:38:04.950 pool-1-thread-1 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-04.amp:38641 20/07/01 10:38:04.952 pool-1-thread-1 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/07/01 10:38:04.958 pool-1-thread-1 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 38641, None) 20/07/01 10:38:04.963 dispatcher-BlockManagerMaster INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-04.amp:38641 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-04.amp, 38641, None) 20/07/01 10:38:04.967 pool-1-thread-1 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 38641, None) 20/07/01 10:38:04.968 pool-1-thread-1 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-04.amp, 38641, None) 20/07/01 10:38:05.202 pool-1-thread-1 INFO log: Logging initialized @5456ms to org.eclipse.jetty.util.log.Slf4jLog 20/07/01 10:38:05.372 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: date' ===== 20/07/01 10:38:06.326 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SharedState: loading hive config file: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/sql/core/target/scala-2.12/test-classes/hive-site.xml 20/07/01 10:38:06.349 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV1LogicalTypeSuite'). 20/07/01 10:38:06.349 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SharedState: Warehouse path is 'file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV1LogicalTypeSuite'. 20/07/01 10:38:07.197 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 44 ms to list leaf files for 1 paths. 20/07/01 10:38:07.817 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:07.818 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:07.993 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:08.000 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:08.001 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:08.005 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:08.100 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:08.101 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:08.154 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:08.155 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:08.155 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:08.155 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:08.296 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:08.450 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:08.454 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:08.459 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 0 from apply at OutcomeOf.scala:85 20/07/01 10:38:08.470 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:08.620 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:08.642 dag-scheduler-event-loop INFO DAGScheduler: Got job 0 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:08.643 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 0 (apply at OutcomeOf.scala:85) 20/07/01 10:38:08.643 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:08.645 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:08.651 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:08.775 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:08.787 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.0 KiB, free 2.1 GiB) 20/07/01 10:38:08.788 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.0 KiB, free: 2.1 GiB) 20/07/01 10:38:08.789 dag-scheduler-event-loop INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:08.808 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:08.809 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 20/07/01 10:38:08.870 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:08.886 Executor task launch worker for task 0 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 20/07/01 10:38:09.677 Executor task launch worker for task 0 INFO CodeGenerator: Code generated in 324.977677 ms 20/07/01 10:38:09.684 Executor task launch worker for task 0 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ccee4b74-7669-4ddd-8116-39a32dbfb9ec/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:09.717 Executor task launch worker for task 0 INFO CodeGenerator: Code generated in 16.764472 ms 20/07/01 10:38:09.931 Executor task launch worker for task 0 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1597 bytes result sent to driver 20/07/01 10:38:10.013 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1154 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:10.015 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 20/07/01 10:38:10.022 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 0 (apply at OutcomeOf.scala:85) finished in 1.351 s 20/07/01 10:38:10.026 dag-scheduler-event-loop INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:10.027 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished 20/07/01 10:38:10.029 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 0 finished: apply at OutcomeOf.scala:85, took 1.408645 s 20/07/01 10:38:10.079 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.102 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:10.103 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:10.104 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 2 from apply at OutcomeOf.scala:85 20/07/01 10:38:10.105 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:10.122 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:10.123 dag-scheduler-event-loop INFO DAGScheduler: Got job 1 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:10.123 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 1 (apply at OutcomeOf.scala:85) 20/07/01 10:38:10.124 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:10.124 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:10.125 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[7] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:10.128 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:10.132 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:10.133 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:10.134 dag-scheduler-event-loop INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:10.135 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[7] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:10.135 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0 20/07/01 10:38:10.137 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:10.138 Executor task launch worker for task 1 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) 20/07/01 10:38:10.161 Executor task launch worker for task 1 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ccee4b74-7669-4ddd-8116-39a32dbfb9ec/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:10.170 Executor task launch worker for task 1 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1431 bytes result sent to driver 20/07/01 10:38:10.173 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 36 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:10.173 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 20/07/01 10:38:10.174 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 1 (apply at OutcomeOf.scala:85) finished in 0.047 s 20/07/01 10:38:10.174 dag-scheduler-event-loop INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:10.174 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished 20/07/01 10:38:10.174 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 1 finished: apply at OutcomeOf.scala:85, took 0.052113 s 20/07/01 10:38:10.197 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:10.207 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.207 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.223 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:10.223 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:10.223 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:10.223 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:10.232 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.232 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.251 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:10.252 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:10.252 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:10.252 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:10.261 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 305.8 KiB, free 2.1 GiB) 20/07/01 10:38:10.282 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 28.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.283 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:10.284 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 4 from apply at OutcomeOf.scala:85 20/07/01 10:38:10.285 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:10.299 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:10.301 dag-scheduler-event-loop INFO DAGScheduler: Got job 2 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:10.301 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 2 (apply at OutcomeOf.scala:85) 20/07/01 10:38:10.301 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:10.301 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:10.302 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[12] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:10.307 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 11.9 KiB, free 2.1 GiB) 20/07/01 10:38:10.311 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 6.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.312 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:10.313 dag-scheduler-event-loop INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:10.314 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[12] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:10.314 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks resource profile 0 20/07/01 10:38:10.315 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:10.316 Executor task launch worker for task 2 INFO Executor: Running task 0.0 in stage 2.0 (TID 2) 20/07/01 10:38:10.327 Executor task launch worker for task 2 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ccee4b74-7669-4ddd-8116-39a32dbfb9ec/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:10.336 Executor task launch worker for task 2 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1554 bytes result sent to driver 20/07/01 10:38:10.338 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 23 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:10.338 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 20/07/01 10:38:10.339 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 2 (apply at OutcomeOf.scala:85) finished in 0.036 s 20/07/01 10:38:10.339 dag-scheduler-event-loop INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:10.340 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished 20/07/01 10:38:10.340 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 2 finished: apply at OutcomeOf.scala:85, took 0.040510 s 20/07/01 10:38:10.364 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 305.8 KiB, free 2.1 GiB) 20/07/01 10:38:10.393 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 28.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.395 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:10.396 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 6 from apply at OutcomeOf.scala:85 20/07/01 10:38:10.397 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:10.411 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:10.412 dag-scheduler-event-loop INFO DAGScheduler: Got job 3 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:10.412 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 3 (apply at OutcomeOf.scala:85) 20/07/01 10:38:10.412 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:10.413 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:10.413 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:10.441 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 7.7 KiB, free 2.1 GiB) 20/07/01 10:38:10.447 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.449 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:10.451 dag-scheduler-event-loop INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:10.454 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:10.454 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks resource profile 0 20/07/01 10:38:10.458 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:10.459 Executor task launch worker for task 3 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) 20/07/01 10:38:10.465 Executor task launch worker for task 3 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ccee4b74-7669-4ddd-8116-39a32dbfb9ec/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:10.472 Executor task launch worker for task 3 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1431 bytes result sent to driver 20/07/01 10:38:10.474 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 17 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:10.475 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 20/07/01 10:38:10.476 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 3 (apply at OutcomeOf.scala:85) finished in 0.061 s 20/07/01 10:38:10.476 dag-scheduler-event-loop INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:10.476 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished 20/07/01 10:38:10.477 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 3 finished: apply at OutcomeOf.scala:85, took 0.065650 s 20/07/01 10:38:10.533 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.534 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:10.560 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:10.560 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:10.561 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:10.561 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:10.600 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 20/07/01 10:38:10.601 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:10.614 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:10.614 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:10.616 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:10.636 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:10.650 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:10.651 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:10.652 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 8 from apply at OutcomeOf.scala:85 20/07/01 10:38:10.653 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:10.683 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:10.685 dag-scheduler-event-loop INFO DAGScheduler: Got job 4 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:10.685 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 4 (apply at OutcomeOf.scala:85) 20/07/01 10:38:10.685 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:10.685 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:10.686 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[17] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:10.778 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 171.7 KiB, free 2.1 GiB) 20/07/01 10:38:10.785 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 61.1 KiB, free 2.1 GiB) 20/07/01 10:38:10.786 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.1 KiB, free: 2.1 GiB) 20/07/01 10:38:10.786 dag-scheduler-event-loop INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:10.787 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[17] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:10.787 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks resource profile 0 20/07/01 10:38:10.789 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:10.789 Executor task launch worker for task 4 INFO Executor: Running task 0.0 in stage 4.0 (TID 4) 20/07/01 10:38:10.841 Executor task launch worker for task 4 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:10.841 Executor task launch worker for task 4 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:10.842 Executor task launch worker for task 4 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:10.903 Executor task launch worker for task 4 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ccee4b74-7669-4ddd-8116-39a32dbfb9ec/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:10.964 Executor task launch worker for task 4 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103810_0004_m_000000_4' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cfd2746f-75a5-4b02-8fec-d7befc49fc2d 20/07/01 10:38:10.965 Executor task launch worker for task 4 INFO SparkHadoopMapRedUtil: attempt_20200701103810_0004_m_000000_4: Committed 20/07/01 10:38:10.971 Executor task launch worker for task 4 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 2241 bytes result sent to driver 20/07/01 10:38:10.973 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 185 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:10.973 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 20/07/01 10:38:10.974 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 4 (apply at OutcomeOf.scala:85) finished in 0.287 s 20/07/01 10:38:10.975 dag-scheduler-event-loop INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:10.975 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 4: Stage finished 20/07/01 10:38:10.976 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 4 finished: apply at OutcomeOf.scala:85, took 0.292776 s 20/07/01 10:38:10.989 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job 05b5d2a1-d62c-452d-93a1-e4f7674aa120 committed. 20/07/01 10:38:10.993 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 05b5d2a1-d62c-452d-93a1-e4f7674aa120. 20/07/01 10:38:11.018 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 2 ms to list leaf files for 1 paths. 20/07/01 10:38:11.031 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.032 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.044 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.045 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.045 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.045 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.057 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.058 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.083 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.083 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.084 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.084 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.090 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.102 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.104 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.105 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 10 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.106 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.121 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.123 dag-scheduler-event-loop INFO DAGScheduler: Got job 5 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.123 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 5 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.123 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.123 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.124 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[24] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.131 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:11.134 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 6.0 KiB, free 2.1 GiB) 20/07/01 10:38:11.135 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.0 KiB, free: 2.1 GiB) 20/07/01 10:38:11.135 dag-scheduler-event-loop INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.136 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[24] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.136 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks resource profile 0 20/07/01 10:38:11.138 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.139 Executor task launch worker for task 5 INFO Executor: Running task 0.0 in stage 5.0 (TID 5) 20/07/01 10:38:11.147 Executor task launch worker for task 5 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cfd2746f-75a5-4b02-8fec-d7befc49fc2d/part-00000-86be2c82-526d-450a-bea1-28aff022f627-c000.avro, range: 0-236, partition values: [empty row] 20/07/01 10:38:11.156 Executor task launch worker for task 5 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1554 bytes result sent to driver 20/07/01 10:38:11.158 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 21 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:11.158 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 20/07/01 10:38:11.159 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 5 (apply at OutcomeOf.scala:85) finished in 0.033 s 20/07/01 10:38:11.159 dag-scheduler-event-loop INFO DAGScheduler: Job 5 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:11.159 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished 20/07/01 10:38:11.160 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 5 finished: apply at OutcomeOf.scala:85, took 0.038865 s 20/07/01 10:38:11.175 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.189 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.190 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.191 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 12 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.192 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.217 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.218 dag-scheduler-event-loop INFO DAGScheduler: Got job 6 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.219 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 6 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.219 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.219 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.220 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[27] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.224 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:11.226 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:11.227 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:11.227 dag-scheduler-event-loop INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.228 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[27] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.228 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks resource profile 0 20/07/01 10:38:11.230 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.230 Executor task launch worker for task 6 INFO Executor: Running task 0.0 in stage 6.0 (TID 6) 20/07/01 10:38:11.235 Executor task launch worker for task 6 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cfd2746f-75a5-4b02-8fec-d7befc49fc2d/part-00000-86be2c82-526d-450a-bea1-28aff022f627-c000.avro, range: 0-236, partition values: [empty row] 20/07/01 10:38:11.242 Executor task launch worker for task 6 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1431 bytes result sent to driver 20/07/01 10:38:11.243 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 14 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:11.243 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 20/07/01 10:38:11.244 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 6 (apply at OutcomeOf.scala:85) finished in 0.022 s 20/07/01 10:38:11.245 dag-scheduler-event-loop INFO DAGScheduler: Job 6 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:11.245 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 6: Stage finished 20/07/01 10:38:11.245 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 6 finished: apply at OutcomeOf.scala:85, took 0.027795 s 20/07/01 10:38:11.284 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: date' ===== 20/07/01 10:38:11.301 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: timestamp_millis' ===== 20/07/01 10:38:11.312 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:11.364 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.365 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.394 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.394 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.395 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.395 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.404 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.405 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.431 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.439 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.453 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.453 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.454 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 14 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.455 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.481 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.483 dag-scheduler-event-loop INFO DAGScheduler: Got job 7 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.483 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 7 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.483 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.483 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.484 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[32] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.490 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:11.492 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.493 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.493 dag-scheduler-event-loop INFO SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.494 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[32] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.494 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks resource profile 0 20/07/01 10:38:11.495 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.496 Executor task launch worker for task 7 INFO Executor: Running task 0.0 in stage 7.0 (TID 7) 20/07/01 10:38:11.521 Executor task launch worker for task 7 INFO CodeGenerator: Code generated in 17.064822 ms 20/07/01 10:38:11.522 Executor task launch worker for task 7 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cf1dc467-9231-4a1d-b17c-4c1971cb9204/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:11.535 Executor task launch worker for task 7 INFO CodeGenerator: Code generated in 11.232918 ms 20/07/01 10:38:11.541 Executor task launch worker for task 7 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1554 bytes result sent to driver 20/07/01 10:38:11.542 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 47 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:11.543 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 20/07/01 10:38:11.543 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 7 (apply at OutcomeOf.scala:85) finished in 0.058 s 20/07/01 10:38:11.544 dag-scheduler-event-loop INFO DAGScheduler: Job 7 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:11.544 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 7: Stage finished 20/07/01 10:38:11.544 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 7 finished: apply at OutcomeOf.scala:85, took 0.062950 s 20/07/01 10:38:11.556 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.569 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.569 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.570 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 16 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.571 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.588 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.589 dag-scheduler-event-loop INFO DAGScheduler: Got job 8 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.589 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 8 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.589 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.589 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.590 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[35] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.592 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:11.594 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:11.595 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:11.595 dag-scheduler-event-loop INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.595 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[35] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.596 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks resource profile 0 20/07/01 10:38:11.597 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.597 Executor task launch worker for task 8 INFO Executor: Running task 0.0 in stage 8.0 (TID 8) 20/07/01 10:38:11.600 Executor task launch worker for task 8 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cf1dc467-9231-4a1d-b17c-4c1971cb9204/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:11.605 Executor task launch worker for task 8 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1426 bytes result sent to driver 20/07/01 10:38:11.606 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:11.606 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 20/07/01 10:38:11.607 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 8 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:11.607 dag-scheduler-event-loop INFO DAGScheduler: Job 8 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:11.607 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 8: Stage finished 20/07/01 10:38:11.608 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 8 finished: apply at OutcomeOf.scala:85, took 0.019327 s 20/07/01 10:38:11.622 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.622 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.645 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.646 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.646 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.646 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.653 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:11.654 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:11.654 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:11.654 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:11.668 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.685 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.685 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.687 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 18 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.688 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.727 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.729 dag-scheduler-event-loop INFO DAGScheduler: Got job 9 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.729 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 9 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.729 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.729 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.729 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[37] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.809 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 171.9 KiB, free 2.1 GiB) 20/07/01 10:38:11.812 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 61.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.814 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.2 KiB, free: 2.1 GiB) 20/07/01 10:38:11.815 dag-scheduler-event-loop INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.815 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[37] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.815 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 9.0 with 1 tasks resource profile 0 20/07/01 10:38:11.816 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.817 Executor task launch worker for task 9 INFO Executor: Running task 0.0 in stage 9.0 (TID 9) 20/07/01 10:38:11.839 Executor task launch worker for task 9 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:11.839 Executor task launch worker for task 9 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:11.839 Executor task launch worker for task 9 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:11.853 Executor task launch worker for task 9 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-cf1dc467-9231-4a1d-b17c-4c1971cb9204/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:11.862 Executor task launch worker for task 9 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103811_0009_m_000000_9' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3d198845-c5d3-448a-8a5f-834d4953bcf5 20/07/01 10:38:11.862 Executor task launch worker for task 9 INFO SparkHadoopMapRedUtil: attempt_20200701103811_0009_m_000000_9: Committed 20/07/01 10:38:11.863 Executor task launch worker for task 9 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 2198 bytes result sent to driver 20/07/01 10:38:11.864 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 48 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:11.864 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 20/07/01 10:38:11.865 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 9 (apply at OutcomeOf.scala:85) finished in 0.135 s 20/07/01 10:38:11.865 dag-scheduler-event-loop INFO DAGScheduler: Job 9 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:11.865 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 9: Stage finished 20/07/01 10:38:11.866 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 9 finished: apply at OutcomeOf.scala:85, took 0.138147 s 20/07/01 10:38:11.874 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job 8751ca9a-f89e-4c6e-921b-48a647ca61ca committed. 20/07/01 10:38:11.874 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 8751ca9a-f89e-4c6e-921b-48a647ca61ca. 20/07/01 10:38:11.884 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:11.893 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.893 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.903 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.904 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.904 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.904 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.910 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.910 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:11.925 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:11.926 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:11.926 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:11.926 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:11.931 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:11.945 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.946 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.947 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 20 from apply at OutcomeOf.scala:85 20/07/01 10:38:11.948 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:11.975 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:11.976 dag-scheduler-event-loop INFO DAGScheduler: Got job 10 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:11.976 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 10 (apply at OutcomeOf.scala:85) 20/07/01 10:38:11.976 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:11.977 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:11.977 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[44] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:11.983 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:11.985 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:11.985 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:11.986 dag-scheduler-event-loop INFO SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:11.987 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[44] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:11.987 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 10.0 with 1 tasks resource profile 0 20/07/01 10:38:11.988 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 10, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:11.989 Executor task launch worker for task 10 INFO Executor: Running task 0.0 in stage 10.0 (TID 10) 20/07/01 10:38:11.995 Executor task launch worker for task 10 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3d198845-c5d3-448a-8a5f-834d4953bcf5/part-00000-84c25e66-024b-4c7e-a1f2-6968307cb042-c000.avro, range: 0-264, partition values: [empty row] 20/07/01 10:38:12.000 Executor task launch worker for task 10 INFO Executor: Finished task 0.0 in stage 10.0 (TID 10). 1554 bytes result sent to driver 20/07/01 10:38:12.002 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 10) in 14 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.002 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 20/07/01 10:38:12.003 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 10 (apply at OutcomeOf.scala:85) finished in 0.024 s 20/07/01 10:38:12.003 dag-scheduler-event-loop INFO DAGScheduler: Job 10 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.003 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 10: Stage finished 20/07/01 10:38:12.004 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 10 finished: apply at OutcomeOf.scala:85, took 0.028896 s 20/07/01 10:38:12.014 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.025 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.026 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.027 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 22 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.028 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.046 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.047 dag-scheduler-event-loop INFO DAGScheduler: Got job 11 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.047 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 11 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.047 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.048 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.048 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[47] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.050 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:12.052 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:12.053 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:12.053 dag-scheduler-event-loop INFO SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.054 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[47] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.054 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 11.0 with 1 tasks resource profile 0 20/07/01 10:38:12.055 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 11, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.056 Executor task launch worker for task 11 INFO Executor: Running task 0.0 in stage 11.0 (TID 11) 20/07/01 10:38:12.060 Executor task launch worker for task 11 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3d198845-c5d3-448a-8a5f-834d4953bcf5/part-00000-84c25e66-024b-4c7e-a1f2-6968307cb042-c000.avro, range: 0-264, partition values: [empty row] 20/07/01 10:38:12.066 Executor task launch worker for task 11 INFO Executor: Finished task 0.0 in stage 11.0 (TID 11). 1426 bytes result sent to driver 20/07/01 10:38:12.068 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 11) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.068 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 20/07/01 10:38:12.069 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 11 (apply at OutcomeOf.scala:85) finished in 0.019 s 20/07/01 10:38:12.069 dag-scheduler-event-loop INFO DAGScheduler: Job 11 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.069 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 11: Stage finished 20/07/01 10:38:12.069 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 11 finished: apply at OutcomeOf.scala:85, took 0.022984 s 20/07/01 10:38:12.079 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: timestamp_millis' ===== 20/07/01 10:38:12.081 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: timestamp_micros' ===== 20/07/01 10:38:12.095 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:12.122 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.122 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.147 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.148 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.148 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.148 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.158 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.159 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.181 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.182 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.182 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.182 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.186 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.199 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.199 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_24_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.200 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 24 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.201 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.224 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.225 dag-scheduler-event-loop INFO DAGScheduler: Got job 12 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.225 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 12 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.225 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.225 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.226 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[52] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.229 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:12.232 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.233 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_25_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.234 dag-scheduler-event-loop INFO SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.234 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[52] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.234 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 12.0 with 1 tasks resource profile 0 20/07/01 10:38:12.235 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 12, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.236 Executor task launch worker for task 12 INFO Executor: Running task 0.0 in stage 12.0 (TID 12) 20/07/01 10:38:12.242 Executor task launch worker for task 12 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-50168b19-e9d0-4dd3-a566-5c88e2ef0362/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:12.247 Executor task launch worker for task 12 INFO Executor: Finished task 0.0 in stage 12.0 (TID 12). 1554 bytes result sent to driver 20/07/01 10:38:12.248 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 12) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.248 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 20/07/01 10:38:12.249 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 12 (apply at OutcomeOf.scala:85) finished in 0.022 s 20/07/01 10:38:12.249 dag-scheduler-event-loop INFO DAGScheduler: Job 12 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.249 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 12: Stage finished 20/07/01 10:38:12.250 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 12 finished: apply at OutcomeOf.scala:85, took 0.025619 s 20/07/01 10:38:12.263 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.274 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.275 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.276 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 26 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.277 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.294 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.295 dag-scheduler-event-loop INFO DAGScheduler: Got job 13 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.295 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 13 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.295 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.295 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.296 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[55] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.298 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:12.302 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:12.303 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_27_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:12.303 dag-scheduler-event-loop INFO SparkContext: Created broadcast 27 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.304 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[55] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.304 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 13.0 with 1 tasks resource profile 0 20/07/01 10:38:12.305 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 13.0 (TID 13, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.305 Executor task launch worker for task 13 INFO Executor: Running task 0.0 in stage 13.0 (TID 13) 20/07/01 10:38:12.309 Executor task launch worker for task 13 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-50168b19-e9d0-4dd3-a566-5c88e2ef0362/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:12.314 Executor task launch worker for task 13 INFO Executor: Finished task 0.0 in stage 13.0 (TID 13). 1426 bytes result sent to driver 20/07/01 10:38:12.315 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 13) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.315 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 20/07/01 10:38:12.316 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 13 (apply at OutcomeOf.scala:85) finished in 0.020 s 20/07/01 10:38:12.316 dag-scheduler-event-loop INFO DAGScheduler: Job 13 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.316 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 13: Stage finished 20/07/01 10:38:12.317 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 13 finished: apply at OutcomeOf.scala:85, took 0.022339 s 20/07/01 10:38:12.331 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.331 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.351 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.352 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.352 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.352 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.357 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:12.357 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:12.357 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:12.358 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:12.369 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.384 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.385 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.386 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 28 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.387 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.407 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.408 dag-scheduler-event-loop INFO DAGScheduler: Got job 14 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.409 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 14 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.409 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.409 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.409 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 14 (MapPartitionsRDD[57] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.506 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 171.9 KiB, free 2.1 GiB) 20/07/01 10:38:12.510 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 61.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.511 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.2 KiB, free: 2.1 GiB) 20/07/01 10:38:12.511 dag-scheduler-event-loop INFO SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.512 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 14 (MapPartitionsRDD[57] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.512 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 14.0 with 1 tasks resource profile 0 20/07/01 10:38:12.513 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 14, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.514 Executor task launch worker for task 14 INFO Executor: Running task 0.0 in stage 14.0 (TID 14) 20/07/01 10:38:12.537 Executor task launch worker for task 14 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:12.537 Executor task launch worker for task 14 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:12.537 Executor task launch worker for task 14 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:12.550 Executor task launch worker for task 14 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-50168b19-e9d0-4dd3-a566-5c88e2ef0362/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:12.558 Executor task launch worker for task 14 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103812_0014_m_000000_14' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-26877ddf-14ec-40e4-b21e-2e56a4d4de0d 20/07/01 10:38:12.558 Executor task launch worker for task 14 INFO SparkHadoopMapRedUtil: attempt_20200701103812_0014_m_000000_14: Committed 20/07/01 10:38:12.559 Executor task launch worker for task 14 INFO Executor: Finished task 0.0 in stage 14.0 (TID 14). 2198 bytes result sent to driver 20/07/01 10:38:12.561 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 14) in 48 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.561 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 20/07/01 10:38:12.562 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 14 (apply at OutcomeOf.scala:85) finished in 0.150 s 20/07/01 10:38:12.562 dag-scheduler-event-loop INFO DAGScheduler: Job 14 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.562 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 14: Stage finished 20/07/01 10:38:12.562 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 14 finished: apply at OutcomeOf.scala:85, took 0.154680 s 20/07/01 10:38:12.571 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job aa5323bf-d579-4e49-aa95-4b1593846036 committed. 20/07/01 10:38:12.572 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job aa5323bf-d579-4e49-aa95-4b1593846036. 20/07/01 10:38:12.582 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:12.591 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.591 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.600 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.600 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.600 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.600 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.607 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.607 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.625 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.626 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.626 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.626 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.630 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.640 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.641 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.642 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 30 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.643 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.672 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.674 dag-scheduler-event-loop INFO DAGScheduler: Got job 15 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.674 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 15 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.674 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.674 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.674 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[64] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.678 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:12.680 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.681 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.681 dag-scheduler-event-loop INFO SparkContext: Created broadcast 31 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.682 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 15 (MapPartitionsRDD[64] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.682 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 15.0 with 1 tasks resource profile 0 20/07/01 10:38:12.684 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 15, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.684 Executor task launch worker for task 15 INFO Executor: Running task 0.0 in stage 15.0 (TID 15) 20/07/01 10:38:12.693 Executor task launch worker for task 15 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-26877ddf-14ec-40e4-b21e-2e56a4d4de0d/part-00000-4e3a8fe2-d590-4fe9-b49d-46a2cb8f3e6a-c000.avro, range: 0-265, partition values: [empty row] 20/07/01 10:38:12.698 Executor task launch worker for task 15 INFO Executor: Finished task 0.0 in stage 15.0 (TID 15). 1554 bytes result sent to driver 20/07/01 10:38:12.699 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 15) in 16 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.699 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 20/07/01 10:38:12.700 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 15 (apply at OutcomeOf.scala:85) finished in 0.024 s 20/07/01 10:38:12.700 dag-scheduler-event-loop INFO DAGScheduler: Job 15 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.700 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 15: Stage finished 20/07/01 10:38:12.700 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 15 finished: apply at OutcomeOf.scala:85, took 0.027552 s 20/07/01 10:38:12.710 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.721 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.722 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.723 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 32 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.723 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.741 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.742 dag-scheduler-event-loop INFO DAGScheduler: Got job 16 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.742 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 16 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.742 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.743 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.743 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[67] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.745 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:12.748 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:12.748 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_33_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:12.749 dag-scheduler-event-loop INFO SparkContext: Created broadcast 33 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.749 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[67] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.749 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 16.0 with 1 tasks resource profile 0 20/07/01 10:38:12.751 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 16, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.752 Executor task launch worker for task 16 INFO Executor: Running task 0.0 in stage 16.0 (TID 16) 20/07/01 10:38:12.755 Executor task launch worker for task 16 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-26877ddf-14ec-40e4-b21e-2e56a4d4de0d/part-00000-4e3a8fe2-d590-4fe9-b49d-46a2cb8f3e6a-c000.avro, range: 0-265, partition values: [empty row] 20/07/01 10:38:12.761 Executor task launch worker for task 16 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 1426 bytes result sent to driver 20/07/01 10:38:12.762 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 16) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:12.762 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 20/07/01 10:38:12.763 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 16 (apply at OutcomeOf.scala:85) finished in 0.019 s 20/07/01 10:38:12.763 dag-scheduler-event-loop INFO DAGScheduler: Job 16 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:12.763 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 16: Stage finished 20/07/01 10:38:12.764 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 16 finished: apply at OutcomeOf.scala:85, took 0.022050 s 20/07/01 10:38:12.774 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: timestamp_micros' ===== 20/07/01 10:38:12.776 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: user specified output schema with different timestamp types' ===== 20/07/01 10:38:12.784 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:12.809 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.809 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:12.830 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:12.830 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:12.830 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:12.830 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:12.835 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:12.835 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:12.835 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:12.835 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:12.848 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:12.858 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.859 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:12.860 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 34 from apply at OutcomeOf.scala:85 20/07/01 10:38:12.861 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:12.883 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:12.884 dag-scheduler-event-loop INFO DAGScheduler: Got job 17 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:12.884 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 17 (apply at OutcomeOf.scala:85) 20/07/01 10:38:12.884 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:12.884 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:12.885 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[69] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:12.964 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 173.1 KiB, free 2.1 GiB) 20/07/01 10:38:12.967 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 61.5 KiB, free 2.1 GiB) 20/07/01 10:38:12.968 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.5 KiB, free: 2.1 GiB) 20/07/01 10:38:12.969 dag-scheduler-event-loop INFO SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:12.969 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[69] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:12.969 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 17.0 with 1 tasks resource profile 0 20/07/01 10:38:12.970 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 17, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:12.971 Executor task launch worker for task 17 INFO Executor: Running task 0.0 in stage 17.0 (TID 17) 20/07/01 10:38:12.988 Executor task launch worker for task 17 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:12.988 Executor task launch worker for task 17 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:12.988 Executor task launch worker for task 17 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:13.003 Executor task launch worker for task 17 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3ac48b62-8739-4014-9ebf-5d6899c69020/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:13.020 Executor task launch worker for task 17 INFO CodeGenerator: Code generated in 14.496639 ms 20/07/01 10:38:13.027 Executor task launch worker for task 17 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103812_0017_m_000000_17' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-05ec8936-c261-4a46-8122-04002b9dff87 20/07/01 10:38:13.027 Executor task launch worker for task 17 INFO SparkHadoopMapRedUtil: attempt_20200701103812_0017_m_000000_17: Committed 20/07/01 10:38:13.028 Executor task launch worker for task 17 INFO Executor: Finished task 0.0 in stage 17.0 (TID 17). 2198 bytes result sent to driver 20/07/01 10:38:13.029 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 17) in 59 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.029 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 20/07/01 10:38:13.030 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 17 (apply at OutcomeOf.scala:85) finished in 0.145 s 20/07/01 10:38:13.031 dag-scheduler-event-loop INFO DAGScheduler: Job 17 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.031 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 17: Stage finished 20/07/01 10:38:13.032 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 17 finished: apply at OutcomeOf.scala:85, took 0.148484 s 20/07/01 10:38:13.039 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job ae377a4c-cbdb-41ff-928c-f13b908a3357 committed. 20/07/01 10:38:13.040 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job ae377a4c-cbdb-41ff-928c-f13b908a3357. 20/07/01 10:38:13.050 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:13.057 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.057 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.066 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.067 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.067 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.067 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.073 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.073 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.117 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.117 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.117 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.117 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.121 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_36 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.131 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.131 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.132 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 36 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.133 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.162 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_4_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.162 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.163 dag-scheduler-event-loop INFO DAGScheduler: Got job 18 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.163 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 18 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.164 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.164 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.164 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 18 (MapPartitionsRDD[76] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.170 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 11.9 KiB, free 2.1 GiB) 20/07/01 10:38:13.171 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 6.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.172 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_37_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.173 dag-scheduler-event-loop INFO SparkContext: Created broadcast 37 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.173 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_21_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.174 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[76] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.174 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 18.0 with 1 tasks resource profile 0 20/07/01 10:38:13.175 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 18, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.176 Executor task launch worker for task 18 INFO Executor: Running task 0.0 in stage 18.0 (TID 18) 20/07/01 10:38:13.176 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_30_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.179 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_5_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.184 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_22_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.187 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_16_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.191 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_27_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.195 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_24_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.198 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_17_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.208 Executor task launch worker for task 18 INFO CodeGenerator: Code generated in 23.352707 ms 20/07/01 10:38:13.208 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_20_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.209 Executor task launch worker for task 18 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-05ec8936-c261-4a46-8122-04002b9dff87/part-00000-859bcb45-d94b-43f9-9b44-c78d6a0ab48d-c000.avro, range: 0-376, partition values: [empty row] 20/07/01 10:38:13.212 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_25_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.215 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_3_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.215 Executor task launch worker for task 18 INFO Executor: Finished task 0.0 in stage 18.0 (TID 18). 1554 bytes result sent to driver 20/07/01 10:38:13.216 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 18) in 41 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.216 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool 20/07/01 10:38:13.217 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 18 (apply at OutcomeOf.scala:85) finished in 0.052 s 20/07/01 10:38:13.218 dag-scheduler-event-loop INFO DAGScheduler: Job 18 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.218 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 18: Stage finished 20/07/01 10:38:13.218 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 18 finished: apply at OutcomeOf.scala:85, took 0.055223 s 20/07/01 10:38:13.219 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_19_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.223 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_33_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.226 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_7_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.229 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_8_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.230 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_38 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.236 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_18_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.240 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_35_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.5 KiB, free: 2.1 GiB) 20/07/01 10:38:13.243 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_32_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.244 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.245 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_38_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.245 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 38 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.247 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.247 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_23_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.250 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_14_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.253 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_15_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.256 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_31_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.258 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_12_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.260 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.263 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_13_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.266 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_11_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.270 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_28_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.270 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.270 dag-scheduler-event-loop INFO DAGScheduler: Got job 19 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.271 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 19 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.271 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.271 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.271 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[79] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.272 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 7.6 KiB, free 2.1 GiB) 20/07/01 10:38:13.273 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_10_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.274 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.274 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.274 dag-scheduler-event-loop INFO SparkContext: Created broadcast 39 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.278 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[79] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.278 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 19.0 with 1 tasks resource profile 0 20/07/01 10:38:13.279 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_9_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.280 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 19, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.280 Executor task launch worker for task 19 INFO Executor: Running task 0.0 in stage 19.0 (TID 19) 20/07/01 10:38:13.282 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_29_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.285 Executor task launch worker for task 19 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-05ec8936-c261-4a46-8122-04002b9dff87/part-00000-859bcb45-d94b-43f9-9b44-c78d6a0ab48d-c000.avro, range: 0-376, partition values: [empty row] 20/07/01 10:38:13.285 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_26_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.289 Executor task launch worker for task 19 INFO Executor: Finished task 0.0 in stage 19.0 (TID 19). 1439 bytes result sent to driver 20/07/01 10:38:13.292 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 19) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.292 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 20/07/01 10:38:13.293 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 19 (apply at OutcomeOf.scala:85) finished in 0.021 s 20/07/01 10:38:13.293 dag-scheduler-event-loop INFO DAGScheduler: Job 19 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.293 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 19: Stage finished 20/07/01 10:38:13.293 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 19 finished: apply at OutcomeOf.scala:85, took 0.023324 s 20/07/01 10:38:13.302 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: user specified output schema with different timestamp types' ===== 20/07/01 10:38:13.303 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Read Long type as Timestamp' ===== 20/07/01 10:38:13.311 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:13.324 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.324 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.335 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.336 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.336 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.336 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.341 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.341 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.356 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.356 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.357 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.357 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.360 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.371 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.372 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_40_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.372 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 40 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.373 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.397 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.398 dag-scheduler-event-loop INFO DAGScheduler: Got job 20 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.398 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 20 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.398 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.398 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.398 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[84] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.403 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:13.405 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.405 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_41_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.406 dag-scheduler-event-loop INFO SparkContext: Created broadcast 41 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.406 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[84] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.406 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 20.0 with 1 tasks resource profile 0 20/07/01 10:38:13.407 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 20, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.408 Executor task launch worker for task 20 INFO Executor: Running task 0.0 in stage 20.0 (TID 20) 20/07/01 10:38:13.413 Executor task launch worker for task 20 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6164d64-0bc8-4d57-9cce-cb10550f8975/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:13.417 Executor task launch worker for task 20 INFO Executor: Finished task 0.0 in stage 20.0 (TID 20). 1554 bytes result sent to driver 20/07/01 10:38:13.417 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 20) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.418 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 20/07/01 10:38:13.418 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 20 (apply at OutcomeOf.scala:85) finished in 0.019 s 20/07/01 10:38:13.418 dag-scheduler-event-loop INFO DAGScheduler: Job 20 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.418 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 20: Stage finished 20/07/01 10:38:13.419 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 20 finished: apply at OutcomeOf.scala:85, took 0.021654 s 20/07/01 10:38:13.427 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_42 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.437 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.438 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_42_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.438 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 42 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.439 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.454 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.455 dag-scheduler-event-loop INFO DAGScheduler: Got job 21 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.455 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 21 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.455 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.456 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.456 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[87] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.458 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:13.459 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:13.460 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_43_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:13.460 dag-scheduler-event-loop INFO SparkContext: Created broadcast 43 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.461 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[87] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.461 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 21.0 with 1 tasks resource profile 0 20/07/01 10:38:13.462 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 21, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.463 Executor task launch worker for task 21 INFO Executor: Running task 0.0 in stage 21.0 (TID 21) 20/07/01 10:38:13.466 Executor task launch worker for task 21 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6164d64-0bc8-4d57-9cce-cb10550f8975/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:13.471 Executor task launch worker for task 21 INFO Executor: Finished task 0.0 in stage 21.0 (TID 21). 1426 bytes result sent to driver 20/07/01 10:38:13.472 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 21) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.472 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 20/07/01 10:38:13.473 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 21 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:13.473 dag-scheduler-event-loop INFO DAGScheduler: Job 21 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.473 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 21: Stage finished 20/07/01 10:38:13.473 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 21 finished: apply at OutcomeOf.scala:85, took 0.018579 s 20/07/01 10:38:13.478 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Read Long type as Timestamp' ===== 20/07/01 10:38:13.480 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: user specified read schema' ===== 20/07/01 10:38:13.492 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 2 ms to list leaf files for 1 paths. 20/07/01 10:38:13.500 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.500 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.507 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.508 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.508 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.508 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.513 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.513 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.525 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.526 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.526 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.526 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.530 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.542 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 28.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.543 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.543 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 44 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.544 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.567 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.567 dag-scheduler-event-loop INFO DAGScheduler: Got job 22 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.567 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 22 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.568 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.568 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.568 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[92] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.576 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 12.5 KiB, free 2.1 GiB) 20/07/01 10:38:13.578 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 6.4 KiB, free 2.1 GiB) 20/07/01 10:38:13.579 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.4 KiB, free: 2.1 GiB) 20/07/01 10:38:13.579 dag-scheduler-event-loop INFO SparkContext: Created broadcast 45 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.580 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 22 (MapPartitionsRDD[92] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.580 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 22.0 with 1 tasks resource profile 0 20/07/01 10:38:13.581 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 22, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.581 Executor task launch worker for task 22 INFO Executor: Running task 0.0 in stage 22.0 (TID 22) 20/07/01 10:38:13.610 Executor task launch worker for task 22 INFO CodeGenerator: Code generated in 17.934381 ms 20/07/01 10:38:13.611 Executor task launch worker for task 22 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-fa5af816-8633-4ad1-b7ea-3aecaee6d02d/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:13.627 Executor task launch worker for task 22 INFO CodeGenerator: Code generated in 13.678525 ms 20/07/01 10:38:13.631 Executor task launch worker for task 22 INFO Executor: Finished task 0.0 in stage 22.0 (TID 22). 1554 bytes result sent to driver 20/07/01 10:38:13.632 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 22) in 51 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.633 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 20/07/01 10:38:13.633 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 22 (apply at OutcomeOf.scala:85) finished in 0.064 s 20/07/01 10:38:13.633 dag-scheduler-event-loop INFO DAGScheduler: Job 22 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.633 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 22: Stage finished 20/07/01 10:38:13.634 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 22 finished: apply at OutcomeOf.scala:85, took 0.066974 s 20/07/01 10:38:13.643 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.653 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 28.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.654 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.654 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 46 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.655 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.665 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.678 dag-scheduler-event-loop INFO DAGScheduler: Got job 23 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.678 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 23 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.678 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.678 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.679 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[95] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.682 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_47 stored as values in memory (estimated size 8.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.684 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_47_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 2.1 GiB) 20/07/01 10:38:13.685 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_47_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.3 KiB, free: 2.1 GiB) 20/07/01 10:38:13.685 dag-scheduler-event-loop INFO SparkContext: Created broadcast 47 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.686 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[95] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.686 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 23.0 with 1 tasks resource profile 0 20/07/01 10:38:13.687 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 23.0 (TID 23, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.688 Executor task launch worker for task 23 INFO Executor: Running task 0.0 in stage 23.0 (TID 23) 20/07/01 10:38:13.691 Executor task launch worker for task 23 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-fa5af816-8633-4ad1-b7ea-3aecaee6d02d/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:13.696 Executor task launch worker for task 23 INFO Executor: Finished task 0.0 in stage 23.0 (TID 23). 1454 bytes result sent to driver 20/07/01 10:38:13.697 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 23.0 (TID 23) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.697 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 20/07/01 10:38:13.698 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 23 (apply at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:13.698 dag-scheduler-event-loop INFO DAGScheduler: Job 23 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.698 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 23: Stage finished 20/07/01 10:38:13.698 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 23 finished: apply at OutcomeOf.scala:85, took 0.021405 s 20/07/01 10:38:13.716 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: user specified read schema' ===== 20/07/01 10:38:13.717 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: Decimal' ===== 20/07/01 10:38:13.746 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:13.754 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.754 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.762 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.762 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.762 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.762 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.767 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.767 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.777 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.778 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.778 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.778 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.781 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.791 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.792 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_48_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.793 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 48 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.794 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.807 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.808 dag-scheduler-event-loop INFO DAGScheduler: Got job 24 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.808 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 24 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.808 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.809 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.809 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[100] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.823 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49 stored as values in memory (estimated size 12.0 KiB, free 2.1 GiB) 20/07/01 10:38:13.825 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49_piece0 stored as bytes in memory (estimated size 6.3 KiB, free 2.1 GiB) 20/07/01 10:38:13.826 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_49_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:13.826 dag-scheduler-event-loop INFO SparkContext: Created broadcast 49 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.827 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[100] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.827 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 24.0 with 1 tasks resource profile 0 20/07/01 10:38:13.828 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 24, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.828 Executor task launch worker for task 24 INFO Executor: Running task 0.0 in stage 24.0 (TID 24) 20/07/01 10:38:13.853 Executor task launch worker for task 24 INFO CodeGenerator: Code generated in 16.429821 ms 20/07/01 10:38:13.855 Executor task launch worker for task 24 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2a94aa47-46d3-42c9-9721-f32b36eb3dfc/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:13.868 Executor task launch worker for task 24 INFO CodeGenerator: Code generated in 10.725594 ms 20/07/01 10:38:13.873 Executor task launch worker for task 24 INFO Executor: Finished task 0.0 in stage 24.0 (TID 24). 1554 bytes result sent to driver 20/07/01 10:38:13.875 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 24) in 46 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.875 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 20/07/01 10:38:13.876 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 24 (apply at OutcomeOf.scala:85) finished in 0.066 s 20/07/01 10:38:13.876 dag-scheduler-event-loop INFO DAGScheduler: Job 24 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.876 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 24: Stage finished 20/07/01 10:38:13.877 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 24 finished: apply at OutcomeOf.scala:85, took 0.069903 s 20/07/01 10:38:13.886 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_50 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.896 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_50_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:13.897 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_50_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:13.898 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 50 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.898 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.907 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.907 dag-scheduler-event-loop INFO DAGScheduler: Got job 25 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.908 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 25 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.908 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.908 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.908 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[103] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.910 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_51 stored as values in memory (estimated size 7.9 KiB, free 2.1 GiB) 20/07/01 10:38:13.912 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_51_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:13.912 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_51_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:13.913 dag-scheduler-event-loop INFO SparkContext: Created broadcast 51 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:13.913 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[103] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:13.913 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 25.0 with 1 tasks resource profile 0 20/07/01 10:38:13.914 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 25.0 (TID 25, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:13.915 Executor task launch worker for task 25 INFO Executor: Running task 0.0 in stage 25.0 (TID 25) 20/07/01 10:38:13.919 Executor task launch worker for task 25 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2a94aa47-46d3-42c9-9721-f32b36eb3dfc/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:13.924 Executor task launch worker for task 25 INFO Executor: Finished task 0.0 in stage 25.0 (TID 25). 1462 bytes result sent to driver 20/07/01 10:38:13.925 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 25.0 (TID 25) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:13.926 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 20/07/01 10:38:13.926 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 25 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:13.926 dag-scheduler-event-loop INFO DAGScheduler: Job 25 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:13.926 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 25: Stage finished 20/07/01 10:38:13.927 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 25 finished: apply at OutcomeOf.scala:85, took 0.019844 s 20/07/01 10:38:13.937 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:13.943 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.943 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.950 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.950 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.950 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.951 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.955 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.955 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:13.965 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:13.965 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:13.965 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:13.966 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:13.968 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_52 stored as values in memory (estimated size 306.3 KiB, free 2.1 GiB) 20/07/01 10:38:13.978 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_52_piece0 stored as bytes in memory (estimated size 28.3 KiB, free 2.1 GiB) 20/07/01 10:38:13.979 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_52_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:13.979 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 52 from apply at OutcomeOf.scala:85 20/07/01 10:38:13.980 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:13.990 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:13.991 dag-scheduler-event-loop INFO DAGScheduler: Got job 26 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:13.991 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 26 (apply at OutcomeOf.scala:85) 20/07/01 10:38:13.991 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:13.991 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:13.991 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 26 (MapPartitionsRDD[108] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:13.997 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53 stored as values in memory (estimated size 12.5 KiB, free 2.1 GiB) 20/07/01 10:38:13.999 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53_piece0 stored as bytes in memory (estimated size 6.5 KiB, free 2.1 GiB) 20/07/01 10:38:13.999 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_53_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.5 KiB, free: 2.1 GiB) 20/07/01 10:38:13.999 dag-scheduler-event-loop INFO SparkContext: Created broadcast 53 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.000 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[108] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.000 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 26.0 with 1 tasks resource profile 0 20/07/01 10:38:14.001 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 26, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.001 Executor task launch worker for task 26 INFO Executor: Running task 0.0 in stage 26.0 (TID 26) 20/07/01 10:38:14.007 Executor task launch worker for task 26 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2a94aa47-46d3-42c9-9721-f32b36eb3dfc/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:14.011 Executor task launch worker for task 26 INFO Executor: Finished task 0.0 in stage 26.0 (TID 26). 1554 bytes result sent to driver 20/07/01 10:38:14.012 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 26.0 (TID 26) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.012 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 26.0, whose tasks have all completed, from pool 20/07/01 10:38:14.013 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 26 (apply at OutcomeOf.scala:85) finished in 0.021 s 20/07/01 10:38:14.013 dag-scheduler-event-loop INFO DAGScheduler: Job 26 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.013 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 26: Stage finished 20/07/01 10:38:14.013 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 26 finished: apply at OutcomeOf.scala:85, took 0.023369 s 20/07/01 10:38:14.021 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_54 stored as values in memory (estimated size 306.3 KiB, free 2.1 GiB) 20/07/01 10:38:14.032 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_54_piece0 stored as bytes in memory (estimated size 28.3 KiB, free 2.1 GiB) 20/07/01 10:38:14.032 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_54_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.033 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 54 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.033 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.041 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.042 dag-scheduler-event-loop INFO DAGScheduler: Got job 27 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.042 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 27 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.042 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.042 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.042 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[111] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.044 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_55 stored as values in memory (estimated size 8.4 KiB, free 2.1 GiB) 20/07/01 10:38:14.046 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_55_piece0 stored as bytes in memory (estimated size 4.4 KiB, free 2.1 GiB) 20/07/01 10:38:14.047 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_55_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.4 KiB, free: 2.1 GiB) 20/07/01 10:38:14.047 dag-scheduler-event-loop INFO SparkContext: Created broadcast 55 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.048 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[111] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.048 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 27.0 with 1 tasks resource profile 0 20/07/01 10:38:14.049 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 27, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.049 Executor task launch worker for task 27 INFO Executor: Running task 0.0 in stage 27.0 (TID 27) 20/07/01 10:38:14.052 Executor task launch worker for task 27 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2a94aa47-46d3-42c9-9721-f32b36eb3dfc/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:14.056 Executor task launch worker for task 27 INFO Executor: Finished task 0.0 in stage 27.0 (TID 27). 1462 bytes result sent to driver 20/07/01 10:38:14.057 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 27) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.057 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 20/07/01 10:38:14.058 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 27 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:14.058 dag-scheduler-event-loop INFO DAGScheduler: Job 27 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.058 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 27: Stage finished 20/07/01 10:38:14.058 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 27 finished: apply at OutcomeOf.scala:85, took 0.017291 s 20/07/01 10:38:14.070 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.070 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.081 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.081 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.081 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.082 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.089 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:14.090 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:14.090 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:14.090 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:14.102 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_56 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.112 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_56_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.113 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_56_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.113 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 56 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.114 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.132 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.133 dag-scheduler-event-loop INFO DAGScheduler: Got job 28 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.133 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 28 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.133 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.134 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.134 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[113] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.210 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_57 stored as values in memory (estimated size 172.7 KiB, free 2.1 GiB) 20/07/01 10:38:14.212 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_57_piece0 stored as bytes in memory (estimated size 61.6 KiB, free 2.1 GiB) 20/07/01 10:38:14.213 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_57_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.6 KiB, free: 2.1 GiB) 20/07/01 10:38:14.213 dag-scheduler-event-loop INFO SparkContext: Created broadcast 57 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.213 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 28 (MapPartitionsRDD[113] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.213 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 28.0 with 1 tasks resource profile 0 20/07/01 10:38:14.214 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 28.0 (TID 28, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.215 Executor task launch worker for task 28 INFO Executor: Running task 0.0 in stage 28.0 (TID 28) 20/07/01 10:38:14.225 Executor task launch worker for task 28 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:14.225 Executor task launch worker for task 28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:14.226 Executor task launch worker for task 28 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:14.235 Executor task launch worker for task 28 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2a94aa47-46d3-42c9-9721-f32b36eb3dfc/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:14.242 Executor task launch worker for task 28 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103814_0028_m_000000_28' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3a612321-4f2b-4726-8f9c-68cc601851fb 20/07/01 10:38:14.242 Executor task launch worker for task 28 INFO SparkHadoopMapRedUtil: attempt_20200701103814_0028_m_000000_28: Committed 20/07/01 10:38:14.243 Executor task launch worker for task 28 INFO Executor: Finished task 0.0 in stage 28.0 (TID 28). 2198 bytes result sent to driver 20/07/01 10:38:14.243 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 28.0 (TID 28) in 29 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.244 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 20/07/01 10:38:14.244 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 28 (apply at OutcomeOf.scala:85) finished in 0.109 s 20/07/01 10:38:14.244 dag-scheduler-event-loop INFO DAGScheduler: Job 28 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.244 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 28: Stage finished 20/07/01 10:38:14.244 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 28 finished: apply at OutcomeOf.scala:85, took 0.112076 s 20/07/01 10:38:14.250 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job 631b8870-d9d5-49ba-af20-1cf77652a1ad committed. 20/07/01 10:38:14.250 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 631b8870-d9d5-49ba-af20-1cf77652a1ad. 20/07/01 10:38:14.258 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:14.265 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.265 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.271 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.271 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.271 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.272 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.276 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.276 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.286 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.287 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.287 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.287 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.290 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_58 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.299 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_58_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.300 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_58_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.301 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 58 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.301 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.311 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.312 dag-scheduler-event-loop INFO DAGScheduler: Got job 29 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.312 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 29 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.312 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.312 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.312 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[120] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.317 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59 stored as values in memory (estimated size 12.0 KiB, free 2.1 GiB) 20/07/01 10:38:14.318 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59_piece0 stored as bytes in memory (estimated size 6.3 KiB, free 2.1 GiB) 20/07/01 10:38:14.318 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_59_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.319 dag-scheduler-event-loop INFO SparkContext: Created broadcast 59 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.319 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[120] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.319 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 29.0 with 1 tasks resource profile 0 20/07/01 10:38:14.320 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 29.0 (TID 29, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.322 Executor task launch worker for task 29 INFO Executor: Running task 0.0 in stage 29.0 (TID 29) 20/07/01 10:38:14.327 Executor task launch worker for task 29 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3a612321-4f2b-4726-8f9c-68cc601851fb/part-00000-c73ffa18-7a79-4161-ac97-6c0c8d5543a5-c000.avro, range: 0-503, partition values: [empty row] 20/07/01 10:38:14.333 Executor task launch worker for task 29 INFO Executor: Finished task 0.0 in stage 29.0 (TID 29). 1554 bytes result sent to driver 20/07/01 10:38:14.334 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 29.0 (TID 29) in 14 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.334 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 20/07/01 10:38:14.334 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 29 (apply at OutcomeOf.scala:85) finished in 0.021 s 20/07/01 10:38:14.334 dag-scheduler-event-loop INFO DAGScheduler: Job 29 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.334 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 29: Stage finished 20/07/01 10:38:14.335 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 29 finished: apply at OutcomeOf.scala:85, took 0.023510 s 20/07/01 10:38:14.343 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_60 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.352 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_60_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.353 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_60_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.353 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 60 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.354 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.361 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.362 dag-scheduler-event-loop INFO DAGScheduler: Got job 30 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.362 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 30 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.362 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.362 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.362 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[123] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.364 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_61 stored as values in memory (estimated size 7.9 KiB, free 2.1 GiB) 20/07/01 10:38:14.365 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_61_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.366 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_61_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.366 dag-scheduler-event-loop INFO SparkContext: Created broadcast 61 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.366 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 30 (MapPartitionsRDD[123] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.366 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 30.0 with 1 tasks resource profile 0 20/07/01 10:38:14.367 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 30.0 (TID 30, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.368 Executor task launch worker for task 30 INFO Executor: Running task 0.0 in stage 30.0 (TID 30) 20/07/01 10:38:14.370 Executor task launch worker for task 30 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3a612321-4f2b-4726-8f9c-68cc601851fb/part-00000-c73ffa18-7a79-4161-ac97-6c0c8d5543a5-c000.avro, range: 0-503, partition values: [empty row] 20/07/01 10:38:14.375 Executor task launch worker for task 30 INFO Executor: Finished task 0.0 in stage 30.0 (TID 30). 1462 bytes result sent to driver 20/07/01 10:38:14.375 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 30.0 (TID 30) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.376 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 30.0, whose tasks have all completed, from pool 20/07/01 10:38:14.376 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 30 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:14.376 dag-scheduler-event-loop INFO DAGScheduler: Job 30 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.376 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 30: Stage finished 20/07/01 10:38:14.376 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 30 finished: apply at OutcomeOf.scala:85, took 0.015062 s 20/07/01 10:38:14.383 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: Decimal' ===== 20/07/01 10:38:14.384 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: write Decimal with BYTES type' ===== 20/07/01 10:38:14.394 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:14.413 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.414 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.427 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.427 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.427 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.428 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:14.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:14.432 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:14.433 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:14.441 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_62 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.449 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_62_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.450 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_62_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.451 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 62 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.451 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.468 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.469 dag-scheduler-event-loop INFO DAGScheduler: Got job 31 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.470 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 31 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.470 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.470 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.470 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.533 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_63 stored as values in memory (estimated size 173.9 KiB, free 2.1 GiB) 20/07/01 10:38:14.535 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_63_piece0 stored as bytes in memory (estimated size 61.7 KiB, free 2.1 GiB) 20/07/01 10:38:14.536 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_63_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 61.7 KiB, free: 2.1 GiB) 20/07/01 10:38:14.536 dag-scheduler-event-loop INFO SparkContext: Created broadcast 63 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.537 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 31 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.537 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 31.0 with 1 tasks resource profile 0 20/07/01 10:38:14.538 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 31.0 (TID 31, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7818 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.538 Executor task launch worker for task 31 INFO Executor: Running task 0.0 in stage 31.0 (TID 31) 20/07/01 10:38:14.549 Executor task launch worker for task 31 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:14.549 Executor task launch worker for task 31 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:14.550 Executor task launch worker for task 31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:14.559 Executor task launch worker for task 31 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-73ad6cc0-6775-41e9-a268-90ce9901bd84/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:14.568 Executor task launch worker for task 31 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103814_0031_m_000000_31' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e5470f6c-85b0-4cee-b784-97313ddc4dc1 20/07/01 10:38:14.568 Executor task launch worker for task 31 INFO SparkHadoopMapRedUtil: attempt_20200701103814_0031_m_000000_31: Committed 20/07/01 10:38:14.569 Executor task launch worker for task 31 INFO Executor: Finished task 0.0 in stage 31.0 (TID 31). 2198 bytes result sent to driver 20/07/01 10:38:14.571 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 31.0 (TID 31) in 33 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.571 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 20/07/01 10:38:14.572 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 31 (apply at OutcomeOf.scala:85) finished in 0.101 s 20/07/01 10:38:14.572 dag-scheduler-event-loop INFO DAGScheduler: Job 31 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.572 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 31: Stage finished 20/07/01 10:38:14.572 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 31 finished: apply at OutcomeOf.scala:85, took 0.104030 s 20/07/01 10:38:14.580 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Write Job 86408772-c6e4-4801-886a-1fd51726c390 committed. 20/07/01 10:38:14.580 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 86408772-c6e4-4801-886a-1fd51726c390. 20/07/01 10:38:14.588 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:14.595 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.595 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.604 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.604 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.604 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.604 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.609 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.609 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.619 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.619 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.619 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.620 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.623 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_64 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.632 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_64_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.633 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_64_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.634 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 64 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.634 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.645 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.647 dag-scheduler-event-loop INFO DAGScheduler: Got job 32 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.647 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 32 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.647 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.647 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.647 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 32 (MapPartitionsRDD[132] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.653 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65 stored as values in memory (estimated size 12.0 KiB, free 2.1 GiB) 20/07/01 10:38:14.654 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65_piece0 stored as bytes in memory (estimated size 6.3 KiB, free 2.1 GiB) 20/07/01 10:38:14.654 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.655 dag-scheduler-event-loop INFO SparkContext: Created broadcast 65 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.655 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 32 (MapPartitionsRDD[132] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.655 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 32.0 with 1 tasks resource profile 0 20/07/01 10:38:14.656 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 32, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.657 Executor task launch worker for task 32 INFO Executor: Running task 0.0 in stage 32.0 (TID 32) 20/07/01 10:38:14.662 Executor task launch worker for task 32 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e5470f6c-85b0-4cee-b784-97313ddc4dc1/part-00000-f1672341-08f8-4cc7-8a7c-bcd86f010bbe-c000.avro, range: 0-417, partition values: [empty row] 20/07/01 10:38:14.667 Executor task launch worker for task 32 INFO Executor: Finished task 0.0 in stage 32.0 (TID 32). 1554 bytes result sent to driver 20/07/01 10:38:14.668 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 32) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.669 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool 20/07/01 10:38:14.669 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 32 (apply at OutcomeOf.scala:85) finished in 0.021 s 20/07/01 10:38:14.669 dag-scheduler-event-loop INFO DAGScheduler: Job 32 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.669 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 32: Stage finished 20/07/01 10:38:14.669 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 32 finished: apply at OutcomeOf.scala:85, took 0.023775 s 20/07/01 10:38:14.679 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_66 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.689 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_66_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.690 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_66_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.690 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 66 from apply at OutcomeOf.scala:85 20/07/01 10:38:14.691 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.699 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:14.700 dag-scheduler-event-loop INFO DAGScheduler: Got job 33 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:14.700 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 33 (apply at OutcomeOf.scala:85) 20/07/01 10:38:14.700 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.700 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.701 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 33 (MapPartitionsRDD[135] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:14.702 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_67 stored as values in memory (estimated size 7.9 KiB, free 2.1 GiB) 20/07/01 10:38:14.704 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_67_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.704 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.705 dag-scheduler-event-loop INFO SparkContext: Created broadcast 67 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.705 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 33 (MapPartitionsRDD[135] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.705 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 33.0 with 1 tasks resource profile 0 20/07/01 10:38:14.706 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 33.0 (TID 33, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.706 Executor task launch worker for task 33 INFO Executor: Running task 0.0 in stage 33.0 (TID 33) 20/07/01 10:38:14.709 Executor task launch worker for task 33 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e5470f6c-85b0-4cee-b784-97313ddc4dc1/part-00000-f1672341-08f8-4cc7-8a7c-bcd86f010bbe-c000.avro, range: 0-417, partition values: [empty row] 20/07/01 10:38:14.717 Executor task launch worker for task 33 INFO Executor: Finished task 0.0 in stage 33.0 (TID 33). 1462 bytes result sent to driver 20/07/01 10:38:14.718 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 33.0 (TID 33) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:14.718 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 33.0, whose tasks have all completed, from pool 20/07/01 10:38:14.718 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 33 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:14.719 dag-scheduler-event-loop INFO DAGScheduler: Job 33 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:14.719 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 33: Stage finished 20/07/01 10:38:14.719 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 33 finished: apply at OutcomeOf.scala:85, took 0.019310 s 20/07/01 10:38:14.727 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: write Decimal with BYTES type' ===== 20/07/01 10:38:14.729 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: Decimal with too large precision' ===== 20/07/01 10:38:14.736 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:14.743 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.743 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:14.751 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:14.751 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:14.751 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:14.752 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:14.761 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_68 stored as values in memory (estimated size 305.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.775 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO MemoryStore: Block broadcast_68_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:14.776 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.776 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Created broadcast 68 from $anonfun$new$47 at Assertions.scala:807 20/07/01 10:38:14.777 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:14.803 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO SparkContext: Starting job: $anonfun$new$47 at Assertions.scala:807 20/07/01 10:38:14.804 dag-scheduler-event-loop INFO DAGScheduler: Got job 34 ($anonfun$new$47 at Assertions.scala:807) with 1 output partitions 20/07/01 10:38:14.804 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 34 ($anonfun$new$47 at Assertions.scala:807) 20/07/01 10:38:14.804 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:14.805 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:14.805 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 34 (MapPartitionsRDD[138] at $anonfun$new$47 at Assertions.scala:807), which has no missing parents 20/07/01 10:38:14.806 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_69 stored as values in memory (estimated size 7.8 KiB, free 2.1 GiB) 20/07/01 10:38:14.808 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_69_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:14.809 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_69_piece0 in memory on amp-jenkins-worker-04.amp:38641 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.809 dag-scheduler-event-loop INFO SparkContext: Created broadcast 69 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:14.810 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 34 (MapPartitionsRDD[138] at $anonfun$new$47 at Assertions.scala:807) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:14.810 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 34.0 with 1 tasks resource profile 0 20/07/01 10:38:14.811 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:14.811 Executor task launch worker for task 34 INFO Executor: Running task 0.0 in stage 34.0 (TID 34) 20/07/01 10:38:14.815 Executor task launch worker for task 34 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-35f34b50-c773-4f90-9564-b14476e83747.avro, range: 0-226, partition values: [empty row] 20/07/01 10:38:14.827 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_47_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.827 Executor task launch worker for task 34 INFO CodeGenerator: Code generated in 11.508091 ms 20/07/01 10:38:14.830 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_52_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.833 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_51_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.835 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_36_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.835 Executor task launch worker for task 34 ERROR Executor: Exception in task 0.0 in stage 34.0 (TID 34) java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:155) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:134) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.next(FileScanRDD.scala:96) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:341) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:14.837 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_46_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.842 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_44_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.844 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_57_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.6 KiB, free: 2.1 GiB) 20/07/01 10:38:14.846 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_64_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.849 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_50_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.852 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_60_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.854 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_67_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.856 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_65_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.858 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_66_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.860 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_34_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.862 task-result-getter-2 WARN TaskSetManager: Lost task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver): java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:155) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:134) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.next(FileScanRDD.scala:96) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:341) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:14.862 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_45_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.4 KiB, free: 2.1 GiB) 20/07/01 10:38:14.863 task-result-getter-2 ERROR TaskSetManager: Task 0 in stage 34.0 failed 1 times; aborting job 20/07/01 10:38:14.865 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_53_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.5 KiB, free: 2.1 GiB) 20/07/01 10:38:14.865 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool 20/07/01 10:38:14.868 dag-scheduler-event-loop INFO TaskSchedulerImpl: Cancelling stage 34 20/07/01 10:38:14.868 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 34: Stage cancelled 20/07/01 10:38:14.868 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_54_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.869 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 34 ($anonfun$new$47 at Assertions.scala:807) failed in 0.063 s due to Job aborted due to stage failure: Task 0 in stage 34.0 failed 1 times, most recent failure: Lost task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver): java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:155) at org.apache.spark.sql.avro.AvroFileFormat$$anon$1.next(AvroFileFormat.scala:134) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.next(FileScanRDD.scala:96) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:341) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: 20/07/01 10:38:14.869 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO DAGScheduler: Job 34 failed: $anonfun$new$47 at Assertions.scala:807, took 0.065844 s 20/07/01 10:38:14.873 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_62_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.876 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_48_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.876 pool-1-thread-1-ScalaTest-running-AvroV1LogicalTypeSuite INFO AvroV1LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV1LogicalTypeSuite: 'Logical type: Decimal with too large precision' ===== 20/07/01 10:38:14.877 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_38_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.878 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_61_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.879 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_56_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.883 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_49_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.884 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_63_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 61.7 KiB, free: 2.1 GiB) 20/07/01 10:38:14.885 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_55_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.4 KiB, free: 2.1 GiB) 20/07/01 10:38:14.886 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_41_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.887 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_58_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.888 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_37_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:14.889 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_39_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.890 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_42_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.891 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_59_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 6.3 KiB, free: 2.1 GiB) 20/07/01 10:38:14.892 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_40_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:14.893 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_43_piece0 on amp-jenkins-worker-04.amp:38641 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.367 dispatcher-event-loop-1 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/07/01 10:38:15.377 pool-1-thread-1 INFO MemoryStore: MemoryStore cleared 20/07/01 10:38:15.381 pool-1-thread-1 INFO BlockManager: BlockManager stopped 20/07/01 10:38:15.388 pool-1-thread-1 INFO BlockManagerMaster: BlockManagerMaster stopped 20/07/01 10:38:15.393 dispatcher-event-loop-1 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/07/01 10:38:15.398 pool-1-thread-1 INFO SparkContext: Successfully stopped SparkContext 20/07/01 10:38:15.401 pool-1-thread-1 WARN AvroV1LogicalTypeSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.avro.AvroV1LogicalTypeSuite, thread names: rpc-boss-3-1, shuffle-boss-6-1 ===== 20/07/01 10:38:15.412 pool-1-thread-1 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceUtils: No custom resources configured for spark.driver. 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:15.413 pool-1-thread-1 INFO SparkContext: Submitted application: test-sql-context 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceProfile: Limiting resource is cpu 20/07/01 10:38:15.413 pool-1-thread-1 INFO ResourceProfileManager: Added ResourceProfile id: 0 20/07/01 10:38:15.414 pool-1-thread-1 INFO SecurityManager: Changing view acls to: jenkins 20/07/01 10:38:15.414 pool-1-thread-1 INFO SecurityManager: Changing modify acls to: jenkins 20/07/01 10:38:15.414 pool-1-thread-1 INFO SecurityManager: Changing view acls groups to: 20/07/01 10:38:15.414 pool-1-thread-1 INFO SecurityManager: Changing modify acls groups to: 20/07/01 10:38:15.414 pool-1-thread-1 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set() 20/07/01 10:38:15.436 pool-1-thread-1 INFO Utils: Successfully started service 'sparkDriver' on port 43369. 20/07/01 10:38:15.439 pool-1-thread-1 INFO SparkEnv: Registering MapOutputTracker 20/07/01 10:38:15.440 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMaster 20/07/01 10:38:15.440 pool-1-thread-1 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/07/01 10:38:15.440 pool-1-thread-1 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/07/01 10:38:15.441 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/07/01 10:38:15.441 pool-1-thread-1 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/blockmgr-29401c8a-c39c-4f6a-8c3a-ea31fc01e92d 20/07/01 10:38:15.442 pool-1-thread-1 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB 20/07/01 10:38:15.444 pool-1-thread-1 INFO SparkEnv: Registering OutputCommitCoordinator 20/07/01 10:38:15.466 pool-1-thread-1 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-04.amp 20/07/01 10:38:15.470 pool-1-thread-1 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35137. 20/07/01 10:38:15.470 pool-1-thread-1 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-04.amp:35137 20/07/01 10:38:15.470 pool-1-thread-1 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/07/01 10:38:15.470 pool-1-thread-1 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 35137, None) 20/07/01 10:38:15.471 dispatcher-BlockManagerMaster INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-04.amp:35137 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-04.amp, 35137, None) 20/07/01 10:38:15.471 pool-1-thread-1 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 35137, None) 20/07/01 10:38:15.471 pool-1-thread-1 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-04.amp, 35137, None) 20/07/01 10:38:15.483 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: date' ===== 20/07/01 10:38:15.485 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SharedState: loading hive config file: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/sql/core/target/scala-2.12/test-classes/hive-site.xml 20/07/01 10:38:15.498 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV2LogicalTypeSuite'). 20/07/01 10:38:15.498 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SharedState: Warehouse path is 'file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV2LogicalTypeSuite'. 20/07/01 10:38:15.516 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:15.530 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.530 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.554 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro Pushed Filters: Post-Scan Filters: Output: date#179 20/07/01 10:38:15.570 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.582 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.582 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.583 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 0 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.593 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.604 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.605 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.605 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 1 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.616 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.616 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.623 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro Pushed Filters: Post-Scan Filters: Output: date#179 20/07/01 10:38:15.629 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.640 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.640 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.641 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 2 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.645 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.656 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.656 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.657 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 3 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.696 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 15.710053 ms 20/07/01 10:38:15.716 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:15.717 dag-scheduler-event-loop INFO DAGScheduler: Got job 0 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:15.717 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 0 (apply at OutcomeOf.scala:85) 20/07/01 10:38:15.717 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:15.718 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:15.718 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[5] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:15.740 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 13.7 KiB, free 2.1 GiB) 20/07/01 10:38:15.742 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 6.9 KiB, free 2.1 GiB) 20/07/01 10:38:15.742 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:15.743 dag-scheduler-event-loop INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:15.743 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:15.743 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 20/07/01 10:38:15.746 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:15.747 Executor task launch worker for task 0 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 20/07/01 10:38:15.777 Executor task launch worker for task 0 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:15.779 Executor task launch worker for task 0 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1299 bytes result sent to driver 20/07/01 10:38:15.780 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 36 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:15.780 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 20/07/01 10:38:15.780 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 0 (apply at OutcomeOf.scala:85) finished in 0.060 s 20/07/01 10:38:15.780 dag-scheduler-event-loop INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:15.780 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished 20/07/01 10:38:15.781 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 0 finished: apply at OutcomeOf.scala:85, took 0.064238 s 20/07/01 10:38:15.801 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:15.802 dag-scheduler-event-loop INFO DAGScheduler: Got job 1 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:15.802 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 1 (apply at OutcomeOf.scala:85) 20/07/01 10:38:15.802 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:15.803 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:15.803 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:15.805 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 9.2 KiB, free 2.1 GiB) 20/07/01 10:38:15.807 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.6 KiB, free 2.1 GiB) 20/07/01 10:38:15.808 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.6 KiB, free: 2.1 GiB) 20/07/01 10:38:15.808 dag-scheduler-event-loop INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:15.808 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:15.808 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0 20/07/01 10:38:15.810 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:15.810 Executor task launch worker for task 1 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) 20/07/01 10:38:15.818 Executor task launch worker for task 1 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:15.819 Executor task launch worker for task 1 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1503 bytes result sent to driver 20/07/01 10:38:15.821 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:15.821 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 20/07/01 10:38:15.821 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 1 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:15.822 dag-scheduler-event-loop INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:15.822 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished 20/07/01 10:38:15.822 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 1 finished: apply at OutcomeOf.scala:85, took 0.020359 s 20/07/01 10:38:15.832 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:15.840 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.840 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.845 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro Pushed Filters: Post-Scan Filters: Output: date#188 20/07/01 10:38:15.849 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 305.6 KiB, free 2.1 GiB) 20/07/01 10:38:15.859 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.860 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.860 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 6 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.863 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 305.6 KiB, free 2.1 GiB) 20/07/01 10:38:15.873 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.873 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.873 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 7 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.880 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.880 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.887 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro Pushed Filters: Post-Scan Filters: Output: date#188 20/07/01 10:38:15.892 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 305.6 KiB, free 2.1 GiB) 20/07/01 10:38:15.902 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.902 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.903 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 8 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.905 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 305.6 KiB, free 2.1 GiB) 20/07/01 10:38:15.915 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.915 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.916 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 9 from apply at OutcomeOf.scala:85 20/07/01 10:38:15.936 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:15.937 dag-scheduler-event-loop INFO DAGScheduler: Got job 2 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:15.937 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 2 (apply at OutcomeOf.scala:85) 20/07/01 10:38:15.937 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:15.938 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:15.938 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:15.941 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 14.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.943 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:15.943 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:15.944 dag-scheduler-event-loop INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:15.944 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:15.944 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks resource profile 0 20/07/01 10:38:15.945 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:15.945 Executor task launch worker for task 2 INFO Executor: Running task 0.0 in stage 2.0 (TID 2) 20/07/01 10:38:15.952 Executor task launch worker for task 2 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:15.953 Executor task launch worker for task 2 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1299 bytes result sent to driver 20/07/01 10:38:15.954 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:15.954 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 20/07/01 10:38:15.954 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 2 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:15.955 dag-scheduler-event-loop INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:15.955 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished 20/07/01 10:38:15.955 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 2 finished: apply at OutcomeOf.scala:85, took 0.018146 s 20/07/01 10:38:15.970 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:15.971 dag-scheduler-event-loop INFO DAGScheduler: Got job 3 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:15.971 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 3 (apply at OutcomeOf.scala:85) 20/07/01 10:38:15.972 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:15.972 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:15.972 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[19] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:15.974 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 9.4 KiB, free 2.1 GiB) 20/07/01 10:38:15.976 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 4.8 KiB, free 2.1 GiB) 20/07/01 10:38:15.976 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.8 KiB, free: 2.1 GiB) 20/07/01 10:38:15.977 dag-scheduler-event-loop INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:15.978 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[19] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:15.978 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks resource profile 0 20/07/01 10:38:15.979 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:15.979 Executor task launch worker for task 3 INFO Executor: Running task 0.0 in stage 3.0 (TID 3) 20/07/01 10:38:15.984 Executor task launch worker for task 3 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:15.986 Executor task launch worker for task 3 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1503 bytes result sent to driver 20/07/01 10:38:15.987 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:15.987 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 20/07/01 10:38:15.987 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 3 (apply at OutcomeOf.scala:85) finished in 0.014 s 20/07/01 10:38:15.987 dag-scheduler-event-loop INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:15.987 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished 20/07/01 10:38:15.988 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 3 finished: apply at OutcomeOf.scala:85, took 0.016851 s 20/07/01 10:38:15.997 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:15.998 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.005 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro Pushed Filters: Post-Scan Filters: Output: date#179 20/07/01 10:38:16.012 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.022 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.023 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.023 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 12 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.026 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.035 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.036 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.037 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 13 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.041 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:16.042 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:16.042 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:16.042 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:16.077 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.078 dag-scheduler-event-loop INFO DAGScheduler: Got job 4 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.078 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 4 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.078 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.078 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.078 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[22] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.149 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 173.7 KiB, free 2.1 GiB) 20/07/01 10:38:16.152 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 61.9 KiB, free 2.1 GiB) 20/07/01 10:38:16.153 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 61.9 KiB, free: 2.1 GiB) 20/07/01 10:38:16.153 dag-scheduler-event-loop INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.155 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[22] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.155 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks resource profile 0 20/07/01 10:38:16.156 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.157 Executor task launch worker for task 4 INFO Executor: Running task 0.0 in stage 4.0 (TID 4) 20/07/01 10:38:16.204 Executor task launch worker for task 4 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:16.204 Executor task launch worker for task 4 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:16.204 Executor task launch worker for task 4 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:16.221 Executor task launch worker for task 4 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-3432062a-ee38-4b69-b8b9-71c78fb45c73/test.avro, range: 0-181, partition values: [empty row] 20/07/01 10:38:16.226 Executor task launch worker for task 4 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103816_0004_m_000000_4' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-d70c8f38-a6a7-4ec7-b889-20163812ec93 20/07/01 10:38:16.226 Executor task launch worker for task 4 INFO SparkHadoopMapRedUtil: attempt_20200701103816_0004_m_000000_4: Committed 20/07/01 10:38:16.227 Executor task launch worker for task 4 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 2270 bytes result sent to driver 20/07/01 10:38:16.230 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 74 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.230 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 20/07/01 10:38:16.231 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 4 (apply at OutcomeOf.scala:85) finished in 0.151 s 20/07/01 10:38:16.231 dag-scheduler-event-loop INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.231 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 4: Stage finished 20/07/01 10:38:16.231 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 4 finished: apply at OutcomeOf.scala:85, took 0.154243 s 20/07/01 10:38:16.244 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job 5d3dbd4f-d334-47fd-9c71-e2c8913e8719 committed. 20/07/01 10:38:16.245 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 5d3dbd4f-d334-47fd-9c71-e2c8913e8719. 20/07/01 10:38:16.257 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 2 ms to list leaf files for 1 paths. 20/07/01 10:38:16.268 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.268 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.273 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-d70c8f38-a6a7-4ec7-b889-20163812ec93 Pushed Filters: Post-Scan Filters: Output: date#200 20/07/01 10:38:16.277 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.288 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.289 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.290 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 15 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.292 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.303 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.304 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.304 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 16 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.309 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.309 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.319 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-d70c8f38-a6a7-4ec7-b889-20163812ec93 Pushed Filters: Post-Scan Filters: Output: date#200 20/07/01 10:38:16.325 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.339 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.339 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.340 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 17 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.344 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.353 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.353 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.354 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 18 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.369 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.370 dag-scheduler-event-loop INFO DAGScheduler: Got job 5 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.370 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 5 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.370 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.370 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.370 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[30] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.374 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 13.8 KiB, free 2.1 GiB) 20/07/01 10:38:16.377 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 6.9 KiB, free 2.1 GiB) 20/07/01 10:38:16.377 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:16.378 dag-scheduler-event-loop INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.378 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[30] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.378 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks resource profile 0 20/07/01 10:38:16.379 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.379 Executor task launch worker for task 5 INFO Executor: Running task 0.0 in stage 5.0 (TID 5) 20/07/01 10:38:16.389 Executor task launch worker for task 5 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-d70c8f38-a6a7-4ec7-b889-20163812ec93/part-00000-731ccddb-b304-44d7-8b23-c6e1a6b2f77e-c000.avro, range: 0-236, partition values: [empty row] 20/07/01 10:38:16.391 Executor task launch worker for task 5 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1299 bytes result sent to driver 20/07/01 10:38:16.392 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.392 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 20/07/01 10:38:16.392 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 5 (apply at OutcomeOf.scala:85) finished in 0.021 s 20/07/01 10:38:16.393 dag-scheduler-event-loop INFO DAGScheduler: Job 5 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.393 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished 20/07/01 10:38:16.393 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 5 finished: apply at OutcomeOf.scala:85, took 0.023883 s 20/07/01 10:38:16.411 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.412 dag-scheduler-event-loop INFO DAGScheduler: Got job 6 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.412 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 6 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.412 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.412 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.412 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[34] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.414 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 9.2 KiB, free 2.1 GiB) 20/07/01 10:38:16.417 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 4.6 KiB, free 2.1 GiB) 20/07/01 10:38:16.418 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.6 KiB, free: 2.1 GiB) 20/07/01 10:38:16.418 dag-scheduler-event-loop INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.418 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[34] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.418 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks resource profile 0 20/07/01 10:38:16.419 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.420 Executor task launch worker for task 6 INFO Executor: Running task 0.0 in stage 6.0 (TID 6) 20/07/01 10:38:16.427 Executor task launch worker for task 6 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-d70c8f38-a6a7-4ec7-b889-20163812ec93/part-00000-731ccddb-b304-44d7-8b23-c6e1a6b2f77e-c000.avro, range: 0-236, partition values: [empty row] 20/07/01 10:38:16.429 Executor task launch worker for task 6 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1503 bytes result sent to driver 20/07/01 10:38:16.430 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.430 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 20/07/01 10:38:16.430 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 6 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:16.431 dag-scheduler-event-loop INFO DAGScheduler: Job 6 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.431 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 6: Stage finished 20/07/01 10:38:16.431 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 6 finished: apply at OutcomeOf.scala:85, took 0.019731 s 20/07/01 10:38:16.438 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: date' ===== 20/07/01 10:38:16.439 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: timestamp_millis' ===== 20/07/01 10:38:16.445 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:16.456 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.456 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.466 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#209 20/07/01 10:38:16.472 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.482 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.483 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.483 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 21 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.486 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.497 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.497 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.498 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 22 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.504 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.504 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.517 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#209 20/07/01 10:38:16.525 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.535 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.535 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.536 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 23 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.540 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.550 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.550 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_24_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.551 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 24 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.565 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 9.796804 ms 20/07/01 10:38:16.576 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.577 dag-scheduler-event-loop INFO DAGScheduler: Got job 7 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.577 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 7 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.577 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.577 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.578 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[40] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.581 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 13.9 KiB, free 2.1 GiB) 20/07/01 10:38:16.583 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.583 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_25_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.583 dag-scheduler-event-loop INFO SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.584 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[40] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.584 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks resource profile 0 20/07/01 10:38:16.585 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.585 Executor task launch worker for task 7 INFO Executor: Running task 0.0 in stage 7.0 (TID 7) 20/07/01 10:38:16.594 Executor task launch worker for task 7 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:16.597 Executor task launch worker for task 7 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1299 bytes result sent to driver 20/07/01 10:38:16.598 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.598 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 20/07/01 10:38:16.599 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 7 (apply at OutcomeOf.scala:85) finished in 0.020 s 20/07/01 10:38:16.599 dag-scheduler-event-loop INFO DAGScheduler: Job 7 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.599 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 7: Stage finished 20/07/01 10:38:16.599 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 7 finished: apply at OutcomeOf.scala:85, took 0.022948 s 20/07/01 10:38:16.623 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.624 dag-scheduler-event-loop INFO DAGScheduler: Got job 8 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.624 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 8 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.624 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.624 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.624 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[44] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.626 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 9.3 KiB, free 2.1 GiB) 20/07/01 10:38:16.628 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:16.628 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:16.629 dag-scheduler-event-loop INFO SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.629 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[44] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.629 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks resource profile 0 20/07/01 10:38:16.630 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.631 Executor task launch worker for task 8 INFO Executor: Running task 0.0 in stage 8.0 (TID 8) 20/07/01 10:38:16.637 Executor task launch worker for task 8 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:16.640 Executor task launch worker for task 8 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1498 bytes result sent to driver 20/07/01 10:38:16.641 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.641 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 20/07/01 10:38:16.641 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 8 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:16.642 dag-scheduler-event-loop INFO DAGScheduler: Job 8 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.642 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 8: Stage finished 20/07/01 10:38:16.642 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 8 finished: apply at OutcomeOf.scala:85, took 0.019152 s 20/07/01 10:38:16.660 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.661 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.680 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#209 20/07/01 10:38:16.687 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.698 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.698 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_27_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.699 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 27 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.702 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.713 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.713 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.714 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 28 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.719 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:16.720 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:16.720 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:16.720 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:16.750 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.751 dag-scheduler-event-loop INFO DAGScheduler: Got job 9 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.751 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 9 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.751 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.751 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.751 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[47] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.830 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 173.9 KiB, free 2.1 GiB) 20/07/01 10:38:16.833 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 62.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.833 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 62.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.834 dag-scheduler-event-loop INFO SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.834 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[47] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.834 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 9.0 with 1 tasks resource profile 0 20/07/01 10:38:16.835 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.836 Executor task launch worker for task 9 INFO Executor: Running task 0.0 in stage 9.0 (TID 9) 20/07/01 10:38:16.849 Executor task launch worker for task 9 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:16.849 Executor task launch worker for task 9 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:16.850 Executor task launch worker for task 9 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:16.865 Executor task launch worker for task 9 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d356e1c-56a4-41b1-8aa1-d595154e6a60/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:16.867 Executor task launch worker for task 9 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103816_0009_m_000000_9' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-00dac8a3-e292-42ba-8f67-4a7701d78b5b 20/07/01 10:38:16.867 Executor task launch worker for task 9 INFO SparkHadoopMapRedUtil: attempt_20200701103816_0009_m_000000_9: Committed 20/07/01 10:38:16.868 Executor task launch worker for task 9 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 2270 bytes result sent to driver 20/07/01 10:38:16.869 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 34 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:16.869 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 20/07/01 10:38:16.870 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 9 (apply at OutcomeOf.scala:85) finished in 0.118 s 20/07/01 10:38:16.870 dag-scheduler-event-loop INFO DAGScheduler: Job 9 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:16.870 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 9: Stage finished 20/07/01 10:38:16.870 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 9 finished: apply at OutcomeOf.scala:85, took 0.120138 s 20/07/01 10:38:16.876 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job a2224c48-abcd-4bd7-b1b8-40403d23651d committed. 20/07/01 10:38:16.877 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job a2224c48-abcd-4bd7-b1b8-40403d23651d. 20/07/01 10:38:16.882 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:16.888 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.888 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.892 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-00dac8a3-e292-42ba-8f67-4a7701d78b5b Pushed Filters: Post-Scan Filters: Output: timestamp_millis#226 20/07/01 10:38:16.896 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.906 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.907 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.908 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 30 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.910 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.920 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.920 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.921 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 31 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.925 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.925 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:16.932 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-00dac8a3-e292-42ba-8f67-4a7701d78b5b Pushed Filters: Post-Scan Filters: Output: timestamp_millis#226 20/07/01 10:38:16.936 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.947 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.948 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.948 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 32 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.951 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.963 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.964 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_33_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.965 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 33 from apply at OutcomeOf.scala:85 20/07/01 10:38:16.980 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:16.980 dag-scheduler-event-loop INFO DAGScheduler: Got job 10 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:16.981 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 10 (apply at OutcomeOf.scala:85) 20/07/01 10:38:16.981 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:16.981 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:16.981 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[55] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:16.985 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 13.8 KiB, free 2.1 GiB) 20/07/01 10:38:16.988 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:16.988 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:16.988 dag-scheduler-event-loop INFO SparkContext: Created broadcast 34 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:16.989 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[55] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:16.989 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 10.0 with 1 tasks resource profile 0 20/07/01 10:38:16.990 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 10, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:16.990 Executor task launch worker for task 10 INFO Executor: Running task 0.0 in stage 10.0 (TID 10) 20/07/01 10:38:16.997 Executor task launch worker for task 10 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-00dac8a3-e292-42ba-8f67-4a7701d78b5b/part-00000-1edaf654-b243-44d3-9cd0-c1b48d82e5b5-c000.avro, range: 0-264, partition values: [empty row] 20/07/01 10:38:16.999 Executor task launch worker for task 10 INFO Executor: Finished task 0.0 in stage 10.0 (TID 10). 1299 bytes result sent to driver 20/07/01 10:38:17.000 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 10) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.000 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 20/07/01 10:38:17.000 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 10 (apply at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:17.001 dag-scheduler-event-loop INFO DAGScheduler: Job 10 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.001 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 10: Stage finished 20/07/01 10:38:17.001 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 10 finished: apply at OutcomeOf.scala:85, took 0.021267 s 20/07/01 10:38:17.019 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.019 dag-scheduler-event-loop INFO DAGScheduler: Got job 11 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.019 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 11 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.020 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.020 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.020 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[59] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.022 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 9.2 KiB, free 2.1 GiB) 20/07/01 10:38:17.025 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:17.025 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.026 dag-scheduler-event-loop INFO SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.026 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[59] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.026 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 11.0 with 1 tasks resource profile 0 20/07/01 10:38:17.027 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 11, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.028 Executor task launch worker for task 11 INFO Executor: Running task 0.0 in stage 11.0 (TID 11) 20/07/01 10:38:17.033 Executor task launch worker for task 11 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-00dac8a3-e292-42ba-8f67-4a7701d78b5b/part-00000-1edaf654-b243-44d3-9cd0-c1b48d82e5b5-c000.avro, range: 0-264, partition values: [empty row] 20/07/01 10:38:17.034 Executor task launch worker for task 11 INFO Executor: Finished task 0.0 in stage 11.0 (TID 11). 1498 bytes result sent to driver 20/07/01 10:38:17.035 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 11) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.035 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 20/07/01 10:38:17.035 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 11 (apply at OutcomeOf.scala:85) finished in 0.014 s 20/07/01 10:38:17.036 dag-scheduler-event-loop INFO DAGScheduler: Job 11 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.036 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 11: Stage finished 20/07/01 10:38:17.036 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 11 finished: apply at OutcomeOf.scala:85, took 0.017038 s 20/07/01 10:38:17.044 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: timestamp_millis' ===== 20/07/01 10:38:17.045 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: timestamp_micros' ===== 20/07/01 10:38:17.050 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:17.060 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.060 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.069 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_micros#236 20/07/01 10:38:17.073 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_36 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.085 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.085 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.086 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 36 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.089 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.100 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.101 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_37_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.101 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 37 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.106 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.106 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.118 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_micros#236 20/07/01 10:38:17.123 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_38 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.135 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.136 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_38_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.136 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 38 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.139 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.151 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.151 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.152 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 39 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.167 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.168 dag-scheduler-event-loop INFO DAGScheduler: Got job 12 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.168 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 12 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.168 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.168 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.168 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[65] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.172 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 13.9 KiB, free 2.1 GiB) 20/07/01 10:38:17.175 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.176 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_40_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.176 dag-scheduler-event-loop INFO SparkContext: Created broadcast 40 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.177 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[65] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.177 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 12.0 with 1 tasks resource profile 0 20/07/01 10:38:17.178 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 12, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.178 Executor task launch worker for task 12 INFO Executor: Running task 0.0 in stage 12.0 (TID 12) 20/07/01 10:38:17.184 Executor task launch worker for task 12 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:17.185 Executor task launch worker for task 12 INFO Executor: Finished task 0.0 in stage 12.0 (TID 12). 1299 bytes result sent to driver 20/07/01 10:38:17.186 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 12) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.186 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 20/07/01 10:38:17.187 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 12 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:17.187 dag-scheduler-event-loop INFO DAGScheduler: Job 12 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.187 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 12: Stage finished 20/07/01 10:38:17.187 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 12 finished: apply at OutcomeOf.scala:85, took 0.020188 s 20/07/01 10:38:17.203 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.204 dag-scheduler-event-loop INFO DAGScheduler: Got job 13 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.204 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 13 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.204 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.204 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.204 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[69] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.205 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 9.3 KiB, free 2.1 GiB) 20/07/01 10:38:17.208 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:17.209 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_41_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.209 dag-scheduler-event-loop INFO SparkContext: Created broadcast 41 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.210 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[69] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.210 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 13.0 with 1 tasks resource profile 0 20/07/01 10:38:17.210 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 13.0 (TID 13, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.211 Executor task launch worker for task 13 INFO Executor: Running task 0.0 in stage 13.0 (TID 13) 20/07/01 10:38:17.216 Executor task launch worker for task 13 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:17.217 Executor task launch worker for task 13 INFO Executor: Finished task 0.0 in stage 13.0 (TID 13). 1498 bytes result sent to driver 20/07/01 10:38:17.218 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 13) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.218 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 20/07/01 10:38:17.219 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 13 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:17.219 dag-scheduler-event-loop INFO DAGScheduler: Job 13 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.219 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 13: Stage finished 20/07/01 10:38:17.219 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 13 finished: apply at OutcomeOf.scala:85, took 0.015828 s 20/07/01 10:38:17.228 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.228 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.261 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_micros#236 20/07/01 10:38:17.267 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_42 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.277 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.277 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_42_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.278 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 42 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.280 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.290 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.290 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_43_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.291 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 43 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.292 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_10_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.294 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_8_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.295 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:17.295 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:17.295 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:17.296 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:17.297 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_22_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.301 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_18_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.303 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_5_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.6 KiB, free: 2.1 GiB) 20/07/01 10:38:17.305 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_7_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.307 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_29_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 62.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.309 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_41_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.311 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_11_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.8 KiB, free: 2.1 GiB) 20/07/01 10:38:17.312 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_17_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.314 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_34_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.317 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_4_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:17.319 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_2_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.320 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_0_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.322 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_19_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:17.324 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_32_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.326 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_21_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.327 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.328 dag-scheduler-event-loop INFO DAGScheduler: Got job 14 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.328 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 14 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.328 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.328 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.328 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_15_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.328 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 14 (MapPartitionsRDD[72] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.330 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_28_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.332 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_23_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.334 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_16_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.336 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_40_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.338 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_35_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.340 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_24_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.342 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_20_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.6 KiB, free: 2.1 GiB) 20/07/01 10:38:17.344 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.346 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_3_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.348 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_25_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.350 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_13_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.352 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_1_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.353 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_33_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.355 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_14_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 61.9 KiB, free: 2.1 GiB) 20/07/01 10:38:17.357 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_30_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.359 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_12_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.360 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_9_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.362 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_27_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.363 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_26_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.364 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_31_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.408 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 173.9 KiB, free 2.1 GiB) 20/07/01 10:38:17.410 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 62.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.411 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 62.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.411 dag-scheduler-event-loop INFO SparkContext: Created broadcast 44 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.411 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 14 (MapPartitionsRDD[72] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.411 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 14.0 with 1 tasks resource profile 0 20/07/01 10:38:17.412 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 14, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.413 Executor task launch worker for task 14 INFO Executor: Running task 0.0 in stage 14.0 (TID 14) 20/07/01 10:38:17.425 Executor task launch worker for task 14 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:17.425 Executor task launch worker for task 14 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:17.425 Executor task launch worker for task 14 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:17.441 Executor task launch worker for task 14 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-1915ee69-9018-423a-a31b-dc29d9054db1/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:17.443 Executor task launch worker for task 14 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103817_0014_m_000000_14' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4ce6f0ed-dd8d-4953-bafd-91185bcc6a5a 20/07/01 10:38:17.443 Executor task launch worker for task 14 INFO SparkHadoopMapRedUtil: attempt_20200701103817_0014_m_000000_14: Committed 20/07/01 10:38:17.444 Executor task launch worker for task 14 INFO Executor: Finished task 0.0 in stage 14.0 (TID 14). 2270 bytes result sent to driver 20/07/01 10:38:17.445 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 14) in 33 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.445 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 20/07/01 10:38:17.445 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 14 (apply at OutcomeOf.scala:85) finished in 0.116 s 20/07/01 10:38:17.446 dag-scheduler-event-loop INFO DAGScheduler: Job 14 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.446 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 14: Stage finished 20/07/01 10:38:17.446 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 14 finished: apply at OutcomeOf.scala:85, took 0.118672 s 20/07/01 10:38:17.454 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job 2086b95a-7ffc-4f95-bd73-0babdb403b3c committed. 20/07/01 10:38:17.454 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 2086b95a-7ffc-4f95-bd73-0babdb403b3c. 20/07/01 10:38:17.462 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:17.470 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.470 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.476 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4ce6f0ed-dd8d-4953-bafd-91185bcc6a5a Pushed Filters: Post-Scan Filters: Output: timestamp_micros#252 20/07/01 10:38:17.482 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.493 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.494 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.494 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 45 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.497 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.507 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.507 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.508 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 46 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.513 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.513 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.520 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4ce6f0ed-dd8d-4953-bafd-91185bcc6a5a Pushed Filters: Post-Scan Filters: Output: timestamp_micros#252 20/07/01 10:38:17.525 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_47 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.535 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_47_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.536 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_47_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.536 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 47 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.539 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.548 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.549 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_48_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.549 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 48 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.565 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.567 dag-scheduler-event-loop INFO DAGScheduler: Got job 15 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.568 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 15 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.568 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.568 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.568 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[80] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.572 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49 stored as values in memory (estimated size 13.8 KiB, free 2.1 GiB) 20/07/01 10:38:17.574 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.575 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_49_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.575 dag-scheduler-event-loop INFO SparkContext: Created broadcast 49 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.576 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 15 (MapPartitionsRDD[80] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.576 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 15.0 with 1 tasks resource profile 0 20/07/01 10:38:17.577 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 15, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.578 Executor task launch worker for task 15 INFO Executor: Running task 0.0 in stage 15.0 (TID 15) 20/07/01 10:38:17.586 Executor task launch worker for task 15 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4ce6f0ed-dd8d-4953-bafd-91185bcc6a5a/part-00000-5352a598-e64d-4e9c-9766-92b6f1c7ad61-c000.avro, range: 0-265, partition values: [empty row] 20/07/01 10:38:17.587 Executor task launch worker for task 15 INFO Executor: Finished task 0.0 in stage 15.0 (TID 15). 1299 bytes result sent to driver 20/07/01 10:38:17.588 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 15) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.588 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 20/07/01 10:38:17.588 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 15 (apply at OutcomeOf.scala:85) finished in 0.020 s 20/07/01 10:38:17.589 dag-scheduler-event-loop INFO DAGScheduler: Job 15 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.589 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 15: Stage finished 20/07/01 10:38:17.589 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 15 finished: apply at OutcomeOf.scala:85, took 0.022178 s 20/07/01 10:38:17.609 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.610 dag-scheduler-event-loop INFO DAGScheduler: Got job 16 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.610 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 16 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.610 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.610 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.611 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[84] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.612 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_50 stored as values in memory (estimated size 9.2 KiB, free 2.1 GiB) 20/07/01 10:38:17.614 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_50_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:17.615 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_50_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:17.615 dag-scheduler-event-loop INFO SparkContext: Created broadcast 50 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.616 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[84] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.616 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 16.0 with 1 tasks resource profile 0 20/07/01 10:38:17.617 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 16, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.618 Executor task launch worker for task 16 INFO Executor: Running task 0.0 in stage 16.0 (TID 16) 20/07/01 10:38:17.623 Executor task launch worker for task 16 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4ce6f0ed-dd8d-4953-bafd-91185bcc6a5a/part-00000-5352a598-e64d-4e9c-9766-92b6f1c7ad61-c000.avro, range: 0-265, partition values: [empty row] 20/07/01 10:38:17.624 Executor task launch worker for task 16 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 1498 bytes result sent to driver 20/07/01 10:38:17.625 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 16) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.625 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 20/07/01 10:38:17.626 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 16 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:17.626 dag-scheduler-event-loop INFO DAGScheduler: Job 16 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.626 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 16: Stage finished 20/07/01 10:38:17.626 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 16 finished: apply at OutcomeOf.scala:85, took 0.016846 s 20/07/01 10:38:17.634 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: timestamp_micros' ===== 20/07/01 10:38:17.635 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: user specified output schema with different timestamp types' ===== 20/07/01 10:38:17.641 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:17.658 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.658 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.672 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-a1aec92b-85a8-4a17-a0f0-4849504ad36e/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#261, timestamp_micros#262 20/07/01 10:38:17.678 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_51 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.688 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_51_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.688 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_51_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.689 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 51 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.691 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_52 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.701 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_52_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.701 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_52_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.702 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 52 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.706 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:17.707 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:17.707 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:17.707 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:17.728 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 10.400627 ms 20/07/01 10:38:17.747 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:17.748 dag-scheduler-event-loop INFO DAGScheduler: Got job 17 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:17.748 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 17 (apply at OutcomeOf.scala:85) 20/07/01 10:38:17.748 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:17.748 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:17.749 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[87] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:17.816 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53 stored as values in memory (estimated size 175.3 KiB, free 2.1 GiB) 20/07/01 10:38:17.818 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53_piece0 stored as bytes in memory (estimated size 62.3 KiB, free 2.1 GiB) 20/07/01 10:38:17.819 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_53_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 62.3 KiB, free: 2.1 GiB) 20/07/01 10:38:17.819 dag-scheduler-event-loop INFO SparkContext: Created broadcast 53 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:17.819 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[87] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:17.820 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 17.0 with 1 tasks resource profile 0 20/07/01 10:38:17.821 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 17, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:17.821 Executor task launch worker for task 17 INFO Executor: Running task 0.0 in stage 17.0 (TID 17) 20/07/01 10:38:17.833 Executor task launch worker for task 17 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:17.833 Executor task launch worker for task 17 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:17.834 Executor task launch worker for task 17 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:17.847 Executor task launch worker for task 17 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-a1aec92b-85a8-4a17-a0f0-4849504ad36e/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:17.849 Executor task launch worker for task 17 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103817_0017_m_000000_17' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-20287591-bad5-48ed-86f5-ab4a9e745a85 20/07/01 10:38:17.849 Executor task launch worker for task 17 INFO SparkHadoopMapRedUtil: attempt_20200701103817_0017_m_000000_17: Committed 20/07/01 10:38:17.850 Executor task launch worker for task 17 INFO Executor: Finished task 0.0 in stage 17.0 (TID 17). 2270 bytes result sent to driver 20/07/01 10:38:17.851 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 17) in 31 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:17.851 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 20/07/01 10:38:17.852 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 17 (apply at OutcomeOf.scala:85) finished in 0.102 s 20/07/01 10:38:17.852 dag-scheduler-event-loop INFO DAGScheduler: Job 17 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:17.852 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 17: Stage finished 20/07/01 10:38:17.853 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 17 finished: apply at OutcomeOf.scala:85, took 0.105257 s 20/07/01 10:38:17.859 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job fd26754c-86a2-4da5-aafb-b9f976b10b48 committed. 20/07/01 10:38:17.859 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job fd26754c-86a2-4da5-aafb-b9f976b10b48. 20/07/01 10:38:17.865 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:17.870 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.870 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.875 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-20287591-bad5-48ed-86f5-ab4a9e745a85 Pushed Filters: Post-Scan Filters: Output: timestamp_millis#275, timestamp_micros#276 20/07/01 10:38:17.879 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_54 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.888 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_54_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.889 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_54_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.890 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 54 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.892 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_55 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.901 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_55_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.902 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_55_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.903 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 55 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.907 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.907 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:17.914 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-20287591-bad5-48ed-86f5-ab4a9e745a85 Pushed Filters: Post-Scan Filters: Output: timestamp_millis#275, timestamp_micros#276 20/07/01 10:38:17.919 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_56 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.929 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_56_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.929 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_56_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.930 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 56 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.945 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_57 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.956 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_57_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:17.957 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_57_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:17.958 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 57 from apply at OutcomeOf.scala:85 20/07/01 10:38:17.970 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:18.229 dag-scheduler-event-loop INFO DAGScheduler: Got job 18 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:18.229 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 18 (apply at OutcomeOf.scala:85) 20/07/01 10:38:18.229 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:18.229 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:18.230 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 18 (MapPartitionsRDD[95] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:18.240 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_58 stored as values in memory (estimated size 14.4 KiB, free 2.1 GiB) 20/07/01 10:38:18.529 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_58_piece0 stored as bytes in memory (estimated size 7.1 KiB, free 2.1 GiB) 20/07/01 10:38:18.549 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_58_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.1 KiB, free: 2.1 GiB) 20/07/01 10:38:18.550 dag-scheduler-event-loop INFO SparkContext: Created broadcast 58 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:18.550 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[95] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:18.550 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 18.0 with 1 tasks resource profile 0 20/07/01 10:38:18.551 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 18, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:18.551 Executor task launch worker for task 18 INFO Executor: Running task 0.0 in stage 18.0 (TID 18) 20/07/01 10:38:18.558 Executor task launch worker for task 18 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-20287591-bad5-48ed-86f5-ab4a9e745a85/part-00000-2a6745d8-7006-4174-a335-24c7eeb6636d-c000.avro, range: 0-376, partition values: [empty row] 20/07/01 10:38:18.560 Executor task launch worker for task 18 INFO Executor: Finished task 0.0 in stage 18.0 (TID 18). 1299 bytes result sent to driver 20/07/01 10:38:18.561 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 18) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:18.561 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool 20/07/01 10:38:18.561 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 18 (apply at OutcomeOf.scala:85) finished in 0.331 s 20/07/01 10:38:18.561 dag-scheduler-event-loop INFO DAGScheduler: Job 18 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:18.561 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 18: Stage finished 20/07/01 10:38:18.562 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 18 finished: apply at OutcomeOf.scala:85, took 0.333390 s 20/07/01 10:38:18.580 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:18.580 dag-scheduler-event-loop INFO DAGScheduler: Got job 19 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:18.580 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 19 (apply at OutcomeOf.scala:85) 20/07/01 10:38:18.580 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:18.580 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:18.581 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[99] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:18.582 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59 stored as values in memory (estimated size 9.6 KiB, free 2.1 GiB) 20/07/01 10:38:18.583 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:18.583 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_59_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:18.584 dag-scheduler-event-loop INFO SparkContext: Created broadcast 59 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:18.584 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[99] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:18.584 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 19.0 with 1 tasks resource profile 0 20/07/01 10:38:18.585 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 19, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:18.585 Executor task launch worker for task 19 INFO Executor: Running task 0.0 in stage 19.0 (TID 19) 20/07/01 10:38:18.591 Executor task launch worker for task 19 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-20287591-bad5-48ed-86f5-ab4a9e745a85/part-00000-2a6745d8-7006-4174-a335-24c7eeb6636d-c000.avro, range: 0-376, partition values: [empty row] 20/07/01 10:38:18.593 Executor task launch worker for task 19 INFO Executor: Finished task 0.0 in stage 19.0 (TID 19). 1511 bytes result sent to driver 20/07/01 10:38:18.593 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 19) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:18.593 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 20/07/01 10:38:18.594 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 19 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:18.594 dag-scheduler-event-loop INFO DAGScheduler: Job 19 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:18.594 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 19: Stage finished 20/07/01 10:38:18.594 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 19 finished: apply at OutcomeOf.scala:85, took 0.014543 s 20/07/01 10:38:18.785 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: user specified output schema with different timestamp types' ===== 20/07/01 10:38:18.788 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Read Long type as Timestamp' ===== 20/07/01 10:38:18.793 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:18.802 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:18.803 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:18.809 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e2b2a3ef-ef39-4264-9682-d2aeb7c114fa/test.avro Pushed Filters: Post-Scan Filters: Output: long#291 20/07/01 10:38:18.814 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_60 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.823 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_60_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.824 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_60_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:18.825 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 60 from apply at OutcomeOf.scala:85 20/07/01 10:38:18.827 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_61 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.837 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_61_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.837 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_61_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:18.838 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 61 from apply at OutcomeOf.scala:85 20/07/01 10:38:18.842 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:18.902 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:18.912 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e2b2a3ef-ef39-4264-9682-d2aeb7c114fa/test.avro Pushed Filters: Post-Scan Filters: Output: long#291 20/07/01 10:38:18.917 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_62 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.926 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_62_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.926 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_62_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:18.927 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 62 from apply at OutcomeOf.scala:85 20/07/01 10:38:18.929 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_63 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.938 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_63_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:18.938 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_63_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:18.939 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 63 from apply at OutcomeOf.scala:85 20/07/01 10:38:18.951 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:18.952 dag-scheduler-event-loop INFO DAGScheduler: Got job 20 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:18.952 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 20 (apply at OutcomeOf.scala:85) 20/07/01 10:38:18.952 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:18.952 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:18.952 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[105] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:18.957 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_64 stored as values in memory (estimated size 13.8 KiB, free 2.1 GiB) 20/07/01 10:38:18.959 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_64_piece0 stored as bytes in memory (estimated size 6.9 KiB, free 2.1 GiB) 20/07/01 10:38:18.959 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_64_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:18.959 dag-scheduler-event-loop INFO SparkContext: Created broadcast 64 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:18.959 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[105] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:18.960 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 20.0 with 1 tasks resource profile 0 20/07/01 10:38:18.960 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 20, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:18.961 Executor task launch worker for task 20 INFO Executor: Running task 0.0 in stage 20.0 (TID 20) 20/07/01 10:38:19.046 Executor task launch worker for task 20 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e2b2a3ef-ef39-4264-9682-d2aeb7c114fa/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:19.048 Executor task launch worker for task 20 INFO Executor: Finished task 0.0 in stage 20.0 (TID 20). 1299 bytes result sent to driver 20/07/01 10:38:19.048 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 20) in 88 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.049 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 20/07/01 10:38:19.049 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 20 (apply at OutcomeOf.scala:85) finished in 0.096 s 20/07/01 10:38:19.049 dag-scheduler-event-loop INFO DAGScheduler: Job 20 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.049 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 20: Stage finished 20/07/01 10:38:19.049 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 20 finished: apply at OutcomeOf.scala:85, took 0.098286 s 20/07/01 10:38:19.066 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.067 dag-scheduler-event-loop INFO DAGScheduler: Got job 21 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.067 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 21 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.067 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.067 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.068 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[109] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.069 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65 stored as values in memory (estimated size 9.2 KiB, free 2.1 GiB) 20/07/01 10:38:19.071 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:19.071 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:19.072 dag-scheduler-event-loop INFO SparkContext: Created broadcast 65 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.072 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[109] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.072 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 21.0 with 1 tasks resource profile 0 20/07/01 10:38:19.073 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 21, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.073 Executor task launch worker for task 21 INFO Executor: Running task 0.0 in stage 21.0 (TID 21) 20/07/01 10:38:19.078 Executor task launch worker for task 21 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-e2b2a3ef-ef39-4264-9682-d2aeb7c114fa/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:19.079 Executor task launch worker for task 21 INFO Executor: Finished task 0.0 in stage 21.0 (TID 21). 1498 bytes result sent to driver 20/07/01 10:38:19.079 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 21) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.080 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 20/07/01 10:38:19.080 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 21 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:19.080 dag-scheduler-event-loop INFO DAGScheduler: Job 21 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.080 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 21: Stage finished 20/07/01 10:38:19.080 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 21 finished: apply at OutcomeOf.scala:85, took 0.014106 s 20/07/01 10:38:19.086 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Read Long type as Timestamp' ===== 20/07/01 10:38:19.087 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: user specified read schema' ===== 20/07/01 10:38:19.093 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:19.099 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.099 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.104 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-40e82ea6-9f89-4d25-a980-67366f8ad2aa/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#301, timestamp_micros#302, long#303L 20/07/01 10:38:19.107 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_66 stored as values in memory (estimated size 305.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.116 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_66_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.117 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_66_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.117 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 66 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.119 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_67 stored as values in memory (estimated size 305.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.128 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_67_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.128 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.129 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 67 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.133 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.133 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.140 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-40e82ea6-9f89-4d25-a980-67366f8ad2aa/test.avro Pushed Filters: Post-Scan Filters: Output: timestamp_millis#301, timestamp_micros#302, long#303L 20/07/01 10:38:19.144 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_68 stored as values in memory (estimated size 305.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.153 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_68_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.154 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.154 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 68 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.156 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_69 stored as values in memory (estimated size 305.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.165 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_69_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.166 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_69_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.166 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 69 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.178 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 9.503318 ms 20/07/01 10:38:19.189 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.189 dag-scheduler-event-loop INFO DAGScheduler: Got job 22 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.189 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 22 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.189 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.190 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.190 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[115] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.194 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_70 stored as values in memory (estimated size 15.2 KiB, free 2.1 GiB) 20/07/01 10:38:19.196 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_70_piece0 stored as bytes in memory (estimated size 7.4 KiB, free 2.1 GiB) 20/07/01 10:38:19.196 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_70_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.4 KiB, free: 2.1 GiB) 20/07/01 10:38:19.196 dag-scheduler-event-loop INFO SparkContext: Created broadcast 70 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.196 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 22 (MapPartitionsRDD[115] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.197 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 22.0 with 1 tasks resource profile 0 20/07/01 10:38:19.197 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 22, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.198 Executor task launch worker for task 22 INFO Executor: Running task 0.0 in stage 22.0 (TID 22) 20/07/01 10:38:19.207 Executor task launch worker for task 22 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-40e82ea6-9f89-4d25-a980-67366f8ad2aa/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:19.209 Executor task launch worker for task 22 INFO Executor: Finished task 0.0 in stage 22.0 (TID 22). 1299 bytes result sent to driver 20/07/01 10:38:19.209 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 22) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.209 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 20/07/01 10:38:19.210 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 22 (apply at OutcomeOf.scala:85) finished in 0.020 s 20/07/01 10:38:19.210 dag-scheduler-event-loop INFO DAGScheduler: Job 22 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.210 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 22: Stage finished 20/07/01 10:38:19.210 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 22 finished: apply at OutcomeOf.scala:85, took 0.021249 s 20/07/01 10:38:19.225 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.225 dag-scheduler-event-loop INFO DAGScheduler: Got job 23 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.226 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 23 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.226 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.226 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.226 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[119] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.228 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_71 stored as values in memory (estimated size 10.3 KiB, free 2.1 GiB) 20/07/01 10:38:19.229 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_71_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.229 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_71_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.230 dag-scheduler-event-loop INFO SparkContext: Created broadcast 71 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.230 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[119] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.230 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 23.0 with 1 tasks resource profile 0 20/07/01 10:38:19.231 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 23.0 (TID 23, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.231 Executor task launch worker for task 23 INFO Executor: Running task 0.0 in stage 23.0 (TID 23) 20/07/01 10:38:19.235 Executor task launch worker for task 23 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-40e82ea6-9f89-4d25-a980-67366f8ad2aa/test.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:19.237 Executor task launch worker for task 23 INFO Executor: Finished task 0.0 in stage 23.0 (TID 23). 1526 bytes result sent to driver 20/07/01 10:38:19.237 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 23.0 (TID 23) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.237 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 20/07/01 10:38:19.238 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 23 (apply at OutcomeOf.scala:85) finished in 0.010 s 20/07/01 10:38:19.238 dag-scheduler-event-loop INFO DAGScheduler: Job 23 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.238 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 23: Stage finished 20/07/01 10:38:19.238 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 23 finished: apply at OutcomeOf.scala:85, took 0.012942 s 20/07/01 10:38:19.244 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: user specified read schema' ===== 20/07/01 10:38:19.246 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: Decimal' ===== 20/07/01 10:38:19.253 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:19.260 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.260 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.265 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#324, fixed#325 20/07/01 10:38:19.269 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_72 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.278 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_72_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.278 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_72_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.279 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 72 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.281 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_73 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.291 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_73_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.291 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_73_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.292 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 73 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.296 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.296 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.303 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#324, fixed#325 20/07/01 10:38:19.308 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_74 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.316 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_74_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.317 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_74_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.318 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 74 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.320 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_75 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.329 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_75_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.329 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_75_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.329 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 75 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.341 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 8.585766 ms 20/07/01 10:38:19.350 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.351 dag-scheduler-event-loop INFO DAGScheduler: Got job 24 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.351 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 24 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.351 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.351 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.352 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.359 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_76 stored as values in memory (estimated size 14.5 KiB, free 2.1 GiB) 20/07/01 10:38:19.361 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_76_piece0 stored as bytes in memory (estimated size 7.2 KiB, free 2.1 GiB) 20/07/01 10:38:19.361 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_76_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.2 KiB, free: 2.1 GiB) 20/07/01 10:38:19.361 dag-scheduler-event-loop INFO SparkContext: Created broadcast 76 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.362 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.362 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 24.0 with 1 tasks resource profile 0 20/07/01 10:38:19.363 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 24, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.364 Executor task launch worker for task 24 INFO Executor: Running task 0.0 in stage 24.0 (TID 24) 20/07/01 10:38:19.376 Executor task launch worker for task 24 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:19.378 Executor task launch worker for task 24 INFO Executor: Finished task 0.0 in stage 24.0 (TID 24). 1299 bytes result sent to driver 20/07/01 10:38:19.378 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 24) in 15 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.378 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 20/07/01 10:38:19.379 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 24 (apply at OutcomeOf.scala:85) finished in 0.027 s 20/07/01 10:38:19.379 dag-scheduler-event-loop INFO DAGScheduler: Job 24 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.379 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 24: Stage finished 20/07/01 10:38:19.379 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 24 finished: apply at OutcomeOf.scala:85, took 0.028634 s 20/07/01 10:38:19.393 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.394 dag-scheduler-event-loop INFO DAGScheduler: Got job 25 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.394 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 25 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.394 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.394 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.394 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[129] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.395 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_77 stored as values in memory (estimated size 10.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.397 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_77_piece0 stored as bytes in memory (estimated size 4.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.397 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_77_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.9 KiB, free: 2.1 GiB) 20/07/01 10:38:19.398 dag-scheduler-event-loop INFO SparkContext: Created broadcast 77 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.398 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[129] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.398 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 25.0 with 1 tasks resource profile 0 20/07/01 10:38:19.399 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 25.0 (TID 25, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.399 Executor task launch worker for task 25 INFO Executor: Running task 0.0 in stage 25.0 (TID 25) 20/07/01 10:38:19.407 Executor task launch worker for task 25 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:19.409 Executor task launch worker for task 25 INFO Executor: Finished task 0.0 in stage 25.0 (TID 25). 1534 bytes result sent to driver 20/07/01 10:38:19.410 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 25.0 (TID 25) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.410 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 20/07/01 10:38:19.410 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 25 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:19.411 dag-scheduler-event-loop INFO DAGScheduler: Job 25 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.411 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 25: Stage finished 20/07/01 10:38:19.411 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 25 finished: apply at OutcomeOf.scala:85, took 0.017702 s 20/07/01 10:38:19.419 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:19.423 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.423 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.429 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#340, fixed#341 20/07/01 10:38:19.433 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_78 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.441 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_78_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.441 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_78_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:19.442 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 78 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.443 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_79 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.451 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_79_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.451 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_79_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:19.452 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 79 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.455 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.455 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.462 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#340, fixed#341 20/07/01 10:38:19.466 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_80 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.475 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_80_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.486 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_80_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:19.486 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 80 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.489 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_81 stored as values in memory (estimated size 306.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.497 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_81_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.497 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_81_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:19.497 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 81 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.510 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.511 dag-scheduler-event-loop INFO DAGScheduler: Got job 26 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.511 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 26 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.511 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.511 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.511 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 26 (MapPartitionsRDD[135] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.517 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_82 stored as values in memory (estimated size 15.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.518 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_82_piece0 stored as bytes in memory (estimated size 7.4 KiB, free 2.1 GiB) 20/07/01 10:38:19.519 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_82_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.4 KiB, free: 2.1 GiB) 20/07/01 10:38:19.519 dag-scheduler-event-loop INFO SparkContext: Created broadcast 82 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.519 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[135] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.519 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 26.0 with 1 tasks resource profile 0 20/07/01 10:38:19.520 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 26, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.521 Executor task launch worker for task 26 INFO Executor: Running task 0.0 in stage 26.0 (TID 26) 20/07/01 10:38:19.526 Executor task launch worker for task 26 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:19.528 Executor task launch worker for task 26 INFO Executor: Finished task 0.0 in stage 26.0 (TID 26). 1299 bytes result sent to driver 20/07/01 10:38:19.528 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 26.0 (TID 26) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.528 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 26.0, whose tasks have all completed, from pool 20/07/01 10:38:19.528 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 26 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:19.529 dag-scheduler-event-loop INFO DAGScheduler: Job 26 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.529 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 26: Stage finished 20/07/01 10:38:19.529 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 26 finished: apply at OutcomeOf.scala:85, took 0.018958 s 20/07/01 10:38:19.543 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.544 dag-scheduler-event-loop INFO DAGScheduler: Got job 27 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.544 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 27 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.544 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.544 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.544 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[139] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.546 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_83 stored as values in memory (estimated size 10.4 KiB, free 2.1 GiB) 20/07/01 10:38:19.548 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_83_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 2.1 GiB) 20/07/01 10:38:19.548 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_83_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 5.1 KiB, free: 2.1 GiB) 20/07/01 10:38:19.548 dag-scheduler-event-loop INFO SparkContext: Created broadcast 83 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.549 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[139] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.549 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 27.0 with 1 tasks resource profile 0 20/07/01 10:38:19.550 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 27, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.550 Executor task launch worker for task 27 INFO Executor: Running task 0.0 in stage 27.0 (TID 27) 20/07/01 10:38:19.554 Executor task launch worker for task 27 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:19.556 Executor task launch worker for task 27 INFO Executor: Finished task 0.0 in stage 27.0 (TID 27). 1534 bytes result sent to driver 20/07/01 10:38:19.556 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 27) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.556 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 20/07/01 10:38:19.557 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 27 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:19.558 dag-scheduler-event-loop INFO DAGScheduler: Job 27 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.558 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 27: Stage finished 20/07/01 10:38:19.558 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 27 finished: apply at OutcomeOf.scala:85, took 0.015171 s 20/07/01 10:38:19.567 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.567 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.575 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#324, fixed#325 20/07/01 10:38:19.579 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_84 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.588 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_84_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.589 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_84_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.589 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 84 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.592 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_85 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.601 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_85_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.601 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_85_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.601 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 85 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.605 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:19.606 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:19.606 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:19.606 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:19.630 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.631 dag-scheduler-event-loop INFO DAGScheduler: Got job 28 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.631 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 28 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.631 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.631 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.631 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[142] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.694 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_86 stored as values in memory (estimated size 175.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.696 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_86_piece0 stored as bytes in memory (estimated size 62.4 KiB, free 2.1 GiB) 20/07/01 10:38:19.697 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_86_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 62.4 KiB, free: 2.1 GiB) 20/07/01 10:38:19.697 dag-scheduler-event-loop INFO SparkContext: Created broadcast 86 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.698 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 28 (MapPartitionsRDD[142] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.698 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 28.0 with 1 tasks resource profile 0 20/07/01 10:38:19.699 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 28.0 (TID 28, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.700 Executor task launch worker for task 28 INFO Executor: Running task 0.0 in stage 28.0 (TID 28) 20/07/01 10:38:19.720 Executor task launch worker for task 28 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:19.720 Executor task launch worker for task 28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:19.720 Executor task launch worker for task 28 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:19.734 Executor task launch worker for task 28 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7fad6fcd-8b83-4e7c-bfbc-0ad6eb3d37f9/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:19.737 Executor task launch worker for task 28 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103819_0028_m_000000_28' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c1d19b27-aaf0-4122-8ebc-93dbb383e2f2 20/07/01 10:38:19.737 Executor task launch worker for task 28 INFO SparkHadoopMapRedUtil: attempt_20200701103819_0028_m_000000_28: Committed 20/07/01 10:38:19.738 Executor task launch worker for task 28 INFO Executor: Finished task 0.0 in stage 28.0 (TID 28). 2270 bytes result sent to driver 20/07/01 10:38:19.740 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 28.0 (TID 28) in 41 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.740 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 20/07/01 10:38:19.741 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 28 (apply at OutcomeOf.scala:85) finished in 0.108 s 20/07/01 10:38:19.741 dag-scheduler-event-loop INFO DAGScheduler: Job 28 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.741 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 28: Stage finished 20/07/01 10:38:19.742 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 28 finished: apply at OutcomeOf.scala:85, took 0.110908 s 20/07/01 10:38:19.747 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job 217384e6-7fbf-4d16-a63c-67aeb8f1b2d9 committed. 20/07/01 10:38:19.748 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 217384e6-7fbf-4d16-a63c-67aeb8f1b2d9. 20/07/01 10:38:19.753 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:19.758 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.758 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.762 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c1d19b27-aaf0-4122-8ebc-93dbb383e2f2 Pushed Filters: Post-Scan Filters: Output: bytes#362, fixed#363 20/07/01 10:38:19.765 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_87 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.773 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_87_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.774 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_87_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.774 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 87 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.776 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_88 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.784 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_88_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.784 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_88_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.784 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 88 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.789 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.789 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.797 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c1d19b27-aaf0-4122-8ebc-93dbb383e2f2 Pushed Filters: Post-Scan Filters: Output: bytes#362, fixed#363 20/07/01 10:38:19.801 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_89 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.809 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_89_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.809 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_89_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.810 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 89 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.812 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_90 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.820 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_90_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.820 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_90_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.821 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 90 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.832 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.833 dag-scheduler-event-loop INFO DAGScheduler: Got job 29 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.833 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 29 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.833 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.833 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.833 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[150] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.841 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_91 stored as values in memory (estimated size 14.5 KiB, free 2.1 GiB) 20/07/01 10:38:19.843 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_91_piece0 stored as bytes in memory (estimated size 7.3 KiB, free 2.1 GiB) 20/07/01 10:38:19.843 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_91_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.3 KiB, free: 2.1 GiB) 20/07/01 10:38:19.843 dag-scheduler-event-loop INFO SparkContext: Created broadcast 91 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.844 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[150] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.844 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 29.0 with 1 tasks resource profile 0 20/07/01 10:38:19.845 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 29.0 (TID 29, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.846 Executor task launch worker for task 29 INFO Executor: Running task 0.0 in stage 29.0 (TID 29) 20/07/01 10:38:19.856 Executor task launch worker for task 29 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c1d19b27-aaf0-4122-8ebc-93dbb383e2f2/part-00000-41905ddc-af92-4143-9140-2e1eb2de75f1-c000.avro, range: 0-503, partition values: [empty row] 20/07/01 10:38:19.858 Executor task launch worker for task 29 INFO Executor: Finished task 0.0 in stage 29.0 (TID 29). 1299 bytes result sent to driver 20/07/01 10:38:19.859 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 29.0 (TID 29) in 14 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.859 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 20/07/01 10:38:19.859 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 29 (apply at OutcomeOf.scala:85) finished in 0.025 s 20/07/01 10:38:19.859 dag-scheduler-event-loop INFO DAGScheduler: Job 29 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.859 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 29: Stage finished 20/07/01 10:38:19.860 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 29 finished: apply at OutcomeOf.scala:85, took 0.027212 s 20/07/01 10:38:19.880 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.881 dag-scheduler-event-loop INFO DAGScheduler: Got job 30 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.881 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 30 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.881 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.881 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.882 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[154] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:19.884 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_92 stored as values in memory (estimated size 10.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.885 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_92_piece0 stored as bytes in memory (estimated size 4.9 KiB, free 2.1 GiB) 20/07/01 10:38:19.885 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_92_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.9 KiB, free: 2.1 GiB) 20/07/01 10:38:19.886 dag-scheduler-event-loop INFO SparkContext: Created broadcast 92 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:19.886 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 30 (MapPartitionsRDD[154] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:19.886 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 30.0 with 1 tasks resource profile 0 20/07/01 10:38:19.887 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 30.0 (TID 30, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:19.888 Executor task launch worker for task 30 INFO Executor: Running task 0.0 in stage 30.0 (TID 30) 20/07/01 10:38:19.892 Executor task launch worker for task 30 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c1d19b27-aaf0-4122-8ebc-93dbb383e2f2/part-00000-41905ddc-af92-4143-9140-2e1eb2de75f1-c000.avro, range: 0-503, partition values: [empty row] 20/07/01 10:38:19.893 Executor task launch worker for task 30 INFO Executor: Finished task 0.0 in stage 30.0 (TID 30). 1534 bytes result sent to driver 20/07/01 10:38:19.893 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 30.0 (TID 30) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:19.893 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 30.0, whose tasks have all completed, from pool 20/07/01 10:38:19.894 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 30 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:19.894 dag-scheduler-event-loop INFO DAGScheduler: Job 30 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:19.894 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 30: Stage finished 20/07/01 10:38:19.894 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 30 finished: apply at OutcomeOf.scala:85, took 0.013595 s 20/07/01 10:38:19.902 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: Decimal' ===== 20/07/01 10:38:19.904 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: write Decimal with BYTES type' ===== 20/07/01 10:38:19.909 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:19.920 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.920 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:19.928 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-9347512d-9021-4c55-8415-e7469eaddfd4/test.avro Pushed Filters: Post-Scan Filters: Output: bytes#378, fixed#379 20/07/01 10:38:19.932 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_93 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.941 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_93_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.941 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_93_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.942 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 93 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.944 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_94 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.953 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_94_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:19.953 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_94_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:19.954 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 94 from apply at OutcomeOf.scala:85 20/07/01 10:38:19.958 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:19.959 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:19.959 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:19.959 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:19.986 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:19.987 dag-scheduler-event-loop INFO DAGScheduler: Got job 31 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:19.987 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 31 (apply at OutcomeOf.scala:85) 20/07/01 10:38:19.987 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:19.987 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:19.987 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[157] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:20.051 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_95 stored as values in memory (estimated size 176.2 KiB, free 2.1 GiB) 20/07/01 10:38:20.053 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_95_piece0 stored as bytes in memory (estimated size 62.7 KiB, free 2.1 GiB) 20/07/01 10:38:20.053 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_95_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 62.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.054 dag-scheduler-event-loop INFO SparkContext: Created broadcast 95 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.054 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 31 (MapPartitionsRDD[157] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:20.054 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 31.0 with 1 tasks resource profile 0 20/07/01 10:38:20.055 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 31.0 (TID 31, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7987 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.055 Executor task launch worker for task 31 INFO Executor: Running task 0.0 in stage 31.0 (TID 31) 20/07/01 10:38:20.067 Executor task launch worker for task 31 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:20.067 Executor task launch worker for task 31 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:20.067 Executor task launch worker for task 31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:20.086 Executor task launch worker for task 31 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-9347512d-9021-4c55-8415-e7469eaddfd4/test.avro, range: 0-356, partition values: [empty row] 20/07/01 10:38:20.090 Executor task launch worker for task 31 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103819_0031_m_000000_31' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4a6726e1-f478-45bb-ab8a-2d2d1e798e77 20/07/01 10:38:20.090 Executor task launch worker for task 31 INFO SparkHadoopMapRedUtil: attempt_20200701103819_0031_m_000000_31: Committed 20/07/01 10:38:20.091 Executor task launch worker for task 31 INFO Executor: Finished task 0.0 in stage 31.0 (TID 31). 2270 bytes result sent to driver 20/07/01 10:38:20.092 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 31.0 (TID 31) in 36 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:20.092 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 20/07/01 10:38:20.092 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 31 (apply at OutcomeOf.scala:85) finished in 0.104 s 20/07/01 10:38:20.092 dag-scheduler-event-loop INFO DAGScheduler: Job 31 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:20.092 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 31: Stage finished 20/07/01 10:38:20.093 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 31 finished: apply at OutcomeOf.scala:85, took 0.106734 s 20/07/01 10:38:20.104 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Write Job 05b3f9a3-3062-4856-96b1-81406e9f153b committed. 20/07/01 10:38:20.104 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO FileFormatWriter: Finished processing stats for write job 05b3f9a3-3062-4856-96b1-81406e9f153b. 20/07/01 10:38:20.112 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:20.119 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.120 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.124 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4a6726e1-f478-45bb-ab8a-2d2d1e798e77 Pushed Filters: Post-Scan Filters: Output: bytes#388, fixed#389 20/07/01 10:38:20.129 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_96 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.138 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_96_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.138 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_96_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.139 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 96 from apply at OutcomeOf.scala:85 20/07/01 10:38:20.141 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_97 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.149 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_97_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.149 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_97_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.150 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 97 from apply at OutcomeOf.scala:85 20/07/01 10:38:20.153 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.153 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.161 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4a6726e1-f478-45bb-ab8a-2d2d1e798e77 Pushed Filters: Post-Scan Filters: Output: bytes#388, fixed#389 20/07/01 10:38:20.166 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_98 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.174 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_98_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.175 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_98_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.175 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 98 from apply at OutcomeOf.scala:85 20/07/01 10:38:20.178 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_99 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.186 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_99_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.186 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_99_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.187 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 99 from apply at OutcomeOf.scala:85 20/07/01 10:38:20.237 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:20.238 dag-scheduler-event-loop INFO DAGScheduler: Got job 32 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:20.238 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 32 (apply at OutcomeOf.scala:85) 20/07/01 10:38:20.238 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:20.238 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:20.238 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 32 (MapPartitionsRDD[165] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:20.242 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_100 stored as values in memory (estimated size 14.5 KiB, free 2.1 GiB) 20/07/01 10:38:20.244 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_100_piece0 stored as bytes in memory (estimated size 7.3 KiB, free 2.1 GiB) 20/07/01 10:38:20.244 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_100_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 7.3 KiB, free: 2.1 GiB) 20/07/01 10:38:20.244 dag-scheduler-event-loop INFO SparkContext: Created broadcast 100 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.245 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 32 (MapPartitionsRDD[165] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:20.245 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 32.0 with 1 tasks resource profile 0 20/07/01 10:38:20.245 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 32, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.246 Executor task launch worker for task 32 INFO Executor: Running task 0.0 in stage 32.0 (TID 32) 20/07/01 10:38:20.252 Executor task launch worker for task 32 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4a6726e1-f478-45bb-ab8a-2d2d1e798e77/part-00000-655ac055-6b4f-4949-8504-aa329ea2d0aa-c000.avro, range: 0-417, partition values: [empty row] 20/07/01 10:38:20.253 Executor task launch worker for task 32 INFO Executor: Finished task 0.0 in stage 32.0 (TID 32). 1299 bytes result sent to driver 20/07/01 10:38:20.253 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 32) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:20.253 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool 20/07/01 10:38:20.254 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 32 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:20.254 dag-scheduler-event-loop INFO DAGScheduler: Job 32 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:20.254 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 32: Stage finished 20/07/01 10:38:20.254 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 32 finished: apply at OutcomeOf.scala:85, took 0.016795 s 20/07/01 10:38:20.269 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:20.269 dag-scheduler-event-loop INFO DAGScheduler: Got job 33 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:20.269 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 33 (apply at OutcomeOf.scala:85) 20/07/01 10:38:20.269 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:20.270 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:20.270 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 33 (MapPartitionsRDD[169] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:20.271 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_101 stored as values in memory (estimated size 10.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.272 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_101_piece0 stored as bytes in memory (estimated size 4.9 KiB, free 2.1 GiB) 20/07/01 10:38:20.273 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_101_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.9 KiB, free: 2.1 GiB) 20/07/01 10:38:20.273 dag-scheduler-event-loop INFO SparkContext: Created broadcast 101 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.273 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 33 (MapPartitionsRDD[169] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:20.273 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 33.0 with 1 tasks resource profile 0 20/07/01 10:38:20.274 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 33.0 (TID 33, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8035 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.275 Executor task launch worker for task 33 INFO Executor: Running task 0.0 in stage 33.0 (TID 33) 20/07/01 10:38:20.280 Executor task launch worker for task 33 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-4a6726e1-f478-45bb-ab8a-2d2d1e798e77/part-00000-655ac055-6b4f-4949-8504-aa329ea2d0aa-c000.avro, range: 0-417, partition values: [empty row] 20/07/01 10:38:20.282 Executor task launch worker for task 33 INFO Executor: Finished task 0.0 in stage 33.0 (TID 33). 1534 bytes result sent to driver 20/07/01 10:38:20.282 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 33.0 (TID 33) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:20.282 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 33.0, whose tasks have all completed, from pool 20/07/01 10:38:20.282 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 33 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:20.283 dag-scheduler-event-loop INFO DAGScheduler: Job 33 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:20.283 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 33: Stage finished 20/07/01 10:38:20.283 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 33 finished: apply at OutcomeOf.scala:85, took 0.013894 s 20/07/01 10:38:20.292 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: write Decimal with BYTES type' ===== 20/07/01 10:38:20.294 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: Decimal with too large precision' ===== 20/07/01 10:38:20.295 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_72_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.297 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_62_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.298 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_74_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.299 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:20.299 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_95_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 62.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.300 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_49_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.301 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_55_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.302 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_52_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.303 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_43_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.304 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.304 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.304 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_78_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.306 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_70_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.4 KiB, free: 2.1 GiB) 20/07/01 10:38:20.307 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_92_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.9 KiB, free: 2.1 GiB) 20/07/01 10:38:20.308 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_47_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.309 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_80_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.310 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_89_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.311 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_91_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.3 KiB, free: 2.1 GiB) 20/07/01 10:38:20.311 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_93_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.312 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO V2ScanRelationPushDown: Pushing operators to avro file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6a3fff26-e0a7-4ae7-99d9-357f8b8171d0.avro Pushed Filters: Post-Scan Filters: Output: decimal#404 20/07/01 10:38:20.312 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_53_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 62.3 KiB, free: 2.1 GiB) 20/07/01 10:38:20.313 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_71_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.313 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_81_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.314 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_38_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.315 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_68_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.316 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_76_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.2 KiB, free: 2.1 GiB) 20/07/01 10:38:20.316 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_102 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.316 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_67_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.317 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_36_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.318 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_86_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 62.4 KiB, free: 2.1 GiB) 20/07/01 10:38:20.318 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_57_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.319 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_59_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.320 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_58_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.320 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_88_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.321 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_84_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.322 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_37_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.322 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_94_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.323 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_46_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.324 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_75_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.324 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_48_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.325 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_102_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.325 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_85_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.326 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_102_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.326 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_61_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.326 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 102 from $anonfun$new$47 at Assertions.scala:807 20/07/01 10:38:20.327 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_73_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.327 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_87_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.328 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_83_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 5.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.328 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_103 stored as values in memory (estimated size 305.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.329 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_50_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.331 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_64_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 6.9 KiB, free: 2.1 GiB) 20/07/01 10:38:20.332 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_39_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.333 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_56_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.334 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_54_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.335 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_45_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.336 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_42_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.336 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_82_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 7.4 KiB, free: 2.1 GiB) 20/07/01 10:38:20.338 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_44_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 62.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.339 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_79_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:20.339 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_90_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.340 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_65_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.340 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO MemoryStore: Block broadcast_103_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.340 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_103_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.341 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_51_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.341 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Created broadcast 103 from $anonfun$new$47 at Assertions.scala:807 20/07/01 10:38:20.342 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_60_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.342 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_63_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.343 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_77_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 4.9 KiB, free: 2.1 GiB) 20/07/01 10:38:20.344 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_69_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.344 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_66_piece0 on amp-jenkins-worker-04.amp:35137 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.356 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO CodeGenerator: Code generated in 8.377625 ms 20/07/01 10:38:20.365 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO SparkContext: Starting job: $anonfun$new$47 at Assertions.scala:807 20/07/01 10:38:20.366 dag-scheduler-event-loop INFO DAGScheduler: Got job 34 ($anonfun$new$47 at Assertions.scala:807) with 1 output partitions 20/07/01 10:38:20.366 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 34 ($anonfun$new$47 at Assertions.scala:807) 20/07/01 10:38:20.366 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:20.366 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:20.366 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 34 (MapPartitionsRDD[173] at $anonfun$new$47 at Assertions.scala:807), which has no missing parents 20/07/01 10:38:20.367 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_104 stored as values in memory (estimated size 9.6 KiB, free 2.1 GiB) 20/07/01 10:38:20.369 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_104_piece0 stored as bytes in memory (estimated size 4.8 KiB, free 2.1 GiB) 20/07/01 10:38:20.370 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_104_piece0 in memory on amp-jenkins-worker-04.amp:35137 (size: 4.8 KiB, free: 2.1 GiB) 20/07/01 10:38:20.370 dag-scheduler-event-loop INFO SparkContext: Created broadcast 104 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.371 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 34 (MapPartitionsRDD[173] at $anonfun$new$47 at Assertions.scala:807) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:20.371 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 34.0 with 1 tasks resource profile 0 20/07/01 10:38:20.372 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7982 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.373 Executor task launch worker for task 34 INFO Executor: Running task 0.0 in stage 34.0 (TID 34) 20/07/01 10:38:20.382 Executor task launch worker for task 34 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6a3fff26-e0a7-4ae7-99d9-357f8b8171d0.avro, range: 0-226, partition values: [empty row] 20/07/01 10:38:20.383 Executor task launch worker for task 34 ERROR Executor: Exception in task 0.0 in stage 34.0 (TID 34) java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:118) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:97) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.$anonfun$rowConverter$1(PartitionReaderWithPartitionValues.scala:41) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:50) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:29) at org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader.get(FilePartitionReaderFactory.scala:56) at org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.get(FilePartitionReader.scala:96) at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:89) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:124) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:121) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:753) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:20.385 task-result-getter-2 WARN TaskSetManager: Lost task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver): java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:118) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:97) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.$anonfun$rowConverter$1(PartitionReaderWithPartitionValues.scala:41) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:50) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:29) at org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader.get(FilePartitionReaderFactory.scala:56) at org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.get(FilePartitionReader.scala:96) at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:89) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:124) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:121) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:753) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:20.385 task-result-getter-2 ERROR TaskSetManager: Task 0 in stage 34.0 failed 1 times; aborting job 20/07/01 10:38:20.385 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool 20/07/01 10:38:20.386 dag-scheduler-event-loop INFO TaskSchedulerImpl: Cancelling stage 34 20/07/01 10:38:20.387 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 34: Stage cancelled 20/07/01 10:38:20.387 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 34 ($anonfun$new$47 at Assertions.scala:807) failed in 0.021 s due to Job aborted due to stage failure: Task 0 in stage 34.0 failed 1 times, most recent failure: Lost task 0.0 in stage 34.0 (TID 34, amp-jenkins-worker-04.amp, executor driver): java.lang.ArithmeticException: Unscaled value too large for precision at org.apache.spark.sql.types.Decimal.set(Decimal.scala:83) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:577) at org.apache.spark.sql.avro.AvroDeserializer.createDecimal(AvroDeserializer.scala:308) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16(AvroDeserializer.scala:177) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$16$adapted(AvroDeserializer.scala:174) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:336) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:332) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:354) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:351) at org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$3(AvroDeserializer.scala:75) at org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:89) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:118) at org.apache.spark.sql.v2.avro.AvroPartitionReaderFactory$$anon$1.get(AvroPartitionReaderFactory.scala:97) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.$anonfun$rowConverter$1(PartitionReaderWithPartitionValues.scala:41) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:50) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.get(PartitionReaderWithPartitionValues.scala:29) at org.apache.spark.sql.execution.datasources.v2.PartitionedFileReader.get(FilePartitionReaderFactory.scala:56) at org.apache.spark.sql.execution.datasources.v2.FilePartitionReader.get(FilePartitionReader.scala:96) at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:89) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:124) at org.apache.spark.sql.execution.datasources.v2.MetricsRowIterator.next(DataSourceRDD.scala:121) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:753) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: 20/07/01 10:38:20.387 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO DAGScheduler: Job 34 failed: $anonfun$new$47 at Assertions.scala:807, took 0.021990 s 20/07/01 10:38:20.390 pool-1-thread-1-ScalaTest-running-AvroV2LogicalTypeSuite INFO AvroV2LogicalTypeSuite: ===== FINISHED o.a.s.sql.avro.AvroV2LogicalTypeSuite: 'Logical type: Decimal with too large precision' ===== 20/07/01 10:38:20.430 dispatcher-event-loop-1 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/07/01 10:38:20.436 pool-1-thread-1 INFO MemoryStore: MemoryStore cleared 20/07/01 10:38:20.436 pool-1-thread-1 INFO BlockManager: BlockManager stopped 20/07/01 10:38:20.437 pool-1-thread-1 INFO BlockManagerMaster: BlockManagerMaster stopped 20/07/01 10:38:20.437 dispatcher-event-loop-1 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/07/01 10:38:20.447 pool-1-thread-1 INFO SparkContext: Successfully stopped SparkContext 20/07/01 10:38:20.448 pool-1-thread-1 WARN AvroV2LogicalTypeSuite: ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.avro.AvroV2LogicalTypeSuite, thread names: rpc-boss-9-1, shuffle-boss-12-1 ===== 20/07/01 10:38:20.479 pool-1-thread-1 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT 20/07/01 10:38:20.479 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:20.479 pool-1-thread-1 INFO ResourceUtils: No custom resources configured for spark.driver. 20/07/01 10:38:20.479 pool-1-thread-1 INFO ResourceUtils: ============================================================== 20/07/01 10:38:20.479 pool-1-thread-1 INFO SparkContext: Submitted application: test-sql-context 20/07/01 10:38:20.480 pool-1-thread-1 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 20/07/01 10:38:20.480 pool-1-thread-1 INFO ResourceProfile: Limiting resource is cpu 20/07/01 10:38:20.480 pool-1-thread-1 INFO ResourceProfileManager: Added ResourceProfile id: 0 20/07/01 10:38:20.481 pool-1-thread-1 INFO SecurityManager: Changing view acls to: jenkins 20/07/01 10:38:20.481 pool-1-thread-1 INFO SecurityManager: Changing modify acls to: jenkins 20/07/01 10:38:20.481 pool-1-thread-1 INFO SecurityManager: Changing view acls groups to: 20/07/01 10:38:20.481 pool-1-thread-1 INFO SecurityManager: Changing modify acls groups to: 20/07/01 10:38:20.481 pool-1-thread-1 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set() 20/07/01 10:38:20.499 pool-1-thread-1 INFO Utils: Successfully started service 'sparkDriver' on port 43345. 20/07/01 10:38:20.500 pool-1-thread-1 INFO SparkEnv: Registering MapOutputTracker 20/07/01 10:38:20.501 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMaster 20/07/01 10:38:20.501 pool-1-thread-1 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/07/01 10:38:20.501 pool-1-thread-1 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/07/01 10:38:20.502 pool-1-thread-1 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/07/01 10:38:20.503 pool-1-thread-1 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/blockmgr-0d8957f2-8828-4839-b3c6-bccae3a24c7a 20/07/01 10:38:20.503 pool-1-thread-1 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB 20/07/01 10:38:20.504 pool-1-thread-1 INFO SparkEnv: Registering OutputCommitCoordinator 20/07/01 10:38:20.523 pool-1-thread-1 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-04.amp 20/07/01 10:38:20.526 pool-1-thread-1 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35025. 20/07/01 10:38:20.526 pool-1-thread-1 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-04.amp:35025 20/07/01 10:38:20.526 pool-1-thread-1 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/07/01 10:38:20.526 pool-1-thread-1 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 35025, None) 20/07/01 10:38:20.527 dispatcher-BlockManagerMaster INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-04.amp:35025 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-04.amp, 35025, None) 20/07/01 10:38:20.550 pool-1-thread-1 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-04.amp, 35025, None) 20/07/01 10:38:20.550 pool-1-thread-1 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-04.amp, 35025, None) 20/07/01 10:38:20.559 pool-1-thread-1 INFO SharedState: loading hive config file: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/sql/core/target/scala-2.12/test-classes/hive-site.xml 20/07/01 10:38:20.567 pool-1-thread-1 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV1Suite'). 20/07/01 10:38:20.567 pool-1-thread-1 INFO SharedState: Warehouse path is 'file:/home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/spark-warehouse/org.apache.spark.sql.avro.AvroV1Suite'. 20/07/01 10:38:20.570 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'resolve avro data source' ===== 20/07/01 10:38:20.583 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'resolve avro data source' ===== 20/07/01 10:38:20.584 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'reading from multiple paths' ===== 20/07/01 10:38:20.591 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 2 ms to list leaf files for 2 paths. 20/07/01 10:38:20.616 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:20.669 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:20.669 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:20.669 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:20.669 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:20.717 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 10.554664 ms 20/07/01 10:38:20.732 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 10.096551 ms 20/07/01 10:38:20.733 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:20.742 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.742 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.743 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 0 from $anonfun$new$5 at OutcomeOf.scala:85 20/07/01 10:38:20.744 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:20.769 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$5 at OutcomeOf.scala:85 20/07/01 10:38:20.772 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 3 ($anonfun$new$5 at OutcomeOf.scala:85) as input to shuffle 0 20/07/01 10:38:20.774 dag-scheduler-event-loop INFO DAGScheduler: Got job 0 ($anonfun$new$5 at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:20.774 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 1 ($anonfun$new$5 at OutcomeOf.scala:85) 20/07/01 10:38:20.774 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 20/07/01 10:38:20.775 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) 20/07/01 10:38:20.776 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at $anonfun$new$5 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:20.786 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:20.788 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:20.788 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:20.789 dag-scheduler-event-loop INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.791 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at $anonfun$new$5 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:20.791 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0 20/07/01 10:38:20.794 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7802 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.795 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7802 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.796 Executor task launch worker for task 0 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 20/07/01 10:38:20.796 Executor task launch worker for task 1 INFO Executor: Running task 1.0 in stage 0.0 (TID 1) 20/07/01 10:38:20.823 Executor task launch worker for task 1 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:20.823 Executor task launch worker for task 0 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:20.832 Executor task launch worker for task 1 INFO CodeGenerator: Code generated in 7.658267 ms 20/07/01 10:38:20.861 Executor task launch worker for task 1 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1978 bytes result sent to driver 20/07/01 10:38:20.861 Executor task launch worker for task 0 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1978 bytes result sent to driver 20/07/01 10:38:20.863 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 69 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:20.863 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 71 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:20.863 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 20/07/01 10:38:20.866 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 0 ($anonfun$new$5 at OutcomeOf.scala:85) finished in 0.086 s 20/07/01 10:38:20.866 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:20.866 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:20.867 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 1) 20/07/01 10:38:20.867 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:20.871 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at $anonfun$new$5 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:20.878 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:20.880 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:20.880 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:20.881 dag-scheduler-event-loop INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:20.881 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at $anonfun$new$5 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:20.881 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0 20/07/01 10:38:20.885 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:20.886 Executor task launch worker for task 2 INFO Executor: Running task 0.0 in stage 1.0 (TID 2) 20/07/01 10:38:20.914 Executor task launch worker for task 2 INFO ShuffleBlockFetcherIterator: Getting 2 (120.0 B) non-empty blocks including 2 (120.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:20.915 Executor task launch worker for task 2 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms 20/07/01 10:38:20.932 Executor task launch worker for task 2 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 2644 bytes result sent to driver 20/07/01 10:38:20.934 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 50 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:20.934 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 20/07/01 10:38:20.934 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 1 ($anonfun$new$5 at OutcomeOf.scala:85) finished in 0.057 s 20/07/01 10:38:20.934 dag-scheduler-event-loop INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:20.934 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished 20/07/01 10:38:20.934 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 0 finished: $anonfun$new$5 at OutcomeOf.scala:85, took 0.164981 s 20/07/01 10:38:20.935 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'reading from multiple paths' ===== 20/07/01 10:38:20.937 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'reading and writing partitioned data' ===== 20/07/01 10:38:20.942 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:21.080 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:21.080 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:21.092 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:21.092 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:21.092 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:21.093 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:21.099 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:21.101 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:21.101 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:21.101 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:21.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:21.451 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:21.452 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:21.452 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 3 from apply at OutcomeOf.scala:85 20/07/01 10:38:21.453 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:21.472 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:21.473 dag-scheduler-event-loop INFO DAGScheduler: Got job 1 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:21.473 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 2 (apply at OutcomeOf.scala:85) 20/07/01 10:38:21.473 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:21.474 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:21.474 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[10] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:21.545 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 177.5 KiB, free 2.1 GiB) 20/07/01 10:38:21.547 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 63.9 KiB, free 2.1 GiB) 20/07/01 10:38:21.548 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 63.9 KiB, free: 2.1 GiB) 20/07/01 10:38:21.548 dag-scheduler-event-loop INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:21.548 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[10] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:21.548 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks resource profile 0 20/07/01 10:38:21.549 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.550 Executor task launch worker for task 3 INFO Executor: Running task 0.0 in stage 2.0 (TID 3) 20/07/01 10:38:21.592 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 13.264367 ms 20/07/01 10:38:21.610 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 10.539581 ms 20/07/01 10:38:21.629 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 11.170135 ms 20/07/01 10:38:21.640 Executor task launch worker for task 3 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:21.649 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 7.822288 ms 20/07/01 10:38:21.682 Executor task launch worker for task 3 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:21.682 Executor task launch worker for task 3 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:21.682 Executor task launch worker for task 3 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:21.691 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 6.905254 ms 20/07/01 10:38:21.698 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 5.222994 ms 20/07/01 10:38:21.732 Executor task launch worker for task 3 INFO CodeGenerator: Code generated in 13.84513 ms 20/07/01 10:38:21.866 Executor task launch worker for task 3 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103821_0002_m_000000_3' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c 20/07/01 10:38:21.866 Executor task launch worker for task 3 INFO SparkHadoopMapRedUtil: attempt_20200701103821_0002_m_000000_3: Committed 20/07/01 10:38:21.873 Executor task launch worker for task 3 INFO Executor: Finished task 0.0 in stage 2.0 (TID 3). 3098 bytes result sent to driver 20/07/01 10:38:21.874 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 3) in 325 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:21.874 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 20/07/01 10:38:21.875 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 2 (apply at OutcomeOf.scala:85) finished in 0.400 s 20/07/01 10:38:21.875 dag-scheduler-event-loop INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:21.875 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished 20/07/01 10:38:21.875 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 1 finished: apply at OutcomeOf.scala:85, took 0.402689 s 20/07/01 10:38:21.884 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 8ad6d5b2-d118-47fb-8c1c-f9610fb99f5e committed. 20/07/01 10:38:21.885 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 8ad6d5b2-d118-47fb-8c1c-f9610fb99f5e. 20/07/01 10:38:21.896 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 5 ms to list leaf files for 1 paths. 20/07/01 10:38:21.933 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:21.933 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:21.942 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:21.943 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:21.943 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:21.943 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:21.948 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:21.957 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:21.957 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:21.958 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 5 from apply at OutcomeOf.scala:85 20/07/01 10:38:21.960 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:21.969 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:21.970 dag-scheduler-event-loop INFO DAGScheduler: Got job 2 (apply at OutcomeOf.scala:85) with 8 output partitions 20/07/01 10:38:21.970 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 3 (apply at OutcomeOf.scala:85) 20/07/01 10:38:21.970 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:21.970 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:21.970 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:21.971 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:21.973 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:21.973 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:21.973 dag-scheduler-event-loop INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:21.974 dag-scheduler-event-loop INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 3 (MapPartitionsRDD[15] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7)) 20/07/01 10:38:21.974 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 3.0 with 8 tasks resource profile 0 20/07/01 10:38:21.975 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 4, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8014 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.975 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 5, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8018 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.976 Executor task launch worker for task 5 INFO Executor: Running task 1.0 in stage 3.0 (TID 5) 20/07/01 10:38:21.976 Executor task launch worker for task 4 INFO Executor: Running task 0.0 in stage 3.0 (TID 4) 20/07/01 10:38:21.978 Executor task launch worker for task 5 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=Horror%20of%20Fang%20Rock/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-266, partition values: [Horror of Fang Rock] 20/07/01 10:38:21.978 Executor task launch worker for task 4 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=An%20Unearthly%20Child/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-266, partition values: [An Unearthly Child] 20/07/01 10:38:21.984 Executor task launch worker for task 5 INFO Executor: Finished task 1.0 in stage 3.0 (TID 5). 1442 bytes result sent to driver 20/07/01 10:38:21.984 Executor task launch worker for task 4 INFO Executor: Finished task 0.0 in stage 3.0 (TID 4). 1442 bytes result sent to driver 20/07/01 10:38:21.984 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 6, amp-jenkins-worker-04.amp, executor driver, partition 2, PROCESS_LOCAL, 8020 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.985 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 5) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/8) 20/07/01 10:38:21.985 Executor task launch worker for task 6 INFO Executor: Running task 2.0 in stage 3.0 (TID 6) 20/07/01 10:38:21.985 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 7, amp-jenkins-worker-04.amp, executor driver, partition 3, PROCESS_LOCAL, 8028 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.985 Executor task launch worker for task 7 INFO Executor: Running task 3.0 in stage 3.0 (TID 7) 20/07/01 10:38:21.985 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 4) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (2/8) 20/07/01 10:38:21.987 Executor task launch worker for task 6 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=The%20Mysterious%20Planet/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-266, partition values: [The Mysterious Planet] 20/07/01 10:38:21.987 Executor task launch worker for task 7 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=The%20Power%20of%20the%20Daleks/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-265, partition values: [The Power of the Daleks] 20/07/01 10:38:21.989 Executor task launch worker for task 7 INFO Executor: Finished task 3.0 in stage 3.0 (TID 7). 1442 bytes result sent to driver 20/07/01 10:38:21.990 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 8, amp-jenkins-worker-04.amp, executor driver, partition 4, PROCESS_LOCAL, 7994 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.990 Executor task launch worker for task 6 INFO Executor: Finished task 2.0 in stage 3.0 (TID 6). 1442 bytes result sent to driver 20/07/01 10:38:21.990 Executor task launch worker for task 8 INFO Executor: Running task 4.0 in stage 3.0 (TID 8) 20/07/01 10:38:21.990 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 5.0 in stage 3.0 (TID 9, amp-jenkins-worker-04.amp, executor driver, partition 5, PROCESS_LOCAL, 7982 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.990 task-result-getter-2 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 7) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (3/8) 20/07/01 10:38:21.991 Executor task launch worker for task 9 INFO Executor: Running task 5.0 in stage 3.0 (TID 9) 20/07/01 10:38:21.991 task-result-getter-3 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 6) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (4/8) 20/07/01 10:38:21.992 Executor task launch worker for task 9 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=Rose/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-263, partition values: [Rose] 20/07/01 10:38:21.992 Executor task launch worker for task 8 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=Castrolava/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-264, partition values: [Castrolava] 20/07/01 10:38:21.994 Executor task launch worker for task 9 INFO Executor: Finished task 5.0 in stage 3.0 (TID 9). 1383 bytes result sent to driver 20/07/01 10:38:21.994 Executor task launch worker for task 8 INFO Executor: Finished task 4.0 in stage 3.0 (TID 8). 1434 bytes result sent to driver 20/07/01 10:38:21.995 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 6.0 in stage 3.0 (TID 10, amp-jenkins-worker-04.amp, executor driver, partition 6, PROCESS_LOCAL, 8012 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.995 task-result-getter-0 INFO TaskSetManager: Finished task 5.0 in stage 3.0 (TID 9) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (5/8) 20/07/01 10:38:21.995 Executor task launch worker for task 10 INFO Executor: Running task 6.0 in stage 3.0 (TID 10) 20/07/01 10:38:21.995 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 7.0 in stage 3.0 (TID 11, amp-jenkins-worker-04.amp, executor driver, partition 7, PROCESS_LOCAL, 8016 bytes) taskResourceAssignments Map() 20/07/01 10:38:21.996 Executor task launch worker for task 11 INFO Executor: Running task 7.0 in stage 3.0 (TID 11) 20/07/01 10:38:21.996 task-result-getter-1 INFO TaskSetManager: Finished task 4.0 in stage 3.0 (TID 8) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (6/8) 20/07/01 10:38:21.997 Executor task launch worker for task 11 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=The%20Doctor%2527s%20Wife/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-261, partition values: [The Doctor's Wife] 20/07/01 10:38:21.997 Executor task launch worker for task 10 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-2f2009b3-0a90-4065-9ff4-b6bcf28fb50d/624e6189-dc58-4a18-be83-f635d40e657c/title=The%20Eleventh%20Hour/part-00000-70656aa0-02f0-4b39-a53c-d864410f8f7c.c000.avro, range: 0-262, partition values: [The Eleventh Hour] 20/07/01 10:38:21.999 Executor task launch worker for task 11 INFO Executor: Finished task 7.0 in stage 3.0 (TID 11). 1399 bytes result sent to driver 20/07/01 10:38:21.999 Executor task launch worker for task 10 INFO Executor: Finished task 6.0 in stage 3.0 (TID 10). 1442 bytes result sent to driver 20/07/01 10:38:22.000 task-result-getter-2 INFO TaskSetManager: Finished task 7.0 in stage 3.0 (TID 11) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (7/8) 20/07/01 10:38:22.000 task-result-getter-3 INFO TaskSetManager: Finished task 6.0 in stage 3.0 (TID 10) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (8/8) 20/07/01 10:38:22.000 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 20/07/01 10:38:22.000 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 3 (apply at OutcomeOf.scala:85) finished in 0.029 s 20/07/01 10:38:22.000 dag-scheduler-event-loop INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.000 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished 20/07/01 10:38:22.001 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 2 finished: apply at OutcomeOf.scala:85, took 0.031629 s 20/07/01 10:38:22.013 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 8.007684 ms 20/07/01 10:38:22.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.032 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.033 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.033 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.033 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:22.040 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.050 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.051 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 7 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.052 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.059 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.059 dag-scheduler-event-loop INFO DAGScheduler: Got job 3 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:22.059 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 4 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.059 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.059 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.060 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[18] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.061 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:22.062 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.063 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.063 dag-scheduler-event-loop INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.063 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[18] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:22.063 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks resource profile 0 20/07/01 10:38:22.064 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 12, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.065 Executor task launch worker for task 12 INFO Executor: Running task 0.0 in stage 4.0 (TID 12) 20/07/01 10:38:22.067 Executor task launch worker for task 12 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:22.070 Executor task launch worker for task 12 INFO Executor: Finished task 0.0 in stage 4.0 (TID 12). 1631 bytes result sent to driver 20/07/01 10:38:22.071 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 12) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:22.071 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 20/07/01 10:38:22.071 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 4 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:22.071 dag-scheduler-event-loop INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.071 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 4: Stage finished 20/07/01 10:38:22.071 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 3 finished: apply at OutcomeOf.scala:85, took 0.012629 s 20/07/01 10:38:22.087 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.087 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.099 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.100 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.100 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.100 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:22.103 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:22.104 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:22.104 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:22.104 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:22.113 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.122 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.125 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.126 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 9 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.127 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.148 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.149 dag-scheduler-event-loop INFO DAGScheduler: Got job 4 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:22.149 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 5 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.149 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.149 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.149 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[22] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.214 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 177.5 KiB, free 2.1 GiB) 20/07/01 10:38:22.216 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 63.9 KiB, free 2.1 GiB) 20/07/01 10:38:22.217 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 63.9 KiB, free: 2.1 GiB) 20/07/01 10:38:22.217 dag-scheduler-event-loop INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.217 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[22] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:22.217 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks resource profile 0 20/07/01 10:38:22.218 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 13, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.219 Executor task launch worker for task 13 INFO Executor: Running task 0.0 in stage 5.0 (TID 13) 20/07/01 10:38:22.242 Executor task launch worker for task 13 INFO CodeGenerator: Code generated in 9.085887 ms 20/07/01 10:38:22.251 Executor task launch worker for task 13 INFO CodeGenerator: Code generated in 6.496404 ms 20/07/01 10:38:22.262 Executor task launch worker for task 13 INFO CodeGenerator: Code generated in 9.757027 ms 20/07/01 10:38:22.264 Executor task launch worker for task 13 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:22.277 Executor task launch worker for task 13 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:22.277 Executor task launch worker for task 13 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:22.277 Executor task launch worker for task 13 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:22.286 Executor task launch worker for task 13 INFO CodeGenerator: Code generated in 7.092572 ms 20/07/01 10:38:22.296 Executor task launch worker for task 13 INFO CodeGenerator: Code generated in 7.845851 ms 20/07/01 10:38:22.422 Executor task launch worker for task 13 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103822_0005_m_000000_13' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6 20/07/01 10:38:22.422 Executor task launch worker for task 13 INFO SparkHadoopMapRedUtil: attempt_20200701103822_0005_m_000000_13: Committed 20/07/01 10:38:22.423 Executor task launch worker for task 13 INFO Executor: Finished task 0.0 in stage 5.0 (TID 13). 3061 bytes result sent to driver 20/07/01 10:38:22.425 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 13) in 207 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:22.425 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 20/07/01 10:38:22.425 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 5 (apply at OutcomeOf.scala:85) finished in 0.275 s 20/07/01 10:38:22.426 dag-scheduler-event-loop INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.426 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished 20/07/01 10:38:22.426 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 4 finished: apply at OutcomeOf.scala:85, took 0.277744 s 20/07/01 10:38:22.433 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 255760f5-4c58-4ce1-9123-27a1efe26544 committed. 20/07/01 10:38:22.434 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 255760f5-4c58-4ce1-9123-27a1efe26544. 20/07/01 10:38:22.444 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 4 ms to list leaf files for 1 paths. 20/07/01 10:38:22.456 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.456 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.467 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.467 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.467 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.467 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:22.472 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.483 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.483 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.484 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 11 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.484 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.491 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.492 dag-scheduler-event-loop INFO DAGScheduler: Got job 5 (apply at OutcomeOf.scala:85) with 8 output partitions 20/07/01 10:38:22.492 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 6 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.492 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.492 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.493 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[27] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.495 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:22.496 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:22.497 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:22.497 dag-scheduler-event-loop INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.497 dag-scheduler-event-loop INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 6 (MapPartitionsRDD[27] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7)) 20/07/01 10:38:22.497 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 6.0 with 8 tasks resource profile 0 20/07/01 10:38:22.498 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 14, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8011 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.499 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 6.0 (TID 15, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8013 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.499 Executor task launch worker for task 15 INFO Executor: Running task 1.0 in stage 6.0 (TID 15) 20/07/01 10:38:22.499 Executor task launch worker for task 14 INFO Executor: Running task 0.0 in stage 6.0 (TID 14) 20/07/01 10:38:22.501 Executor task launch worker for task 15 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=6%20September%201986/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-268, partition values: [6 September 1986] 20/07/01 10:38:22.502 Executor task launch worker for task 14 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=5%20November%201966/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-270, partition values: [5 November 1966] 20/07/01 10:38:22.505 Executor task launch worker for task 15 INFO Executor: Finished task 1.0 in stage 6.0 (TID 15). 1430 bytes result sent to driver 20/07/01 10:38:22.506 Executor task launch worker for task 14 INFO Executor: Finished task 0.0 in stage 6.0 (TID 14). 1434 bytes result sent to driver 20/07/01 10:38:22.506 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 2.0 in stage 6.0 (TID 16, amp-jenkins-worker-04.amp, executor driver, partition 2, PROCESS_LOCAL, 8013 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.506 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 6.0 (TID 15) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/8) 20/07/01 10:38:22.506 Executor task launch worker for task 16 INFO Executor: Running task 2.0 in stage 6.0 (TID 16) 20/07/01 10:38:22.506 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 3.0 in stage 6.0 (TID 17, amp-jenkins-worker-04.amp, executor driver, partition 3, PROCESS_LOCAL, 8013 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.506 Executor task launch worker for task 17 INFO Executor: Running task 3.0 in stage 6.0 (TID 17) 20/07/01 10:38:22.507 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 14) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (2/8) 20/07/01 10:38:22.508 Executor task launch worker for task 17 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=23%20November%201963/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-265, partition values: [23 November 1963] 20/07/01 10:38:22.509 Executor task launch worker for task 16 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=3%20September%201977/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-266, partition values: [3 September 1977] 20/07/01 10:38:22.511 Executor task launch worker for task 17 INFO Executor: Finished task 3.0 in stage 6.0 (TID 17). 1430 bytes result sent to driver 20/07/01 10:38:22.511 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 4.0 in stage 6.0 (TID 18, amp-jenkins-worker-04.amp, executor driver, partition 4, PROCESS_LOCAL, 8003 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.511 task-result-getter-0 INFO TaskSetManager: Finished task 3.0 in stage 6.0 (TID 17) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (3/8) 20/07/01 10:38:22.511 Executor task launch worker for task 18 INFO Executor: Running task 4.0 in stage 6.0 (TID 18) 20/07/01 10:38:22.512 Executor task launch worker for task 16 INFO Executor: Finished task 2.0 in stage 6.0 (TID 16). 1430 bytes result sent to driver 20/07/01 10:38:22.513 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 5.0 in stage 6.0 (TID 19, amp-jenkins-worker-04.amp, executor driver, partition 5, PROCESS_LOCAL, 8005 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.513 task-result-getter-1 INFO TaskSetManager: Finished task 2.0 in stage 6.0 (TID 16) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (4/8) 20/07/01 10:38:22.513 Executor task launch worker for task 19 INFO Executor: Running task 5.0 in stage 6.0 (TID 19) 20/07/01 10:38:22.514 Executor task launch worker for task 18 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=14%20May%202011/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-264, partition values: [14 May 2011] 20/07/01 10:38:22.515 Executor task launch worker for task 19 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=3%20April%202010/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-264, partition values: [3 April 2010] 20/07/01 10:38:22.516 Executor task launch worker for task 18 INFO Executor: Finished task 4.0 in stage 6.0 (TID 18). 1434 bytes result sent to driver 20/07/01 10:38:22.516 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 6.0 in stage 6.0 (TID 20, amp-jenkins-worker-04.amp, executor driver, partition 6, PROCESS_LOCAL, 8009 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.517 Executor task launch worker for task 20 INFO Executor: Running task 6.0 in stage 6.0 (TID 20) 20/07/01 10:38:22.517 task-result-getter-2 INFO TaskSetManager: Finished task 4.0 in stage 6.0 (TID 18) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (5/8) 20/07/01 10:38:22.518 Executor task launch worker for task 20 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=4%20January%201982/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-257, partition values: [4 January 1982] 20/07/01 10:38:22.518 Executor task launch worker for task 19 INFO Executor: Finished task 5.0 in stage 6.0 (TID 19). 1434 bytes result sent to driver 20/07/01 10:38:22.519 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 7.0 in stage 6.0 (TID 21, amp-jenkins-worker-04.amp, executor driver, partition 7, PROCESS_LOCAL, 8007 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.519 task-result-getter-3 INFO TaskSetManager: Finished task 5.0 in stage 6.0 (TID 19) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (6/8) 20/07/01 10:38:22.519 Executor task launch worker for task 21 INFO Executor: Running task 7.0 in stage 6.0 (TID 21) 20/07/01 10:38:22.520 Executor task launch worker for task 20 INFO Executor: Finished task 6.0 in stage 6.0 (TID 20). 1434 bytes result sent to driver 20/07/01 10:38:22.520 task-result-getter-0 INFO TaskSetManager: Finished task 6.0 in stage 6.0 (TID 20) in 4 ms on amp-jenkins-worker-04.amp (executor driver) (7/8) 20/07/01 10:38:22.521 Executor task launch worker for task 21 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6609303b-dd95-408a-b566-72ad2bdbf9e1/ac88d9a0-c8ab-4bfc-a903-10f159bb34c6/air_date=26%20March%202005/part-00000-b622a62a-e610-4c21-a81a-846d4f3c9f01.c000.avro, range: 0-251, partition values: [26 March 2005] 20/07/01 10:38:22.523 Executor task launch worker for task 21 INFO Executor: Finished task 7.0 in stage 6.0 (TID 21). 1434 bytes result sent to driver 20/07/01 10:38:22.523 task-result-getter-1 INFO TaskSetManager: Finished task 7.0 in stage 6.0 (TID 21) in 4 ms on amp-jenkins-worker-04.amp (executor driver) (8/8) 20/07/01 10:38:22.524 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 20/07/01 10:38:22.524 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 6 (apply at OutcomeOf.scala:85) finished in 0.030 s 20/07/01 10:38:22.524 dag-scheduler-event-loop INFO DAGScheduler: Job 5 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.524 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 6: Stage finished 20/07/01 10:38:22.524 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 5 finished: apply at OutcomeOf.scala:85, took 0.032731 s 20/07/01 10:38:22.533 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.533 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.543 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.543 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.543 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.543 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:22.548 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.558 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.558 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.559 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 13 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.560 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.567 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.567 dag-scheduler-event-loop INFO DAGScheduler: Got job 6 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:22.567 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 7 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.567 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.567 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.568 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[30] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.569 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:22.570 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.570 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.571 dag-scheduler-event-loop INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.571 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[30] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:22.571 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks resource profile 0 20/07/01 10:38:22.572 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 22, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.572 Executor task launch worker for task 22 INFO Executor: Running task 0.0 in stage 7.0 (TID 22) 20/07/01 10:38:22.574 Executor task launch worker for task 22 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:22.577 Executor task launch worker for task 22 INFO Executor: Finished task 0.0 in stage 7.0 (TID 22). 1565 bytes result sent to driver 20/07/01 10:38:22.578 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 22) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:22.578 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 20/07/01 10:38:22.578 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 7 (apply at OutcomeOf.scala:85) finished in 0.010 s 20/07/01 10:38:22.578 dag-scheduler-event-loop INFO DAGScheduler: Job 6 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.579 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 7: Stage finished 20/07/01 10:38:22.579 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 6 finished: apply at OutcomeOf.scala:85, took 0.012103 s 20/07/01 10:38:22.593 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.593 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.604 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.604 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.604 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.605 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:22.608 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:22.608 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:22.608 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:22.609 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:22.619 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.628 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.629 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.629 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 15 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.630 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.649 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.650 dag-scheduler-event-loop INFO DAGScheduler: Got job 7 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:22.650 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 8 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.650 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.650 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.650 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[33] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.723 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 176.3 KiB, free 2.1 GiB) 20/07/01 10:38:22.726 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 63.3 KiB, free 2.1 GiB) 20/07/01 10:38:22.726 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 63.3 KiB, free: 2.1 GiB) 20/07/01 10:38:22.726 dag-scheduler-event-loop INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.727 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[33] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:22.727 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks resource profile 0 20/07/01 10:38:22.728 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 23, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.728 Executor task launch worker for task 23 INFO Executor: Running task 0.0 in stage 8.0 (TID 23) 20/07/01 10:38:22.748 Executor task launch worker for task 23 INFO CodeGenerator: Code generated in 7.591489 ms 20/07/01 10:38:22.762 Executor task launch worker for task 23 INFO CodeGenerator: Code generated in 11.770527 ms 20/07/01 10:38:22.764 Executor task launch worker for task 23 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:22.786 Executor task launch worker for task 23 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:22.787 Executor task launch worker for task 23 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:22.787 Executor task launch worker for task 23 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:22.795 Executor task launch worker for task 23 INFO CodeGenerator: Code generated in 7.365091 ms 20/07/01 10:38:22.803 Executor task launch worker for task 23 INFO CodeGenerator: Code generated in 6.096893 ms 20/07/01 10:38:22.822 Executor task launch worker for task 23 INFO CodeGenerator: Code generated in 14.190739 ms 20/07/01 10:38:22.919 Executor task launch worker for task 23 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103822_0008_m_000000_23' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3 20/07/01 10:38:22.919 Executor task launch worker for task 23 INFO SparkHadoopMapRedUtil: attempt_20200701103822_0008_m_000000_23: Committed 20/07/01 10:38:22.920 Executor task launch worker for task 23 INFO Executor: Finished task 0.0 in stage 8.0 (TID 23). 2930 bytes result sent to driver 20/07/01 10:38:22.920 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 23) in 192 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:22.921 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 20/07/01 10:38:22.921 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 8 (apply at OutcomeOf.scala:85) finished in 0.270 s 20/07/01 10:38:22.921 dag-scheduler-event-loop INFO DAGScheduler: Job 7 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:22.921 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 8: Stage finished 20/07/01 10:38:22.921 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 7 finished: apply at OutcomeOf.scala:85, took 0.272242 s 20/07/01 10:38:22.926 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job b16ac4d3-2d7f-4d12-9c34-ab54e3086aa8 committed. 20/07/01 10:38:22.927 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job b16ac4d3-2d7f-4d12-9c34-ab54e3086aa8. 20/07/01 10:38:22.936 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 4 ms to list leaf files for 1 paths. 20/07/01 10:38:22.946 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.946 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:22.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:22.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:22.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:22.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:22.961 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:22.971 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:22.978 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:22.979 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 17 from apply at OutcomeOf.scala:85 20/07/01 10:38:22.980 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:22.986 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:22.987 dag-scheduler-event-loop INFO DAGScheduler: Got job 8 (apply at OutcomeOf.scala:85) with 7 output partitions 20/07/01 10:38:22.987 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 9 (apply at OutcomeOf.scala:85) 20/07/01 10:38:22.987 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:22.987 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:22.987 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[38] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:22.988 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:22.990 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:22.990 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:22.990 dag-scheduler-event-loop INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:22.991 dag-scheduler-event-loop INFO DAGScheduler: Submitting 7 missing tasks from ResultStage 9 (MapPartitionsRDD[38] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6)) 20/07/01 10:38:22.991 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 9.0 with 7 tasks resource profile 0 20/07/01 10:38:22.991 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 24, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7923 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.992 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 25, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.992 Executor task launch worker for task 25 INFO Executor: Running task 1.0 in stage 9.0 (TID 25) 20/07/01 10:38:22.992 Executor task launch worker for task 24 INFO Executor: Running task 0.0 in stage 9.0 (TID 24) 20/07/01 10:38:22.995 Executor task launch worker for task 25 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=2/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-290, partition values: [2] 20/07/01 10:38:22.995 Executor task launch worker for task 24 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=11/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-313, partition values: [11] 20/07/01 10:38:22.998 Executor task launch worker for task 25 INFO Executor: Finished task 1.0 in stage 9.0 (TID 25). 1417 bytes result sent to driver 20/07/01 10:38:22.998 Executor task launch worker for task 24 INFO Executor: Finished task 0.0 in stage 9.0 (TID 24). 1418 bytes result sent to driver 20/07/01 10:38:22.999 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 26, amp-jenkins-worker-04.amp, executor driver, partition 2, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:22.999 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 9.0 (TID 25) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/7) 20/07/01 10:38:22.999 Executor task launch worker for task 26 INFO Executor: Running task 2.0 in stage 9.0 (TID 26) 20/07/01 10:38:23.000 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 27, amp-jenkins-worker-04.amp, executor driver, partition 3, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.000 Executor task launch worker for task 27 INFO Executor: Running task 3.0 in stage 9.0 (TID 27) 20/07/01 10:38:23.000 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 24) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (2/7) 20/07/01 10:38:23.001 Executor task launch worker for task 26 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=6/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-289, partition values: [6] 20/07/01 10:38:23.001 Executor task launch worker for task 27 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=4/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-287, partition values: [4] 20/07/01 10:38:23.004 Executor task launch worker for task 26 INFO Executor: Finished task 2.0 in stage 9.0 (TID 26). 1417 bytes result sent to driver 20/07/01 10:38:23.004 Executor task launch worker for task 27 INFO Executor: Finished task 3.0 in stage 9.0 (TID 27). 1374 bytes result sent to driver 20/07/01 10:38:23.005 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 28, amp-jenkins-worker-04.amp, executor driver, partition 4, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.006 task-result-getter-2 INFO TaskSetManager: Finished task 2.0 in stage 9.0 (TID 26) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (3/7) 20/07/01 10:38:23.006 Executor task launch worker for task 28 INFO Executor: Running task 4.0 in stage 9.0 (TID 28) 20/07/01 10:38:23.006 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 29, amp-jenkins-worker-04.amp, executor driver, partition 5, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.006 task-result-getter-3 INFO TaskSetManager: Finished task 3.0 in stage 9.0 (TID 27) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (4/7) 20/07/01 10:38:23.006 Executor task launch worker for task 29 INFO Executor: Running task 5.0 in stage 9.0 (TID 29) 20/07/01 10:38:23.008 Executor task launch worker for task 28 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=1/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-286, partition values: [1] 20/07/01 10:38:23.008 Executor task launch worker for task 29 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=5/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-276, partition values: [5] 20/07/01 10:38:23.011 Executor task launch worker for task 29 INFO Executor: Finished task 5.0 in stage 9.0 (TID 29). 1417 bytes result sent to driver 20/07/01 10:38:23.011 Executor task launch worker for task 28 INFO Executor: Finished task 4.0 in stage 9.0 (TID 28). 1417 bytes result sent to driver 20/07/01 10:38:23.012 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 30, amp-jenkins-worker-04.amp, executor driver, partition 6, PROCESS_LOCAL, 7922 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.012 Executor task launch worker for task 30 INFO Executor: Running task 6.0 in stage 9.0 (TID 30) 20/07/01 10:38:23.012 task-result-getter-0 INFO TaskSetManager: Finished task 5.0 in stage 9.0 (TID 29) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (5/7) 20/07/01 10:38:23.012 task-result-getter-1 INFO TaskSetManager: Finished task 4.0 in stage 9.0 (TID 28) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (6/7) 20/07/01 10:38:23.014 Executor task launch worker for task 30 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-ea4f0c85-d865-4b77-84ce-7a59a9fe2044/72bdfcf3-91a4-41d3-9af5-9a164e2189e3/doctor=9/part-00000-2627bd6f-9907-42da-82d6-75aec4a8fe62.c000.avro, range: 0-269, partition values: [9] 20/07/01 10:38:23.017 Executor task launch worker for task 30 INFO Executor: Finished task 6.0 in stage 9.0 (TID 30). 1417 bytes result sent to driver 20/07/01 10:38:23.018 task-result-getter-2 INFO TaskSetManager: Finished task 6.0 in stage 9.0 (TID 30) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (7/7) 20/07/01 10:38:23.018 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 20/07/01 10:38:23.018 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 9 (apply at OutcomeOf.scala:85) finished in 0.030 s 20/07/01 10:38:23.018 dag-scheduler-event-loop INFO DAGScheduler: Job 8 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:23.018 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 9: Stage finished 20/07/01 10:38:23.019 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 8 finished: apply at OutcomeOf.scala:85, took 0.032159 s 20/07/01 10:38:23.028 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.978486 ms 20/07/01 10:38:23.035 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.035 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.045 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:23.046 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:23.046 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:23.046 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:23.051 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:23.060 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:23.061 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:23.061 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 19 from apply at OutcomeOf.scala:85 20/07/01 10:38:23.062 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:23.068 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:23.069 dag-scheduler-event-loop INFO DAGScheduler: Got job 9 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:23.069 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 10 (apply at OutcomeOf.scala:85) 20/07/01 10:38:23.069 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:23.069 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:23.070 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[41] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:23.071 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:23.073 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:23.073 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:23.073 dag-scheduler-event-loop INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:23.074 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[41] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:23.074 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 10.0 with 1 tasks resource profile 0 20/07/01 10:38:23.075 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 31, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.075 Executor task launch worker for task 31 INFO Executor: Running task 0.0 in stage 10.0 (TID 31) 20/07/01 10:38:23.077 Executor task launch worker for task 31 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:23.080 Executor task launch worker for task 31 INFO Executor: Finished task 0.0 in stage 10.0 (TID 31). 1475 bytes result sent to driver 20/07/01 10:38:23.081 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 31) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:23.081 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 20/07/01 10:38:23.082 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 10 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:23.082 dag-scheduler-event-loop INFO DAGScheduler: Job 9 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:23.082 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 10: Stage finished 20/07/01 10:38:23.082 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 9 finished: apply at OutcomeOf.scala:85, took 0.013412 s 20/07/01 10:38:23.087 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'reading and writing partitioned data' ===== 20/07/01 10:38:23.088 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'request no fields' ===== 20/07/01 10:38:23.092 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:23.107 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.107 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.179 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.180 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.197 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:23.197 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:23.197 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:23.198 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:23.209 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:23.217 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:23.218 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:23.218 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 21 from $anonfun$new$11 at OutcomeOf.scala:85 20/07/01 10:38:23.219 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:23.229 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$11 at OutcomeOf.scala:85 20/07/01 10:38:23.230 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 45 ($anonfun$new$11 at OutcomeOf.scala:85) as input to shuffle 1 20/07/01 10:38:23.230 dag-scheduler-event-loop INFO DAGScheduler: Got job 10 ($anonfun$new$11 at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:23.230 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 12 ($anonfun$new$11 at OutcomeOf.scala:85) 20/07/01 10:38:23.231 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 11) 20/07/01 10:38:23.231 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 11) 20/07/01 10:38:23.231 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 11 (MapPartitionsRDD[45] at $anonfun$new$11 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:23.232 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:23.234 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:23.234 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:23.234 dag-scheduler-event-loop INFO SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:23.234 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 11 (MapPartitionsRDD[45] at $anonfun$new$11 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:23.235 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 11.0 with 1 tasks resource profile 0 20/07/01 10:38:23.235 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 32, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7802 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.236 Executor task launch worker for task 32 INFO Executor: Running task 0.0 in stage 11.0 (TID 32) 20/07/01 10:38:23.238 Executor task launch worker for task 32 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:23.242 Executor task launch worker for task 32 INFO Executor: Finished task 0.0 in stage 11.0 (TID 32). 1935 bytes result sent to driver 20/07/01 10:38:23.242 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 32) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:23.243 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 11 ($anonfun$new$11 at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 12) 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:23.243 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[48] at $anonfun$new$11 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:23.245 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:23.246 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:23.247 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:23.247 dag-scheduler-event-loop INFO SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:23.247 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[48] at $anonfun$new$11 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:23.247 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 12.0 with 1 tasks resource profile 0 20/07/01 10:38:23.248 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 33, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.249 Executor task launch worker for task 33 INFO Executor: Running task 0.0 in stage 12.0 (TID 33) 20/07/01 10:38:23.251 Executor task launch worker for task 33 INFO ShuffleBlockFetcherIterator: Getting 1 (60.0 B) non-empty blocks including 1 (60.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:23.251 Executor task launch worker for task 33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:23.253 Executor task launch worker for task 33 INFO Executor: Finished task 0.0 in stage 12.0 (TID 33). 2648 bytes result sent to driver 20/07/01 10:38:23.253 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 33) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:23.253 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 20/07/01 10:38:23.254 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 12 ($anonfun$new$11 at OutcomeOf.scala:85) finished in 0.010 s 20/07/01 10:38:23.254 dag-scheduler-event-loop INFO DAGScheduler: Job 10 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:23.254 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 12: Stage finished 20/07/01 10:38:23.254 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 10 finished: $anonfun$new$11 at OutcomeOf.scala:85, took 0.024513 s 20/07/01 10:38:23.261 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.055862 ms 20/07/01 10:38:23.263 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'request no fields' ===== 20/07/01 10:38:23.264 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'convert formats' ===== 20/07/01 10:38:23.270 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:23.301 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.301 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:23.311 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:23.312 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:23.312 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:23.312 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:23.327 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter 20/07/01 10:38:23.350 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:23.350 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:23.351 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter 20/07/01 10:38:23.351 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:23.351 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:23.351 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter 20/07/01 10:38:23.360 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:23.368 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:23.369 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_24_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:23.369 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 24 from apply at OutcomeOf.scala:85 20/07/01 10:38:23.370 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:23.388 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:23.389 dag-scheduler-event-loop INFO DAGScheduler: Got job 11 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:23.389 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 13 (apply at OutcomeOf.scala:85) 20/07/01 10:38:23.389 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:23.389 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:23.389 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[50] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:23.463 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 172.9 KiB, free 2.1 GiB) 20/07/01 10:38:23.466 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 61.6 KiB, free 2.1 GiB) 20/07/01 10:38:23.466 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_25_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 61.6 KiB, free: 2.1 GiB) 20/07/01 10:38:23.466 dag-scheduler-event-loop INFO SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:23.467 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[50] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:23.467 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 13.0 with 1 tasks resource profile 0 20/07/01 10:38:23.468 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 13.0 (TID 34, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:23.468 Executor task launch worker for task 34 INFO Executor: Running task 0.0 in stage 13.0 (TID 34) 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:23.483 Executor task launch worker for task 34 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter 20/07/01 10:38:23.486 Executor task launch worker for task 34 INFO CodecConfig: Compression: SNAPPY 20/07/01 10:38:23.486 Executor task launch worker for task 34 INFO CodecConfig: Compression: SNAPPY 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Parquet block size to 134217728 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Parquet page size to 1048576 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Dictionary is on 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Validation is off 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Page size checking is: estimated 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Min row count for page size check is: 100 20/07/01 10:38:23.502 Executor task launch worker for task 34 INFO ParquetOutputFormat: Max row count for page size check is: 10000 20/07/01 10:38:23.545 Executor task launch worker for task 34 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema: { "type" : "struct", "fields" : [ { "name" : "title", "type" : "string", "nullable" : true, "metadata" : { } }, { "name" : "air_date", "type" : "string", "nullable" : true, "metadata" : { } }, { "name" : "doctor", "type" : "integer", "nullable" : true, "metadata" : { } } ] } and corresponding Parquet message type: message spark_schema { optional binary title (UTF8); optional binary air_date (UTF8); optional int32 doctor; } 20/07/01 10:38:23.600 Executor task launch worker for task 34 INFO CodecPool: Got brand-new compressor [.snappy] 20/07/01 10:38:23.878 Executor task launch worker for task 34 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:23.891 Executor task launch worker for task 34 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 430 20/07/01 10:38:24.033 Executor task launch worker for task 34 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103823_0013_m_000000_34' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6899cdf-2262-45a4-8c61-fc327a4a1058 20/07/01 10:38:24.052 Executor task launch worker for task 34 INFO SparkHadoopMapRedUtil: attempt_20200701103823_0013_m_000000_34: Committed 20/07/01 10:38:24.053 Executor task launch worker for task 34 INFO Executor: Finished task 0.0 in stage 13.0 (TID 34). 2198 bytes result sent to driver 20/07/01 10:38:24.054 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 34) in 587 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.054 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 20/07/01 10:38:24.055 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 13 (apply at OutcomeOf.scala:85) finished in 0.665 s 20/07/01 10:38:24.055 dag-scheduler-event-loop INFO DAGScheduler: Job 11 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.055 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 13: Stage finished 20/07/01 10:38:24.055 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 11 finished: apply at OutcomeOf.scala:85, took 0.667007 s 20/07/01 10:38:24.062 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job ab44646c-1e4c-4e9b-8e51-fa21a8feec99 committed. 20/07/01 10:38:24.062 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job ab44646c-1e4c-4e9b-8e51-fa21a8feec99. 20/07/01 10:38:24.074 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:24.122 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.123 dag-scheduler-event-loop INFO DAGScheduler: Got job 12 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.123 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 14 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.123 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:24.123 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:24.124 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 14 (MapPartitionsRDD[54] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.159 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 84.9 KiB, free 2.1 GiB) 20/07/01 10:38:24.161 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 30.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.162 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 30.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.162 dag-scheduler-event-loop INFO SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.163 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 14 (MapPartitionsRDD[54] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.163 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 14.0 with 1 tasks resource profile 0 20/07/01 10:38:24.166 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 35, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7625 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.166 Executor task launch worker for task 35 INFO Executor: Running task 0.0 in stage 14.0 (TID 35) 20/07/01 10:38:24.211 Executor task launch worker for task 35 INFO Executor: Finished task 0.0 in stage 14.0 (TID 35). 1735 bytes result sent to driver 20/07/01 10:38:24.212 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 35) in 49 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.212 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 20/07/01 10:38:24.212 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 14 (apply at OutcomeOf.scala:85) finished in 0.087 s 20/07/01 10:38:24.212 dag-scheduler-event-loop INFO DAGScheduler: Job 12 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.212 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 14: Stage finished 20/07/01 10:38:24.213 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 12 finished: apply at OutcomeOf.scala:85, took 0.090027 s 20/07/01 10:38:24.221 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.221 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.260 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO V2ScanRelationPushDown: Pushing operators to parquet file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6899cdf-2262-45a4-8c61-fc327a4a1058 Pushed Filters: Post-Scan Filters: Output: 20/07/01 10:38:24.268 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 311.6 KiB, free 2.1 GiB) 20/07/01 10:38:24.281 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 28.3 KiB, free 2.1 GiB) 20/07/01 10:38:24.281 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_27_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:24.282 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 27 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.291 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 311.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.301 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 28.3 KiB, free 2.1 GiB) 20/07/01 10:38:24.301 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:24.302 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 28 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.330 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 14.224789 ms 20/07/01 10:38:24.347 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.347 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 58 (apply at OutcomeOf.scala:85) as input to shuffle 2 20/07/01 10:38:24.348 dag-scheduler-event-loop INFO DAGScheduler: Got job 13 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.348 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 16 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.348 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 15) 20/07/01 10:38:24.348 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 15) 20/07/01 10:38:24.348 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[58] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.355 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 14.4 KiB, free 2.1 GiB) 20/07/01 10:38:24.358 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.358 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.358 dag-scheduler-event-loop INFO SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.359 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[58] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:24.359 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 15.0 with 2 tasks resource profile 0 20/07/01 10:38:24.360 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 36, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 8034 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.360 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 15.0 (TID 37, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8034 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.360 Executor task launch worker for task 36 INFO Executor: Running task 0.0 in stage 15.0 (TID 36) 20/07/01 10:38:24.360 Executor task launch worker for task 37 INFO Executor: Running task 1.0 in stage 15.0 (TID 37) 20/07/01 10:38:24.392 Executor task launch worker for task 36 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6899cdf-2262-45a4-8c61-fc327a4a1058/part-00000-aff731e4-88e5-4a84-86de-206eefc7f1cf-c000.snappy.parquet, range: 0-1024, partition values: [empty row] 20/07/01 10:38:24.392 Executor task launch worker for task 37 INFO FilePartitionReader: Reading file path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b6899cdf-2262-45a4-8c61-fc327a4a1058/part-00000-aff731e4-88e5-4a84-86de-206eefc7f1cf-c000.snappy.parquet, range: 1024-1216, partition values: [empty row] 20/07/01 10:38:24.396 Executor task launch worker for task 37 INFO Executor: Finished task 1.0 in stage 15.0 (TID 37). 2080 bytes result sent to driver 20/07/01 10:38:24.397 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 15.0 (TID 37) in 37 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:24.399 Executor task launch worker for task 36 INFO Executor: Finished task 0.0 in stage 15.0 (TID 36). 2080 bytes result sent to driver 20/07/01 10:38:24.399 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 36) in 39 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:24.399 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 15 (apply at OutcomeOf.scala:85) finished in 0.051 s 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 16) 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:24.400 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[61] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.402 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:24.443 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.444 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.445 dag-scheduler-event-loop INFO SparkContext: Created broadcast 30 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.445 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[61] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.445 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 16.0 with 1 tasks resource profile 0 20/07/01 10:38:24.446 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 38, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.447 Executor task launch worker for task 38 INFO Executor: Running task 0.0 in stage 16.0 (TID 38) 20/07/01 10:38:24.449 Executor task launch worker for task 38 INFO ShuffleBlockFetcherIterator: Getting 2 (120.0 B) non-empty blocks including 2 (120.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:24.449 Executor task launch worker for task 38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:24.451 Executor task launch worker for task 38 INFO Executor: Finished task 0.0 in stage 16.0 (TID 38). 2648 bytes result sent to driver 20/07/01 10:38:24.451 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 38) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.452 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 20/07/01 10:38:24.452 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 16 (apply at OutcomeOf.scala:85) finished in 0.051 s 20/07/01 10:38:24.452 dag-scheduler-event-loop INFO DAGScheduler: Job 13 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.452 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 16: Stage finished 20/07/01 10:38:24.452 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 13 finished: apply at OutcomeOf.scala:85, took 0.105559 s 20/07/01 10:38:24.454 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:24.456 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_14_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.458 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_18_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:24.459 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_17_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.461 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.461 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.461 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_22_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:24.463 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_15_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.467 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_8_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.469 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_23_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.471 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_21_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.473 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_9_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.474 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_1_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:24.476 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_10_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 63.9 KiB, free: 2.1 GiB) 20/07/01 10:38:24.477 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_12_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:24.479 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_25_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 61.6 KiB, free: 2.1 GiB) 20/07/01 10:38:24.480 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_5_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.482 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_4_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 63.9 KiB, free: 2.1 GiB) 20/07/01 10:38:24.484 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_26_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 30.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.485 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:24.485 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:24.485 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_20_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:24.485 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:24.485 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:24.487 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_16_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 63.3 KiB, free: 2.1 GiB) 20/07/01 10:38:24.488 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_7_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.490 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_3_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.492 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_0_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.493 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_24_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.495 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_11_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.496 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_2_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.497 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.498 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_13_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.499 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_19_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.510 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.511 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.512 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 31 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.512 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:24.525 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 65 (apply at OutcomeOf.scala:85) as input to shuffle 3 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Got job 14 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 18 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 17) 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 17) 20/07/01 10:38:24.526 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 17 (MapPartitionsRDD[65] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.528 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:24.529 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.530 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:24.530 dag-scheduler-event-loop INFO SparkContext: Created broadcast 32 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.530 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 17 (MapPartitionsRDD[65] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.530 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 17.0 with 1 tasks resource profile 0 20/07/01 10:38:24.531 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 39, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7802 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.532 Executor task launch worker for task 39 INFO Executor: Running task 0.0 in stage 17.0 (TID 39) 20/07/01 10:38:24.535 Executor task launch worker for task 39 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:24.538 Executor task launch worker for task 39 INFO Executor: Finished task 0.0 in stage 17.0 (TID 39). 1935 bytes result sent to driver 20/07/01 10:38:24.539 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 39) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.539 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 17 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 18) 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:24.540 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 18 (MapPartitionsRDD[68] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.542 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:24.544 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.544 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_33_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.544 dag-scheduler-event-loop INFO SparkContext: Created broadcast 33 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.545 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[68] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.545 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 18.0 with 1 tasks resource profile 0 20/07/01 10:38:24.546 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 40, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.546 Executor task launch worker for task 40 INFO Executor: Running task 0.0 in stage 18.0 (TID 40) 20/07/01 10:38:24.548 Executor task launch worker for task 40 INFO ShuffleBlockFetcherIterator: Getting 1 (60.0 B) non-empty blocks including 1 (60.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:24.548 Executor task launch worker for task 40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:24.550 Executor task launch worker for task 40 INFO Executor: Finished task 0.0 in stage 18.0 (TID 40). 2648 bytes result sent to driver 20/07/01 10:38:24.550 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 40) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.551 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool 20/07/01 10:38:24.551 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 18 (apply at OutcomeOf.scala:85) finished in 0.010 s 20/07/01 10:38:24.551 dag-scheduler-event-loop INFO DAGScheduler: Job 14 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.551 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 18: Stage finished 20/07/01 10:38:24.551 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 14 finished: apply at OutcomeOf.scala:85, took 0.026074 s 20/07/01 10:38:24.555 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'convert formats' ===== 20/07/01 10:38:24.556 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'rearrange internal schema' ===== 20/07/01 10:38:24.562 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:24.577 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.578 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.592 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:24.592 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:24.592 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:24.593 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:24.596 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:24.597 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:24.597 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:24.597 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:24.615 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 9.034395 ms 20/07/01 10:38:24.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.626 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.626 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.627 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 34 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.627 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:24.644 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.644 dag-scheduler-event-loop INFO DAGScheduler: Got job 15 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.644 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 19 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.644 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:24.644 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:24.645 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[71] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.707 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 175.2 KiB, free 2.1 GiB) 20/07/01 10:38:24.709 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:24.710 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:24.710 dag-scheduler-event-loop INFO SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.711 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[71] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.711 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 19.0 with 1 tasks resource profile 0 20/07/01 10:38:24.712 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 41, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.713 Executor task launch worker for task 41 INFO Executor: Running task 0.0 in stage 19.0 (TID 41) 20/07/01 10:38:24.729 Executor task launch worker for task 41 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:24.730 Executor task launch worker for task 41 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:24.730 Executor task launch worker for task 41 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:24.748 Executor task launch worker for task 41 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:24.768 Executor task launch worker for task 41 INFO CodeGenerator: Code generated in 18.553491 ms 20/07/01 10:38:24.773 Executor task launch worker for task 41 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103824_0019_m_000000_41' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd90fd29-3a77-4c68-bb91-700763d2f06b 20/07/01 10:38:24.773 Executor task launch worker for task 41 INFO SparkHadoopMapRedUtil: attempt_20200701103824_0019_m_000000_41: Committed 20/07/01 10:38:24.774 Executor task launch worker for task 41 INFO Executor: Finished task 0.0 in stage 19.0 (TID 41). 2270 bytes result sent to driver 20/07/01 10:38:24.775 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 41) in 63 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.775 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 20/07/01 10:38:24.775 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 19 (apply at OutcomeOf.scala:85) finished in 0.130 s 20/07/01 10:38:24.775 dag-scheduler-event-loop INFO DAGScheduler: Job 15 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.775 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 19: Stage finished 20/07/01 10:38:24.776 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 15 finished: apply at OutcomeOf.scala:85, took 0.131792 s 20/07/01 10:38:24.782 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 5f990b74-e347-4ab9-ae89-ed497c035482 committed. 20/07/01 10:38:24.782 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 5f990b74-e347-4ab9-ae89-ed497c035482. 20/07/01 10:38:24.786 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'rearrange internal schema' ===== 20/07/01 10:38:24.787 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'union(int, long) is read as long' ===== 20/07/01 10:38:24.793 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:24.801 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.801 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.807 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:24.807 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:24.807 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:24.808 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:24.812 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_36 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.822 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.822 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_36_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.823 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 36 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.823 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:24.834 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.834 dag-scheduler-event-loop INFO DAGScheduler: Got job 16 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.834 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 20 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.835 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:24.835 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:24.835 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[76] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.836 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:24.838 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.838 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_37_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.838 dag-scheduler-event-loop INFO SparkContext: Created broadcast 37 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.838 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[76] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.839 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 20.0 with 1 tasks resource profile 0 20/07/01 10:38:24.839 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 42, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.840 Executor task launch worker for task 42 INFO Executor: Running task 0.0 in stage 20.0 (TID 42) 20/07/01 10:38:24.842 Executor task launch worker for task 42 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-9ae0df59-a776-444d-b51e-7f169f5ba5a1.avro, range: 0-189, partition values: [empty row] 20/07/01 10:38:24.847 Executor task launch worker for task 42 INFO Executor: Finished task 0.0 in stage 20.0 (TID 42). 1424 bytes result sent to driver 20/07/01 10:38:24.848 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 42) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.848 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 20/07/01 10:38:24.848 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 20 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:24.849 dag-scheduler-event-loop INFO DAGScheduler: Job 16 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.849 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 20: Stage finished 20/07/01 10:38:24.849 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 16 finished: apply at OutcomeOf.scala:85, took 0.015057 s 20/07/01 10:38:24.858 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.873685 ms 20/07/01 10:38:24.861 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'union(int, long) is read as long' ===== 20/07/01 10:38:24.863 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'union(float, double) is read as double' ===== 20/07/01 10:38:24.868 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:24.873 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.873 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.879 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:24.879 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:24.879 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:24.879 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:24.885 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_38 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.893 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.894 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_38_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.895 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 38 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.895 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:24.904 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.904 dag-scheduler-event-loop INFO DAGScheduler: Got job 17 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.904 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 21 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.904 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:24.905 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:24.905 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[79] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.906 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:24.908 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.908 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.908 dag-scheduler-event-loop INFO SparkContext: Created broadcast 39 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:24.908 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[79] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:24.908 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 21.0 with 1 tasks resource profile 0 20/07/01 10:38:24.909 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 43, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:24.910 Executor task launch worker for task 43 INFO Executor: Running task 0.0 in stage 21.0 (TID 43) 20/07/01 10:38:24.912 Executor task launch worker for task 43 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-6d7dc8d7-0eba-4a03-a253-7ee3945a0f23.avro, range: 0-203, partition values: [empty row] 20/07/01 10:38:24.922 Executor task launch worker for task 43 INFO CodeGenerator: Code generated in 9.360769 ms 20/07/01 10:38:24.928 Executor task launch worker for task 43 INFO Executor: Finished task 0.0 in stage 21.0 (TID 43). 1416 bytes result sent to driver 20/07/01 10:38:24.929 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 43) in 20 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:24.929 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 20/07/01 10:38:24.929 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 21 (apply at OutcomeOf.scala:85) finished in 0.024 s 20/07/01 10:38:24.929 dag-scheduler-event-loop INFO DAGScheduler: Job 17 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:24.929 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 21: Stage finished 20/07/01 10:38:24.930 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 17 finished: apply at OutcomeOf.scala:85, took 0.025738 s 20/07/01 10:38:24.942 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 10.841452 ms 20/07/01 10:38:24.947 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'union(float, double) is read as double' ===== 20/07/01 10:38:24.948 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'union(float, double, null) is read as nullable double' ===== 20/07/01 10:38:24.953 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:24.959 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.959 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:24.965 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:24.965 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:24.965 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:24.965 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:24.973 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:24.986 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:24.986 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_40_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:24.987 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 40 from apply at OutcomeOf.scala:85 20/07/01 10:38:24.988 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:24.995 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:24.996 dag-scheduler-event-loop INFO DAGScheduler: Got job 18 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:24.996 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 22 (apply at OutcomeOf.scala:85) 20/07/01 10:38:24.996 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:24.996 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:24.997 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[82] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:24.998 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:24.999 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.000 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_41_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.000 dag-scheduler-event-loop INFO SparkContext: Created broadcast 41 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.000 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 22 (MapPartitionsRDD[82] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.000 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 22.0 with 1 tasks resource profile 0 20/07/01 10:38:25.001 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 44, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.001 Executor task launch worker for task 44 INFO Executor: Running task 0.0 in stage 22.0 (TID 44) 20/07/01 10:38:25.004 Executor task launch worker for task 44 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-8cef8647-5413-46e2-8563-916a8571fb39.avro, range: 0-202, partition values: [empty row] 20/07/01 10:38:25.007 Executor task launch worker for task 44 INFO Executor: Finished task 0.0 in stage 22.0 (TID 44). 1420 bytes result sent to driver 20/07/01 10:38:25.007 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 44) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.007 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 20/07/01 10:38:25.008 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 22 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:25.008 dag-scheduler-event-loop INFO DAGScheduler: Job 18 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.008 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 22: Stage finished 20/07/01 10:38:25.008 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 18 finished: apply at OutcomeOf.scala:85, took 0.012726 s 20/07/01 10:38:25.014 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'union(float, double, null) is read as nullable double' ===== 20/07/01 10:38:25.016 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Union of a single type' ===== 20/07/01 10:38:25.023 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:25.035 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.035 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:25.056 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_42 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.064 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.065 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_42_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.065 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 42 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.066 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.071 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.072 dag-scheduler-event-loop INFO DAGScheduler: Got job 19 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.072 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 23 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.072 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.072 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.072 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[85] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.074 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 7.4 KiB, free 2.1 GiB) 20/07/01 10:38:25.075 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.075 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_43_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.076 dag-scheduler-event-loop INFO SparkContext: Created broadcast 43 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.076 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[85] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.076 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 23.0 with 1 tasks resource profile 0 20/07/01 10:38:25.077 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 23.0 (TID 45, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.077 Executor task launch worker for task 45 INFO Executor: Running task 0.0 in stage 23.0 (TID 45) 20/07/01 10:38:25.080 Executor task launch worker for task 45 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-69c0ba44-e8be-405f-af62-0a8a5d13a585.avro, range: 0-180, partition values: [empty row] 20/07/01 10:38:25.083 Executor task launch worker for task 45 INFO Executor: Finished task 0.0 in stage 23.0 (TID 45). 1378 bytes result sent to driver 20/07/01 10:38:25.084 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 23.0 (TID 45) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.084 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 20/07/01 10:38:25.084 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 23 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:25.084 dag-scheduler-event-loop INFO DAGScheduler: Job 19 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.084 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 23: Stage finished 20/07/01 10:38:25.085 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 19 finished: apply at OutcomeOf.scala:85, took 0.013257 s 20/07/01 10:38:25.088 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Union of a single type' ===== 20/07/01 10:38:25.090 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'SPARK-27858 Union type: More than one non-null type' ===== 20/07/01 10:38:25.095 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:25.205 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.205 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.211 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.211 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.211 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.211 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:25.223 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.254 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.254 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.255 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 44 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.255 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.262 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.263 dag-scheduler-event-loop INFO DAGScheduler: Got job 20 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.263 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 24 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.263 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.263 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.263 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[88] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.265 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 7.6 KiB, free 2.1 GiB) 20/07/01 10:38:25.266 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:25.266 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:25.266 dag-scheduler-event-loop INFO SparkContext: Created broadcast 45 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.267 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[88] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.267 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 24.0 with 1 tasks resource profile 0 20/07/01 10:38:25.268 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 46, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.268 Executor task launch worker for task 46 INFO Executor: Running task 0.0 in stage 24.0 (TID 46) 20/07/01 10:38:25.270 Executor task launch worker for task 46 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-112cf5f5-c5e8-4303-99d1-ddda29436b97.avro, range: 0-203, partition values: [empty row] 20/07/01 10:38:25.287 Executor task launch worker for task 46 INFO CodeGenerator: Code generated in 13.056689 ms 20/07/01 10:38:25.296 Executor task launch worker for task 46 INFO Executor: Finished task 0.0 in stage 24.0 (TID 46). 1455 bytes result sent to driver 20/07/01 10:38:25.297 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 46) in 30 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.297 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 20/07/01 10:38:25.298 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 24 (apply at OutcomeOf.scala:85) finished in 0.034 s 20/07/01 10:38:25.298 dag-scheduler-event-loop INFO DAGScheduler: Job 20 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.298 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 24: Stage finished 20/07/01 10:38:25.299 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 20 finished: apply at OutcomeOf.scala:85, took 0.036355 s 20/07/01 10:38:25.340 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 26.944548 ms 20/07/01 10:38:25.344 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'SPARK-27858 Union type: More than one non-null type' ===== 20/07/01 10:38:25.345 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Complex Union Type' ===== 20/07/01 10:38:25.353 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:25.377 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.377 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.393 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.393 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.393 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.394 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:25.410 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 7.442495 ms 20/07/01 10:38:25.411 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.445 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.445 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.446 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 46 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.447 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.454 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.454 dag-scheduler-event-loop INFO DAGScheduler: Got job 21 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.455 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 25 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.455 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.455 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.455 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[92] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.458 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_47 stored as values in memory (estimated size 10.6 KiB, free 2.1 GiB) 20/07/01 10:38:25.459 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_47_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 2.1 GiB) 20/07/01 10:38:25.459 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_47_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:25.459 dag-scheduler-event-loop INFO SparkContext: Created broadcast 47 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.460 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[92] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.460 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 25.0 with 1 tasks resource profile 0 20/07/01 10:38:25.461 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 25.0 (TID 47, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.461 Executor task launch worker for task 47 INFO Executor: Running task 0.0 in stage 25.0 (TID 47) 20/07/01 10:38:25.466 Executor task launch worker for task 47 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-89dff60b-cf02-4e71-a812-3a1cda103603.avro, range: 0-593, partition values: [empty row] 20/07/01 10:38:25.476 Executor task launch worker for task 47 INFO CodeGenerator: Code generated in 8.713452 ms 20/07/01 10:38:25.480 Executor task launch worker for task 47 INFO Executor: Finished task 0.0 in stage 25.0 (TID 47). 1450 bytes result sent to driver 20/07/01 10:38:25.481 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 25.0 (TID 47) in 21 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.481 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 20/07/01 10:38:25.482 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 25 (apply at OutcomeOf.scala:85) finished in 0.026 s 20/07/01 10:38:25.482 dag-scheduler-event-loop INFO DAGScheduler: Job 21 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.482 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 25: Stage finished 20/07/01 10:38:25.482 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 21 finished: apply at OutcomeOf.scala:85, took 0.028140 s 20/07/01 10:38:25.492 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.492 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.506 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.506 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.506 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.506 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:25.520 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.932266 ms 20/07/01 10:38:25.522 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.553 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.553 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_48_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.554 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 48 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.555 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.561 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.561 dag-scheduler-event-loop INFO DAGScheduler: Got job 22 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.562 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 26 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.562 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.562 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.562 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 26 (MapPartitionsRDD[96] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.564 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49 stored as values in memory (estimated size 10.6 KiB, free 2.1 GiB) 20/07/01 10:38:25.565 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_49_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 2.1 GiB) 20/07/01 10:38:25.565 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_49_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:25.566 dag-scheduler-event-loop INFO SparkContext: Created broadcast 49 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.566 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[96] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.566 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 26.0 with 1 tasks resource profile 0 20/07/01 10:38:25.567 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 48, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.567 Executor task launch worker for task 48 INFO Executor: Running task 0.0 in stage 26.0 (TID 48) 20/07/01 10:38:25.572 Executor task launch worker for task 48 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-89dff60b-cf02-4e71-a812-3a1cda103603.avro, range: 0-593, partition values: [empty row] 20/07/01 10:38:25.576 Executor task launch worker for task 48 INFO Executor: Finished task 0.0 in stage 26.0 (TID 48). 1486 bytes result sent to driver 20/07/01 10:38:25.576 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 26.0 (TID 48) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.577 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 26.0, whose tasks have all completed, from pool 20/07/01 10:38:25.577 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 26 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:25.577 dag-scheduler-event-loop INFO DAGScheduler: Job 22 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.577 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 26: Stage finished 20/07/01 10:38:25.577 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 22 finished: apply at OutcomeOf.scala:85, took 0.016579 s 20/07/01 10:38:25.586 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.586 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.602 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.603 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.603 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.603 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:25.620 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 8.066267 ms 20/07/01 10:38:25.622 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_50 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.654 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_50_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.655 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_50_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.655 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 50 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.656 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.661 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.662 dag-scheduler-event-loop INFO DAGScheduler: Got job 23 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.662 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 27 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.662 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.662 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.663 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[100] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.665 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_51 stored as values in memory (estimated size 10.6 KiB, free 2.1 GiB) 20/07/01 10:38:25.666 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_51_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 2.1 GiB) 20/07/01 10:38:25.666 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_51_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:25.666 dag-scheduler-event-loop INFO SparkContext: Created broadcast 51 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.667 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[100] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.667 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 27.0 with 1 tasks resource profile 0 20/07/01 10:38:25.667 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 49, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.668 Executor task launch worker for task 49 INFO Executor: Running task 0.0 in stage 27.0 (TID 49) 20/07/01 10:38:25.672 Executor task launch worker for task 49 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-89dff60b-cf02-4e71-a812-3a1cda103603.avro, range: 0-593, partition values: [empty row] 20/07/01 10:38:25.676 Executor task launch worker for task 49 INFO Executor: Finished task 0.0 in stage 27.0 (TID 49). 1459 bytes result sent to driver 20/07/01 10:38:25.677 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 49) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.677 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 20/07/01 10:38:25.677 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 27 (apply at OutcomeOf.scala:85) finished in 0.014 s 20/07/01 10:38:25.677 dag-scheduler-event-loop INFO DAGScheduler: Job 23 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.677 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 27: Stage finished 20/07/01 10:38:25.678 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 23 finished: apply at OutcomeOf.scala:85, took 0.016156 s 20/07/01 10:38:25.685 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.063166 ms 20/07/01 10:38:25.693 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.693 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.709 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:25.710 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:25.710 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:25.710 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:25.723 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.47473 ms 20/07/01 10:38:25.725 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_52 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:25.757 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_52_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:25.757 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_52_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:25.758 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 52 from apply at OutcomeOf.scala:85 20/07/01 10:38:25.758 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:25.764 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.764 dag-scheduler-event-loop INFO DAGScheduler: Got job 24 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:25.765 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 28 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.765 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.765 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.765 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[104] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:25.767 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53 stored as values in memory (estimated size 10.6 KiB, free 2.1 GiB) 20/07/01 10:38:25.769 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_53_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 2.1 GiB) 20/07/01 10:38:25.769 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_53_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:25.769 dag-scheduler-event-loop INFO SparkContext: Created broadcast 53 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:25.770 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 28 (MapPartitionsRDD[104] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:25.770 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 28.0 with 1 tasks resource profile 0 20/07/01 10:38:25.771 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 28.0 (TID 50, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:25.771 Executor task launch worker for task 50 INFO Executor: Running task 0.0 in stage 28.0 (TID 50) 20/07/01 10:38:25.776 Executor task launch worker for task 50 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-89dff60b-cf02-4e71-a812-3a1cda103603.avro, range: 0-593, partition values: [empty row] 20/07/01 10:38:25.781 Executor task launch worker for task 50 INFO Executor: Finished task 0.0 in stage 28.0 (TID 50). 1459 bytes result sent to driver 20/07/01 10:38:25.781 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 28.0 (TID 50) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:25.781 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 20/07/01 10:38:25.782 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 28 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:25.782 dag-scheduler-event-loop INFO DAGScheduler: Job 24 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:25.782 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 28: Stage finished 20/07/01 10:38:25.782 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 24 finished: apply at OutcomeOf.scala:85, took 0.018467 s 20/07/01 10:38:25.786 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Complex Union Type' ===== 20/07/01 10:38:25.788 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Lots of nulls' ===== 20/07/01 10:38:25.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:25.853 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:25.854 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:25.854 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:25.854 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:25.885 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 16.608386 ms 20/07/01 10:38:25.957 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:25.958 dag-scheduler-event-loop INFO DAGScheduler: Got job 25 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:25.958 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 29 (apply at OutcomeOf.scala:85) 20/07/01 10:38:25.958 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:25.958 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:25.958 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[107] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.057 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_54 stored as values in memory (estimated size 188.4 KiB, free 2.1 GiB) 20/07/01 10:38:26.065 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_54_piece0 stored as bytes in memory (estimated size 66.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.065 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_54_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 66.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.066 dag-scheduler-event-loop INFO SparkContext: Created broadcast 54 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.066 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 29 (MapPartitionsRDD[107] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.066 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 29.0 with 2 tasks resource profile 0 20/07/01 10:38:26.069 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 29.0 (TID 51, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7620 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.069 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 29.0 (TID 52, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7506 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.069 Executor task launch worker for task 51 INFO Executor: Running task 0.0 in stage 29.0 (TID 51) 20/07/01 10:38:26.069 Executor task launch worker for task 52 INFO Executor: Running task 1.0 in stage 29.0 (TID 52) 20/07/01 10:38:26.102 Executor task launch worker for task 51 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:26.102 Executor task launch worker for task 51 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:26.102 Executor task launch worker for task 51 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:26.106 Executor task launch worker for task 52 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:26.106 Executor task launch worker for task 52 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:26.106 Executor task launch worker for task 52 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:26.239 Executor task launch worker for task 51 INFO CodeGenerator: Code generated in 59.033233 ms 20/07/01 10:38:26.251 Executor task launch worker for task 51 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103825_0029_m_000000_51' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-64ae5fda-4132-46b4-ad73-ceeea7100d1b 20/07/01 10:38:26.251 Executor task launch worker for task 51 INFO SparkHadoopMapRedUtil: attempt_20200701103825_0029_m_000000_51: Committed 20/07/01 10:38:26.252 Executor task launch worker for task 51 INFO Executor: Finished task 0.0 in stage 29.0 (TID 51). 2184 bytes result sent to driver 20/07/01 10:38:26.253 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 29.0 (TID 51) in 186 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.258 Executor task launch worker for task 52 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103825_0029_m_000001_52' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-64ae5fda-4132-46b4-ad73-ceeea7100d1b 20/07/01 10:38:26.258 Executor task launch worker for task 52 INFO SparkHadoopMapRedUtil: attempt_20200701103825_0029_m_000001_52: Committed 20/07/01 10:38:26.259 Executor task launch worker for task 52 INFO Executor: Finished task 1.0 in stage 29.0 (TID 52). 2184 bytes result sent to driver 20/07/01 10:38:26.260 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 29.0 (TID 52) in 191 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.260 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 20/07/01 10:38:26.260 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 29 (apply at OutcomeOf.scala:85) finished in 0.301 s 20/07/01 10:38:26.260 dag-scheduler-event-loop INFO DAGScheduler: Job 25 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.260 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 29: Stage finished 20/07/01 10:38:26.261 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 25 finished: apply at OutcomeOf.scala:85, took 0.303630 s 20/07/01 10:38:26.267 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 17c73735-39a0-4301-bf47-506fd17b58e4 committed. 20/07/01 10:38:26.267 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 17c73735-39a0-4301-bf47-506fd17b58e4. 20/07/01 10:38:26.273 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:26.287 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.287 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.303 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:26.304 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:26.304 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:26.304 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:26.312 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_55 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:26.351 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_55_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.352 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_55_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.352 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 55 from apply at OutcomeOf.scala:85 20/07/01 10:38:26.353 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:26.362 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:26.363 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 113 (apply at OutcomeOf.scala:85) as input to shuffle 4 20/07/01 10:38:26.364 dag-scheduler-event-loop INFO DAGScheduler: Got job 26 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:26.364 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 31 (apply at OutcomeOf.scala:85) 20/07/01 10:38:26.364 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 30) 20/07/01 10:38:26.364 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 30) 20/07/01 10:38:26.364 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 30 (MapPartitionsRDD[113] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.366 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_56 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:26.367 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_56_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:26.367 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_56_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:26.368 dag-scheduler-event-loop INFO SparkContext: Created broadcast 56 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.368 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 30 (MapPartitionsRDD[113] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.368 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 30.0 with 2 tasks resource profile 0 20/07/01 10:38:26.369 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 30.0 (TID 53, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.369 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 30.0 (TID 54, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.370 Executor task launch worker for task 54 INFO Executor: Running task 1.0 in stage 30.0 (TID 54) 20/07/01 10:38:26.370 Executor task launch worker for task 53 INFO Executor: Running task 0.0 in stage 30.0 (TID 53) 20/07/01 10:38:26.372 Executor task launch worker for task 54 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-64ae5fda-4132-46b4-ad73-ceeea7100d1b/part-00001-26fbb083-3315-46a3-bf84-c4d428ed7bde-c000.avro, range: 0-590, partition values: [empty row] 20/07/01 10:38:26.373 Executor task launch worker for task 53 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-64ae5fda-4132-46b4-ad73-ceeea7100d1b/part-00000-26fbb083-3315-46a3-bf84-c4d428ed7bde-c000.avro, range: 0-600, partition values: [empty row] 20/07/01 10:38:26.377 Executor task launch worker for task 54 INFO Executor: Finished task 1.0 in stage 30.0 (TID 54). 1935 bytes result sent to driver 20/07/01 10:38:26.378 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 30.0 (TID 54) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.379 Executor task launch worker for task 53 INFO Executor: Finished task 0.0 in stage 30.0 (TID 53). 1935 bytes result sent to driver 20/07/01 10:38:26.380 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 30.0 (TID 53) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.380 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 30.0, whose tasks have all completed, from pool 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 30 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 31) 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:26.380 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[116] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.382 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_57 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:26.383 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_57_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.383 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_57_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.384 dag-scheduler-event-loop INFO SparkContext: Created broadcast 57 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.384 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 31 (MapPartitionsRDD[116] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:26.384 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 31.0 with 1 tasks resource profile 0 20/07/01 10:38:26.385 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 31.0 (TID 55, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.386 Executor task launch worker for task 55 INFO Executor: Running task 0.0 in stage 31.0 (TID 55) 20/07/01 10:38:26.388 Executor task launch worker for task 55 INFO ShuffleBlockFetcherIterator: Getting 2 (120.0 B) non-empty blocks including 2 (120.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:26.389 Executor task launch worker for task 55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:26.391 Executor task launch worker for task 55 INFO Executor: Finished task 0.0 in stage 31.0 (TID 55). 2648 bytes result sent to driver 20/07/01 10:38:26.391 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 31.0 (TID 55) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:26.391 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 20/07/01 10:38:26.392 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 31 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:26.392 dag-scheduler-event-loop INFO DAGScheduler: Job 26 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.392 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 31: Stage finished 20/07/01 10:38:26.392 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 26 finished: apply at OutcomeOf.scala:85, took 0.029708 s 20/07/01 10:38:26.398 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:26.398 dag-scheduler-event-loop INFO DAGScheduler: Got job 27 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:26.398 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 32 (apply at OutcomeOf.scala:85) 20/07/01 10:38:26.398 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:26.398 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:26.399 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 32 (ParallelCollectionRDD[105] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.400 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_58 stored as values in memory (estimated size 2024.0 B, free 2.1 GiB) 20/07/01 10:38:26.401 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_58_piece0 stored as bytes in memory (estimated size 1276.0 B, free 2.1 GiB) 20/07/01 10:38:26.401 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_58_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:26.401 dag-scheduler-event-loop INFO SparkContext: Created broadcast 58 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.402 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 32 (ParallelCollectionRDD[105] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.402 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 32.0 with 2 tasks resource profile 0 20/07/01 10:38:26.403 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 56, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7620 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.403 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 32.0 (TID 57, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7506 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.403 Executor task launch worker for task 56 INFO Executor: Running task 0.0 in stage 32.0 (TID 56) 20/07/01 10:38:26.403 Executor task launch worker for task 57 INFO Executor: Running task 1.0 in stage 32.0 (TID 57) 20/07/01 10:38:26.405 Executor task launch worker for task 57 INFO Executor: Finished task 1.0 in stage 32.0 (TID 57). 789 bytes result sent to driver 20/07/01 10:38:26.405 Executor task launch worker for task 56 INFO Executor: Finished task 0.0 in stage 32.0 (TID 56). 832 bytes result sent to driver 20/07/01 10:38:26.489 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 32.0 (TID 57) in 86 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.490 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 56) in 88 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.490 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool 20/07/01 10:38:26.490 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 32 (apply at OutcomeOf.scala:85) finished in 0.091 s 20/07/01 10:38:26.490 dag-scheduler-event-loop INFO DAGScheduler: Job 27 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.490 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 32: Stage finished 20/07/01 10:38:26.491 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 27 finished: apply at OutcomeOf.scala:85, took 0.092708 s 20/07/01 10:38:26.494 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Lots of nulls' ===== 20/07/01 10:38:26.495 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Struct field type' ===== 20/07/01 10:38:26.507 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.507 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.521 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:26.522 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:26.522 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:26.522 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:26.540 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 9.293419 ms 20/07/01 10:38:26.618 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:26.619 dag-scheduler-event-loop INFO DAGScheduler: Got job 28 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:26.619 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 33 (apply at OutcomeOf.scala:85) 20/07/01 10:38:26.619 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:26.619 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:26.619 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 33 (MapPartitionsRDD[119] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.699 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59 stored as values in memory (estimated size 178.3 KiB, free 2.1 GiB) 20/07/01 10:38:26.701 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_59_piece0 stored as bytes in memory (estimated size 63.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.701 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_59_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 63.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.702 dag-scheduler-event-loop INFO SparkContext: Created broadcast 59 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.702 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 33 (MapPartitionsRDD[119] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.702 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 33.0 with 2 tasks resource profile 0 20/07/01 10:38:26.704 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 33.0 (TID 58, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7661 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.705 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 33.0 (TID 59, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7707 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.705 Executor task launch worker for task 58 INFO Executor: Running task 0.0 in stage 33.0 (TID 58) 20/07/01 10:38:26.705 Executor task launch worker for task 59 INFO Executor: Running task 1.0 in stage 33.0 (TID 59) 20/07/01 10:38:26.724 Executor task launch worker for task 59 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:26.724 Executor task launch worker for task 58 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:26.724 Executor task launch worker for task 59 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:26.724 Executor task launch worker for task 58 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:26.724 Executor task launch worker for task 59 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:26.724 Executor task launch worker for task 58 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:26.751 Executor task launch worker for task 59 INFO CodeGenerator: Code generated in 21.887124 ms 20/07/01 10:38:26.758 Executor task launch worker for task 58 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103826_0033_m_000000_58' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-603ff065-2883-49c1-b480-ce81064ce037 20/07/01 10:38:26.758 Executor task launch worker for task 58 INFO SparkHadoopMapRedUtil: attempt_20200701103826_0033_m_000000_58: Committed 20/07/01 10:38:26.759 Executor task launch worker for task 58 INFO Executor: Finished task 0.0 in stage 33.0 (TID 58). 2184 bytes result sent to driver 20/07/01 10:38:26.760 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 33.0 (TID 58) in 57 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.765 Executor task launch worker for task 59 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103826_0033_m_000001_59' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-603ff065-2883-49c1-b480-ce81064ce037 20/07/01 10:38:26.765 Executor task launch worker for task 59 INFO SparkHadoopMapRedUtil: attempt_20200701103826_0033_m_000001_59: Committed 20/07/01 10:38:26.766 Executor task launch worker for task 59 INFO Executor: Finished task 1.0 in stage 33.0 (TID 59). 2184 bytes result sent to driver 20/07/01 10:38:26.767 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 33.0 (TID 59) in 62 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.767 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 33.0, whose tasks have all completed, from pool 20/07/01 10:38:26.768 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 33 (apply at OutcomeOf.scala:85) finished in 0.148 s 20/07/01 10:38:26.768 dag-scheduler-event-loop INFO DAGScheduler: Job 28 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.768 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 33: Stage finished 20/07/01 10:38:26.768 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 28 finished: apply at OutcomeOf.scala:85, took 0.149876 s 20/07/01 10:38:26.774 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 3a792ea0-62ca-4f5a-bdda-f4443e8a36c0 committed. 20/07/01 10:38:26.774 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 3a792ea0-62ca-4f5a-bdda-f4443e8a36c0. 20/07/01 10:38:26.781 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:26.795 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.795 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:26.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:26.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:26.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:26.817 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:26.826 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_60 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:26.894 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_60_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.894 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_60_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.895 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 60 from apply at OutcomeOf.scala:85 20/07/01 10:38:26.896 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:26.909 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:26.910 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 125 (apply at OutcomeOf.scala:85) as input to shuffle 5 20/07/01 10:38:26.910 dag-scheduler-event-loop INFO DAGScheduler: Got job 29 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:26.910 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 35 (apply at OutcomeOf.scala:85) 20/07/01 10:38:26.910 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 34) 20/07/01 10:38:26.910 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 34) 20/07/01 10:38:26.911 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 34 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.912 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_61 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:26.914 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_61_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:26.914 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_61_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:26.915 dag-scheduler-event-loop INFO SparkContext: Created broadcast 61 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.915 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 34 (MapPartitionsRDD[125] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.915 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 34.0 with 2 tasks resource profile 0 20/07/01 10:38:26.916 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 34.0 (TID 60, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.916 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 34.0 (TID 61, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.917 Executor task launch worker for task 60 INFO Executor: Running task 0.0 in stage 34.0 (TID 60) 20/07/01 10:38:26.917 Executor task launch worker for task 61 INFO Executor: Running task 1.0 in stage 34.0 (TID 61) 20/07/01 10:38:26.921 Executor task launch worker for task 60 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-603ff065-2883-49c1-b480-ce81064ce037/part-00001-b9d589d6-bcc2-4a98-901d-17812cf589f7-c000.avro, range: 0-346, partition values: [empty row] 20/07/01 10:38:26.922 Executor task launch worker for task 61 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-603ff065-2883-49c1-b480-ce81064ce037/part-00000-b9d589d6-bcc2-4a98-901d-17812cf589f7-c000.avro, range: 0-335, partition values: [empty row] 20/07/01 10:38:26.927 Executor task launch worker for task 61 INFO Executor: Finished task 1.0 in stage 34.0 (TID 61). 1935 bytes result sent to driver 20/07/01 10:38:26.928 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 34.0 (TID 61) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.928 Executor task launch worker for task 60 INFO Executor: Finished task 0.0 in stage 34.0 (TID 60). 1935 bytes result sent to driver 20/07/01 10:38:26.929 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 34.0 (TID 60) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.929 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool 20/07/01 10:38:26.929 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 34 (apply at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:26.929 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:26.930 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:26.930 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 35) 20/07/01 10:38:26.930 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:26.930 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 35 (MapPartitionsRDD[128] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.931 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_62 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:26.933 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_62_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:26.933 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_62_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:26.938 dag-scheduler-event-loop INFO SparkContext: Created broadcast 62 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.938 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 35 (MapPartitionsRDD[128] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:26.938 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 35.0 with 1 tasks resource profile 0 20/07/01 10:38:26.940 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 35.0 (TID 62, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.940 Executor task launch worker for task 62 INFO Executor: Running task 0.0 in stage 35.0 (TID 62) 20/07/01 10:38:26.943 Executor task launch worker for task 62 INFO ShuffleBlockFetcherIterator: Getting 2 (120.0 B) non-empty blocks including 2 (120.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:26.943 Executor task launch worker for task 62 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:26.945 Executor task launch worker for task 62 INFO Executor: Finished task 0.0 in stage 35.0 (TID 62). 2648 bytes result sent to driver 20/07/01 10:38:26.946 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 35.0 (TID 62) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:26.946 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 35.0, whose tasks have all completed, from pool 20/07/01 10:38:26.947 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 35 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:26.948 dag-scheduler-event-loop INFO DAGScheduler: Job 29 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.948 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 35: Stage finished 20/07/01 10:38:26.948 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 29 finished: apply at OutcomeOf.scala:85, took 0.038547 s 20/07/01 10:38:26.955 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:26.956 dag-scheduler-event-loop INFO DAGScheduler: Got job 30 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:26.956 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 36 (apply at OutcomeOf.scala:85) 20/07/01 10:38:26.956 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:26.956 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:26.956 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 36 (ParallelCollectionRDD[117] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:26.957 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_63 stored as values in memory (estimated size 2024.0 B, free 2.1 GiB) 20/07/01 10:38:26.959 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_63_piece0 stored as bytes in memory (estimated size 1276.0 B, free 2.1 GiB) 20/07/01 10:38:26.959 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_63_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:26.960 dag-scheduler-event-loop INFO SparkContext: Created broadcast 63 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:26.960 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 36 (ParallelCollectionRDD[117] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:26.960 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 36.0 with 2 tasks resource profile 0 20/07/01 10:38:26.961 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 36.0 (TID 63, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7661 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.961 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 36.0 (TID 64, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7707 bytes) taskResourceAssignments Map() 20/07/01 10:38:26.961 Executor task launch worker for task 64 INFO Executor: Running task 1.0 in stage 36.0 (TID 64) 20/07/01 10:38:26.961 Executor task launch worker for task 63 INFO Executor: Running task 0.0 in stage 36.0 (TID 63) 20/07/01 10:38:26.963 Executor task launch worker for task 64 INFO Executor: Finished task 1.0 in stage 36.0 (TID 64). 832 bytes result sent to driver 20/07/01 10:38:26.963 Executor task launch worker for task 63 INFO Executor: Finished task 0.0 in stage 36.0 (TID 63). 832 bytes result sent to driver 20/07/01 10:38:26.964 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 36.0 (TID 64) in 3 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:26.964 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 36.0 (TID 63) in 3 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:26.964 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 36.0, whose tasks have all completed, from pool 20/07/01 10:38:26.965 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 36 (apply at OutcomeOf.scala:85) finished in 0.008 s 20/07/01 10:38:26.965 dag-scheduler-event-loop INFO DAGScheduler: Job 30 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:26.965 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 36: Stage finished 20/07/01 10:38:26.966 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 30 finished: apply at OutcomeOf.scala:85, took 0.009974 s 20/07/01 10:38:26.971 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Struct field type' ===== 20/07/01 10:38:26.972 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled' ===== 20/07/01 10:38:26.997 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:27.006 pool-1-thread-1-ScalaTest-running-AvroV1Suite WARN AvroUtils: Skipped the footer in the corrupted file: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/corrupt.avro java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63) at org.apache.spark.sql.avro.AvroUtils$.$anonfun$inferAvroSchemaFromFiles$3(AvroUtils.scala:136) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539) at org.apache.spark.sql.avro.AvroUtils$.$anonfun$inferAvroSchemaFromFiles$1(AvroUtils.scala:134) at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) at scala.collection.TraversableOnce.collectFirst(TraversableOnce.scala:148) at scala.collection.TraversableOnce.collectFirst$(TraversableOnce.scala:135) at scala.collection.AbstractIterator.collectFirst(Iterator.scala:1429) at org.apache.spark.sql.avro.AvroUtils$.inferAvroSchemaFromFiles(AvroUtils.scala:148) at org.apache.spark.sql.avro.AvroUtils$.$anonfun$inferSchema$3(AvroUtils.scala:57) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.avro.AvroUtils$.inferSchema(AvroUtils.scala:56) at org.apache.spark.sql.avro.AvroFileFormat.inferSchema(AvroFileFormat.scala:58) at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$11(DataSource.scala:197) at scala.Option.orElse(Option.scala:447) at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:194) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:405) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279) at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:214) at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$35(AvroSuite.scala:378) at org.scalatest.Assertions.intercept(Assertions.scala:807) at org.scalatest.Assertions.intercept$(Assertions.scala:804) at org.scalatest.FunSuite.intercept(FunSuite.scala:1560) at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:377) at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:375) at org.apache.spark.sql.catalyst.plans.SQLHelper.withTempPath(SQLHelper.scala:69) at org.apache.spark.sql.catalyst.plans.SQLHelper.withTempPath$(SQLHelper.scala:66) at org.apache.spark.sql.QueryTest.withTempPath(QueryTest.scala:34) at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$33(AvroSuite.scala:375) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38) at org.apache.spark.sql.avro.AvroSuite.org$apache$spark$sql$test$SQLTestUtilsBase$$super$withSQLConf(AvroSuite.scala:52) at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf(SQLTestUtils.scala:246) at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf$(SQLTestUtils.scala:244) at org.apache.spark.sql.avro.AvroSuite.withSQLConf(AvroSuite.scala:52) at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$32(AvroSuite.scala:375) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:157) at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184) at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286) at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196) at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:59) at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221) at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214) at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:59) at org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229) at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:393) at scala.collection.immutable.List.foreach(List.scala:392) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381) at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:376) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:458) at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229) at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228) at org.scalatest.FunSuite.runTests(FunSuite.scala:1560) at org.scalatest.Suite.run(Suite.scala:1124) at org.scalatest.Suite.run$(Suite.scala:1106) at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560) at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233) at org.scalatest.SuperEngine.runImpl(Engine.scala:518) at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233) at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232) at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:59) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:59) at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:317) at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:510) at sbt.ForkMain$Run$2.call(ForkMain.java:296) at sbt.ForkMain$Run$2.call(ForkMain.java:286) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:27.015 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:27.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.026 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.026 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.026 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.027 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:27.032 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_64 stored as values in memory (estimated size 307.8 KiB, free 2.1 GiB) 20/07/01 10:38:27.065 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_64_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:27.066 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_64_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:27.066 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 64 from apply at OutcomeOf.scala:85 20/07/01 10:38:27.067 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:27.073 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.074 dag-scheduler-event-loop INFO DAGScheduler: Got job 31 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:27.074 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 37 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.074 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:27.074 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:27.074 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 37 (MapPartitionsRDD[131] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.075 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65 stored as values in memory (estimated size 7.7 KiB, free 2.1 GiB) 20/07/01 10:38:27.076 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_65_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:27.077 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:27.077 dag-scheduler-event-loop INFO SparkContext: Created broadcast 65 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.077 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 37 (MapPartitionsRDD[131] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:27.077 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 37.0 with 1 tasks resource profile 0 20/07/01 10:38:27.078 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 37.0 (TID 65, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7801 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.078 Executor task launch worker for task 65 INFO Executor: Running task 0.0 in stage 37.0 (TID 65) 20/07/01 10:38:27.081 Executor task launch worker for task 65 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/src/test/resources/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:27.087 Executor task launch worker for task 65 INFO Executor: Finished task 0.0 in stage 37.0 (TID 65). 1796 bytes result sent to driver 20/07/01 10:38:27.088 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 37.0 (TID 65) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:27.088 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 37.0, whose tasks have all completed, from pool 20/07/01 10:38:27.088 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 37 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:27.089 dag-scheduler-event-loop INFO DAGScheduler: Job 31 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.089 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 37: Stage finished 20/07/01 10:38:27.089 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 31 finished: apply at OutcomeOf.scala:85, took 0.015445 s 20/07/01 10:38:27.099 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 8.21642 ms 20/07/01 10:38:27.104 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:27.109 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.110 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.116 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.116 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.116 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.116 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:27.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.130 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.130 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.130 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.130 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:27.133 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_66 stored as values in memory (estimated size 307.8 KiB, free 2.1 GiB) 20/07/01 10:38:27.179 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_66_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:27.180 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_66_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:27.180 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 66 from apply at OutcomeOf.scala:85 20/07/01 10:38:27.181 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:27.193 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.194 dag-scheduler-event-loop INFO DAGScheduler: Got job 32 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:27.194 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 38 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.194 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:27.194 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:27.194 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 38 (MapPartitionsRDD[136] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.198 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_67 stored as values in memory (estimated size 11.9 KiB, free 2.1 GiB) 20/07/01 10:38:27.200 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_67_piece0 stored as bytes in memory (estimated size 6.2 KiB, free 2.1 GiB) 20/07/01 10:38:27.200 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:27.200 dag-scheduler-event-loop INFO SparkContext: Created broadcast 67 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.201 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 38 (MapPartitionsRDD[136] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:27.201 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 38.0 with 2 tasks resource profile 0 20/07/01 10:38:27.202 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 38.0 (TID 66, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7822 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.202 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 38.0 (TID 67, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7821 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.202 Executor task launch worker for task 66 INFO Executor: Running task 0.0 in stage 38.0 (TID 66) 20/07/01 10:38:27.202 Executor task launch worker for task 67 INFO Executor: Running task 1.0 in stage 38.0 (TID 67) 20/07/01 10:38:27.207 Executor task launch worker for task 66 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:27.210 Executor task launch worker for task 67 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/corrupt.avro, range: 0-7, partition values: [empty row] 20/07/01 10:38:27.211 Executor task launch worker for task 66 INFO Executor: Finished task 0.0 in stage 38.0 (TID 66). 1554 bytes result sent to driver 20/07/01 10:38:27.212 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 38.0 (TID 66) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:27.212 Executor task launch worker for task 67 ERROR AvroFileFormat: Exception while opening DataFileReader java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63) at org.apache.spark.sql.avro.AvroFileFormat.$anonfun$buildReader$1(AvroFileFormat.scala:107) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:147) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:132) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter$lzycompute(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:145) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1804) at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1253) at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1253) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2148) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:27.213 Executor task launch worker for task 67 WARN FileScanRDD: Skipped the rest of the content in the corrupted file: path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/corrupt.avro, range: 0-7, partition values: [empty row] java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63) at org.apache.spark.sql.avro.AvroFileFormat.$anonfun$buildReader$1(AvroFileFormat.scala:107) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:147) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:132) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter$lzycompute(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:145) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1804) at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1253) at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1253) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2148) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:27.213 Executor task launch worker for task 67 INFO Executor: Finished task 1.0 in stage 38.0 (TID 67). 1511 bytes result sent to driver 20/07/01 10:38:27.214 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 38.0 (TID 67) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:27.214 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 38.0, whose tasks have all completed, from pool 20/07/01 10:38:27.214 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 38 (apply at OutcomeOf.scala:85) finished in 0.019 s 20/07/01 10:38:27.214 dag-scheduler-event-loop INFO DAGScheduler: Job 32 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.214 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 38: Stage finished 20/07/01 10:38:27.215 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 32 finished: apply at OutcomeOf.scala:85, took 0.021477 s 20/07/01 10:38:27.222 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_68 stored as values in memory (estimated size 307.8 KiB, free 2.1 GiB) 20/07/01 10:38:27.267 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_68_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:27.268 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:27.268 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 68 from apply at OutcomeOf.scala:85 20/07/01 10:38:27.269 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:27.275 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.276 dag-scheduler-event-loop INFO DAGScheduler: Got job 33 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:27.276 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 39 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.276 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:27.276 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:27.276 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 39 (MapPartitionsRDD[139] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.277 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_69 stored as values in memory (estimated size 7.7 KiB, free 2.1 GiB) 20/07/01 10:38:27.278 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_69_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:27.279 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_69_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:27.279 dag-scheduler-event-loop INFO SparkContext: Created broadcast 69 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.279 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 39 (MapPartitionsRDD[139] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:27.279 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 39.0 with 2 tasks resource profile 0 20/07/01 10:38:27.280 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 39.0 (TID 68, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7822 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.280 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 39.0 (TID 69, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7821 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.281 Executor task launch worker for task 68 INFO Executor: Running task 0.0 in stage 39.0 (TID 68) 20/07/01 10:38:27.281 Executor task launch worker for task 69 INFO Executor: Running task 1.0 in stage 39.0 (TID 69) 20/07/01 10:38:27.284 Executor task launch worker for task 68 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:27.284 Executor task launch worker for task 69 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/corrupt.avro, range: 0-7, partition values: [empty row] 20/07/01 10:38:27.286 Executor task launch worker for task 69 ERROR AvroFileFormat: Exception while opening DataFileReader java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63) at org.apache.spark.sql.avro.AvroFileFormat.$anonfun$buildReader$1(AvroFileFormat.scala:107) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:147) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:132) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter$lzycompute(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:145) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:27.287 Executor task launch worker for task 69 WARN FileScanRDD: Skipped the rest of the content in the corrupted file: path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b88cb402-6e73-4c7f-954c-36c4c2a599d1/corrupt.avro, range: 0-7, partition values: [empty row] java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:63) at org.apache.spark.sql.avro.AvroFileFormat.$anonfun$buildReader$1(AvroFileFormat.scala:107) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:147) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:132) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter$lzycompute(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.internalIter(FileScanRDD.scala:141) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:145) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20/07/01 10:38:27.288 Executor task launch worker for task 69 INFO Executor: Finished task 1.0 in stage 39.0 (TID 69). 1357 bytes result sent to driver 20/07/01 10:38:27.289 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 39.0 (TID 69) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:27.289 Executor task launch worker for task 68 INFO Executor: Finished task 0.0 in stage 39.0 (TID 68). 1796 bytes result sent to driver 20/07/01 10:38:27.289 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 39.0 (TID 68) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:27.289 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 39.0, whose tasks have all completed, from pool 20/07/01 10:38:27.290 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 39 (apply at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:27.290 dag-scheduler-event-loop INFO DAGScheduler: Job 33 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.290 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 39: Stage finished 20/07/01 10:38:27.290 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 33 finished: apply at OutcomeOf.scala:85, took 0.014834 s 20/07/01 10:38:27.296 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled' ===== 20/07/01 10:38:27.298 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Throws IOException on reading corrupt Avro file if flag IGNORE_CORRUPT_FILES disabled' ===== 20/07/01 10:38:27.306 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:27.310 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Throws IOException on reading corrupt Avro file if flag IGNORE_CORRUPT_FILES disabled' ===== 20/07/01 10:38:27.311 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Date field type' ===== 20/07/01 10:38:27.373 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.374 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.395 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:27.395 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:27.395 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:27.396 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:27.415 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 8.968908 ms 20/07/01 10:38:27.499 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.500 dag-scheduler-event-loop INFO DAGScheduler: Got job 34 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:27.500 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 40 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.500 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:27.500 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:27.501 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 40 (MapPartitionsRDD[142] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.592 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_70 stored as values in memory (estimated size 176.5 KiB, free 2.1 GiB) 20/07/01 10:38:27.594 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_70_piece0 stored as bytes in memory (estimated size 62.8 KiB, free 2.1 GiB) 20/07/01 10:38:27.595 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_70_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.8 KiB, free: 2.1 GiB) 20/07/01 10:38:27.595 dag-scheduler-event-loop INFO SparkContext: Created broadcast 70 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.595 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 40 (MapPartitionsRDD[142] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:27.595 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 40.0 with 2 tasks resource profile 0 20/07/01 10:38:27.596 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 40.0 (TID 70, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7529 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.597 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 40.0 (TID 71, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7641 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.597 Executor task launch worker for task 70 INFO Executor: Running task 0.0 in stage 40.0 (TID 70) 20/07/01 10:38:27.598 Executor task launch worker for task 71 INFO Executor: Running task 1.0 in stage 40.0 (TID 71) 20/07/01 10:38:27.616 Executor task launch worker for task 70 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:27.617 Executor task launch worker for task 70 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:27.617 Executor task launch worker for task 70 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:27.619 Executor task launch worker for task 71 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:27.619 Executor task launch worker for task 71 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:27.619 Executor task launch worker for task 71 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:27.648 Executor task launch worker for task 71 INFO CodeGenerator: Code generated in 25.505815 ms 20/07/01 10:38:27.654 Executor task launch worker for task 70 INFO FileOutputCommitter: Saved output of task 'attempt_20200701173827_0040_m_000000_70' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9 20/07/01 10:38:27.654 Executor task launch worker for task 70 INFO SparkHadoopMapRedUtil: attempt_20200701173827_0040_m_000000_70: Committed 20/07/01 10:38:27.655 Executor task launch worker for task 70 INFO Executor: Finished task 0.0 in stage 40.0 (TID 70). 2184 bytes result sent to driver 20/07/01 10:38:27.656 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 40.0 (TID 70) in 60 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:27.680 Executor task launch worker for task 71 INFO FileOutputCommitter: Saved output of task 'attempt_20200701173827_0040_m_000001_71' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9 20/07/01 10:38:27.680 Executor task launch worker for task 71 INFO SparkHadoopMapRedUtil: attempt_20200701173827_0040_m_000001_71: Committed 20/07/01 10:38:27.681 Executor task launch worker for task 71 INFO Executor: Finished task 1.0 in stage 40.0 (TID 71). 2184 bytes result sent to driver 20/07/01 10:38:27.681 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 40.0 (TID 71) in 85 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:27.681 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 40.0, whose tasks have all completed, from pool 20/07/01 10:38:27.682 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 40 (apply at OutcomeOf.scala:85) finished in 0.181 s 20/07/01 10:38:27.682 dag-scheduler-event-loop INFO DAGScheduler: Job 34 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.682 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 40: Stage finished 20/07/01 10:38:27.684 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 34 finished: apply at OutcomeOf.scala:85, took 0.184859 s 20/07/01 10:38:27.742 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 761cdb88-7acd-496f-b991-4c8f6545001d committed. 20/07/01 10:38:27.743 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 761cdb88-7acd-496f-b991-4c8f6545001d. 20/07/01 10:38:27.751 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:27.761 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.761 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.784 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.784 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.784 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.784 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:27.795 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_71 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:27.826 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_71_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:27.826 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_71_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:27.827 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 71 from apply at OutcomeOf.scala:85 20/07/01 10:38:27.828 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:27.840 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.841 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 148 (apply at OutcomeOf.scala:85) as input to shuffle 6 20/07/01 10:38:27.841 dag-scheduler-event-loop INFO DAGScheduler: Got job 35 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:27.841 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 42 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.841 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 41) 20/07/01 10:38:27.841 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 41) 20/07/01 10:38:27.842 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 41 (MapPartitionsRDD[148] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.843 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_72 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:27.845 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_72_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:27.845 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_72_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:27.845 dag-scheduler-event-loop INFO SparkContext: Created broadcast 72 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.845 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 41 (MapPartitionsRDD[148] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:27.846 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 41.0 with 2 tasks resource profile 0 20/07/01 10:38:27.846 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 41.0 (TID 72, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.847 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 41.0 (TID 73, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.847 Executor task launch worker for task 73 INFO Executor: Running task 1.0 in stage 41.0 (TID 73) 20/07/01 10:38:27.847 Executor task launch worker for task 72 INFO Executor: Running task 0.0 in stage 41.0 (TID 72) 20/07/01 10:38:27.850 Executor task launch worker for task 73 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00000-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-276, partition values: [empty row] 20/07/01 10:38:27.851 Executor task launch worker for task 72 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00001-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-288, partition values: [empty row] 20/07/01 10:38:27.855 Executor task launch worker for task 72 INFO Executor: Finished task 0.0 in stage 41.0 (TID 72). 1935 bytes result sent to driver 20/07/01 10:38:27.855 Executor task launch worker for task 73 INFO Executor: Finished task 1.0 in stage 41.0 (TID 73). 1935 bytes result sent to driver 20/07/01 10:38:27.856 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 41.0 (TID 72) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:27.857 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 41.0 (TID 73) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:27.857 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 41.0, whose tasks have all completed, from pool 20/07/01 10:38:27.857 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 41 (apply at OutcomeOf.scala:85) finished in 0.015 s 20/07/01 10:38:27.857 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:27.857 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:27.857 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 42) 20/07/01 10:38:27.857 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:27.858 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 42 (MapPartitionsRDD[151] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.859 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_73 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:27.861 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_73_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:27.861 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_73_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:27.861 dag-scheduler-event-loop INFO SparkContext: Created broadcast 73 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.862 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 42 (MapPartitionsRDD[151] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:27.862 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 42.0 with 1 tasks resource profile 0 20/07/01 10:38:27.863 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 42.0 (TID 74, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.863 Executor task launch worker for task 74 INFO Executor: Running task 0.0 in stage 42.0 (TID 74) 20/07/01 10:38:27.866 Executor task launch worker for task 74 INFO ShuffleBlockFetcherIterator: Getting 2 (120.0 B) non-empty blocks including 2 (120.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:27.866 Executor task launch worker for task 74 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:27.868 Executor task launch worker for task 74 INFO Executor: Finished task 0.0 in stage 42.0 (TID 74). 2648 bytes result sent to driver 20/07/01 10:38:27.869 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 42.0 (TID 74) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:27.869 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 42.0, whose tasks have all completed, from pool 20/07/01 10:38:27.869 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 42 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:27.870 dag-scheduler-event-loop INFO DAGScheduler: Job 35 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.870 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 42: Stage finished 20/07/01 10:38:27.870 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 35 finished: apply at OutcomeOf.scala:85, took 0.029636 s 20/07/01 10:38:27.876 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:27.877 dag-scheduler-event-loop INFO DAGScheduler: Got job 36 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:27.877 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 43 (apply at OutcomeOf.scala:85) 20/07/01 10:38:27.877 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:27.877 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:27.877 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 43 (ParallelCollectionRDD[140] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:27.879 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_74 stored as values in memory (estimated size 2024.0 B, free 2.1 GiB) 20/07/01 10:38:27.880 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_74_piece0 stored as bytes in memory (estimated size 1276.0 B, free 2.1 GiB) 20/07/01 10:38:27.880 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_74_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:27.880 dag-scheduler-event-loop INFO SparkContext: Created broadcast 74 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:27.881 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 43 (ParallelCollectionRDD[140] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:27.881 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 43.0 with 2 tasks resource profile 0 20/07/01 10:38:27.882 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 43.0 (TID 75, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7529 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.882 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 43.0 (TID 76, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7641 bytes) taskResourceAssignments Map() 20/07/01 10:38:27.882 Executor task launch worker for task 76 INFO Executor: Running task 1.0 in stage 43.0 (TID 76) 20/07/01 10:38:27.882 Executor task launch worker for task 75 INFO Executor: Running task 0.0 in stage 43.0 (TID 75) 20/07/01 10:38:27.885 Executor task launch worker for task 76 INFO Executor: Finished task 1.0 in stage 43.0 (TID 76). 832 bytes result sent to driver 20/07/01 10:38:27.885 Executor task launch worker for task 75 INFO Executor: Finished task 0.0 in stage 43.0 (TID 75). 832 bytes result sent to driver 20/07/01 10:38:27.885 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 43.0 (TID 76) in 3 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:27.885 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 43.0 (TID 75) in 3 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:27.885 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 43.0, whose tasks have all completed, from pool 20/07/01 10:38:27.886 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 43 (apply at OutcomeOf.scala:85) finished in 0.007 s 20/07/01 10:38:27.886 dag-scheduler-event-loop INFO DAGScheduler: Job 36 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:27.886 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 43: Stage finished 20/07/01 10:38:27.886 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 36 finished: apply at OutcomeOf.scala:85, took 0.009321 s 20/07/01 10:38:27.891 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:27.899 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.899 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.908 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.909 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.909 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.909 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:27.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:27.925 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:27.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:27.956 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:27.957 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:27.959 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_75 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:27.991 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_75_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:27.992 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_75_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:27.993 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 75 from apply at OutcomeOf.scala:85 20/07/01 10:38:27.993 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:28.003 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.004 dag-scheduler-event-loop INFO DAGScheduler: Got job 37 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:28.004 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 44 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.004 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:28.004 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:28.004 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 44 (MapPartitionsRDD[156] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.008 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_76 stored as values in memory (estimated size 11.7 KiB, free 2.1 GiB) 20/07/01 10:38:28.010 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_76_piece0 stored as bytes in memory (estimated size 6.1 KiB, free 2.1 GiB) 20/07/01 10:38:28.010 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_76_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.010 dag-scheduler-event-loop INFO SparkContext: Created broadcast 76 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.010 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 44 (MapPartitionsRDD[156] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:28.011 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 44.0 with 2 tasks resource profile 0 20/07/01 10:38:28.011 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 44.0 (TID 77, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.012 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 44.0 (TID 78, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.012 Executor task launch worker for task 78 INFO Executor: Running task 1.0 in stage 44.0 (TID 78) 20/07/01 10:38:28.012 Executor task launch worker for task 77 INFO Executor: Running task 0.0 in stage 44.0 (TID 77) 20/07/01 10:38:28.016 Executor task launch worker for task 78 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00000-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-276, partition values: [empty row] 20/07/01 10:38:28.016 Executor task launch worker for task 77 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00001-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-288, partition values: [empty row] 20/07/01 10:38:28.019 Executor task launch worker for task 77 INFO Executor: Finished task 0.0 in stage 44.0 (TID 77). 1554 bytes result sent to driver 20/07/01 10:38:28.019 Executor task launch worker for task 78 INFO Executor: Finished task 1.0 in stage 44.0 (TID 78). 1554 bytes result sent to driver 20/07/01 10:38:28.020 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 44.0 (TID 77) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:28.020 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 44.0 (TID 78) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:28.020 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 44.0, whose tasks have all completed, from pool 20/07/01 10:38:28.020 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 44 (apply at OutcomeOf.scala:85) finished in 0.014 s 20/07/01 10:38:28.020 dag-scheduler-event-loop INFO DAGScheduler: Job 37 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:28.020 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 44: Stage finished 20/07/01 10:38:28.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 37 finished: apply at OutcomeOf.scala:85, took 0.017269 s 20/07/01 10:38:28.027 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_77 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:28.061 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_77_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:28.061 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_77_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.062 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 77 from apply at OutcomeOf.scala:85 20/07/01 10:38:28.062 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:28.069 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.069 dag-scheduler-event-loop INFO DAGScheduler: Got job 38 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:28.069 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 45 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.069 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:28.070 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:28.070 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 45 (MapPartitionsRDD[159] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.071 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_78 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:28.072 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_78_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:28.072 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_78_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.073 dag-scheduler-event-loop INFO SparkContext: Created broadcast 78 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.073 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 45 (MapPartitionsRDD[159] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:28.073 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 45.0 with 2 tasks resource profile 0 20/07/01 10:38:28.074 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 45.0 (TID 79, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.074 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 45.0 (TID 80, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7866 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.074 Executor task launch worker for task 80 INFO Executor: Running task 1.0 in stage 45.0 (TID 80) 20/07/01 10:38:28.075 Executor task launch worker for task 79 INFO Executor: Running task 0.0 in stage 45.0 (TID 79) 20/07/01 10:38:28.077 Executor task launch worker for task 79 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00001-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-288, partition values: [empty row] 20/07/01 10:38:28.077 Executor task launch worker for task 80 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-bd790f87-698a-48d2-bbbd-848a0f6f5fb9/part-00000-a5b74ba5-c267-440f-9022-250a67b64799-c000.avro, range: 0-276, partition values: [empty row] 20/07/01 10:38:28.080 Executor task launch worker for task 79 INFO Executor: Finished task 0.0 in stage 45.0 (TID 79). 1425 bytes result sent to driver 20/07/01 10:38:28.080 Executor task launch worker for task 80 INFO Executor: Finished task 1.0 in stage 45.0 (TID 80). 1411 bytes result sent to driver 20/07/01 10:38:28.080 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 45.0 (TID 79) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:28.081 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 45.0 (TID 80) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:28.081 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 45.0, whose tasks have all completed, from pool 20/07/01 10:38:28.081 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 45 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:28.081 dag-scheduler-event-loop INFO DAGScheduler: Job 38 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:28.081 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 45: Stage finished 20/07/01 10:38:28.081 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 38 finished: apply at OutcomeOf.scala:85, took 0.012548 s 20/07/01 10:38:28.086 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Date field type' ===== 20/07/01 10:38:28.087 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'Array data types' ===== 20/07/01 10:38:28.109 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.109 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.125 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:28.126 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:28.126 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:28.126 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:28.170 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 30.031909 ms 20/07/01 10:38:28.248 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.248 dag-scheduler-event-loop INFO DAGScheduler: Got job 39 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:28.248 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 46 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.248 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:28.249 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:28.249 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 46 (MapPartitionsRDD[162] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.364 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_79 stored as values in memory (estimated size 206.2 KiB, free 2.1 GiB) 20/07/01 10:38:28.367 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_79_piece0 stored as bytes in memory (estimated size 69.8 KiB, free 2.1 GiB) 20/07/01 10:38:28.367 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_79_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 69.8 KiB, free: 2.1 GiB) 20/07/01 10:38:28.367 dag-scheduler-event-loop INFO SparkContext: Created broadcast 79 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.368 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 46 (MapPartitionsRDD[162] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:28.368 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 46.0 with 2 tasks resource profile 0 20/07/01 10:38:28.369 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 46.0 (TID 81, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7357 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.375 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 46.0 (TID 82, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8651 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.375 Executor task launch worker for task 81 INFO Executor: Running task 0.0 in stage 46.0 (TID 81) 20/07/01 10:38:28.376 Executor task launch worker for task 82 INFO Executor: Running task 1.0 in stage 46.0 (TID 82) 20/07/01 10:38:28.457 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_50_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.458 Executor task launch worker for task 81 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:28.458 Executor task launch worker for task 81 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:28.459 Executor task launch worker for task 82 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:28.459 Executor task launch worker for task 82 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:28.459 Executor task launch worker for task 81 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:28.459 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_31_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.459 Executor task launch worker for task 82 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:28.461 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_65_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:28.463 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_49_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.465 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_28_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.466 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_76_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.468 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_59_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 63.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.469 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_51_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.470 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_57_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.472 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_55_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.473 Executor task launch worker for task 81 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103828_0046_m_000000_81' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321 20/07/01 10:38:28.474 Executor task launch worker for task 81 INFO SparkHadoopMapRedUtil: attempt_20200701103828_0046_m_000000_81: Committed 20/07/01 10:38:28.474 Executor task launch worker for task 81 INFO Executor: Finished task 0.0 in stage 46.0 (TID 81). 2184 bytes result sent to driver 20/07/01 10:38:28.474 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_29_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 7.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.475 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 46.0 (TID 81) in 107 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:28.476 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_66_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.478 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_48_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.479 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_54_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 66.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.480 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_56_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:28.481 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_58_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:28.482 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_78_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.483 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_35_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:28.485 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_67_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.2 KiB, free: 2.1 GiB) 20/07/01 10:38:28.487 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_46_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.489 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_70_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.8 KiB, free: 2.1 GiB) 20/07/01 10:38:28.490 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_32_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:28.491 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_74_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:28.492 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_36_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.494 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_53_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.495 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_44_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.495 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_43_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.496 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_27_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.497 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_64_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.498 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_61_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:28.501 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_33_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.502 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_47_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.3 KiB, free: 2.1 GiB) 20/07/01 10:38:28.503 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_45_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.503 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_40_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.504 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_71_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.505 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_72_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:28.506 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_73_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.507 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_34_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.507 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_77_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.508 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_38_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.509 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_68_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:28.510 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_41_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.510 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_37_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.511 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_63_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:28.511 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_60_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.512 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_69_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:28.513 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_75_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.513 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_39_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.514 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_62_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.515 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_42_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.516 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_30_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.516 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_52_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.560 Executor task launch worker for task 82 INFO CodeGenerator: Code generated in 79.913657 ms 20/07/01 10:38:28.578 Executor task launch worker for task 82 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103828_0046_m_000001_82' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321 20/07/01 10:38:28.578 Executor task launch worker for task 82 INFO SparkHadoopMapRedUtil: attempt_20200701103828_0046_m_000001_82: Committed 20/07/01 10:38:28.579 Executor task launch worker for task 82 INFO Executor: Finished task 1.0 in stage 46.0 (TID 82). 2227 bytes result sent to driver 20/07/01 10:38:28.580 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 46.0 (TID 82) in 211 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:28.580 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 46.0, whose tasks have all completed, from pool 20/07/01 10:38:28.580 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 46 (apply at OutcomeOf.scala:85) finished in 0.331 s 20/07/01 10:38:28.580 dag-scheduler-event-loop INFO DAGScheduler: Job 39 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:28.580 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 46: Stage finished 20/07/01 10:38:28.580 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 39 finished: apply at OutcomeOf.scala:85, took 0.332696 s 20/07/01 10:38:28.588 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 72a1c084-aa77-4db0-bf98-5058f29b6ae6 committed. 20/07/01 10:38:28.588 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 72a1c084-aa77-4db0-bf98-5058f29b6ae6. 20/07/01 10:38:28.596 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:28.618 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.618 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.636 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:28.637 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:28.637 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:28.637 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct<> 20/07/01 10:38:28.645 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_80 stored as values in memory (estimated size 307.7 KiB, free 2.1 GiB) 20/07/01 10:38:28.680 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_80_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:28.681 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_80_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.681 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 80 from apply at OutcomeOf.scala:85 20/07/01 10:38:28.682 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:28.693 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.694 dag-scheduler-event-loop INFO DAGScheduler: Registering RDD 168 (apply at OutcomeOf.scala:85) as input to shuffle 7 20/07/01 10:38:28.695 dag-scheduler-event-loop INFO DAGScheduler: Got job 40 (apply at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:28.695 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 48 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.695 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 47) 20/07/01 10:38:28.695 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List(ShuffleMapStage 47) 20/07/01 10:38:28.695 dag-scheduler-event-loop INFO DAGScheduler: Submitting ShuffleMapStage 47 (MapPartitionsRDD[168] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.697 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_81 stored as values in memory (estimated size 13.3 KiB, free 2.1 GiB) 20/07/01 10:38:28.699 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_81_piece0 stored as bytes in memory (estimated size 6.7 KiB, free 2.1 GiB) 20/07/01 10:38:28.699 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_81_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:28.700 dag-scheduler-event-loop INFO SparkContext: Created broadcast 81 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.700 dag-scheduler-event-loop INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 47 (MapPartitionsRDD[168] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 20/07/01 10:38:28.700 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 47.0 with 4 tasks resource profile 0 20/07/01 10:38:28.701 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 47.0 (TID 83, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.701 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 47.0 (TID 84, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.702 Executor task launch worker for task 83 INFO Executor: Running task 0.0 in stage 47.0 (TID 83) 20/07/01 10:38:28.702 Executor task launch worker for task 84 INFO Executor: Running task 1.0 in stage 47.0 (TID 84) 20/07/01 10:38:28.706 Executor task launch worker for task 84 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321/part-00001-29bb2070-5579-497c-8b38-7797884bfcb8-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:28.706 Executor task launch worker for task 83 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321/part-00000-29bb2070-5579-497c-8b38-7797884bfcb8-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:28.710 Executor task launch worker for task 84 INFO Executor: Finished task 1.0 in stage 47.0 (TID 84). 1892 bytes result sent to driver 20/07/01 10:38:28.710 Executor task launch worker for task 83 INFO Executor: Finished task 0.0 in stage 47.0 (TID 83). 1892 bytes result sent to driver 20/07/01 10:38:28.711 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 2.0 in stage 47.0 (TID 85, amp-jenkins-worker-04.amp, executor driver, partition 2, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.711 Executor task launch worker for task 85 INFO Executor: Running task 2.0 in stage 47.0 (TID 85) 20/07/01 10:38:28.711 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 47.0 (TID 84) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/4) 20/07/01 10:38:28.712 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 3.0 in stage 47.0 (TID 86, amp-jenkins-worker-04.amp, executor driver, partition 3, PROCESS_LOCAL, 7855 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.712 Executor task launch worker for task 86 INFO Executor: Running task 3.0 in stage 47.0 (TID 86) 20/07/01 10:38:28.712 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 47.0 (TID 83) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (2/4) 20/07/01 10:38:28.714 Executor task launch worker for task 85 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321/part-00001-29bb2070-5579-497c-8b38-7797884bfcb8-c000.avro, range: 1024-1530, partition values: [empty row] 20/07/01 10:38:28.714 Executor task launch worker for task 86 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-b51813c8-13dc-4e79-8f7e-29e653a84321/part-00000-29bb2070-5579-497c-8b38-7797884bfcb8-c000.avro, range: 1024-1343, partition values: [empty row] 20/07/01 10:38:28.718 Executor task launch worker for task 86 INFO Executor: Finished task 3.0 in stage 47.0 (TID 86). 1892 bytes result sent to driver 20/07/01 10:38:28.718 Executor task launch worker for task 85 INFO Executor: Finished task 2.0 in stage 47.0 (TID 85). 1935 bytes result sent to driver 20/07/01 10:38:28.719 task-result-getter-1 INFO TaskSetManager: Finished task 3.0 in stage 47.0 (TID 86) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (3/4) 20/07/01 10:38:28.719 task-result-getter-2 INFO TaskSetManager: Finished task 2.0 in stage 47.0 (TID 85) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (4/4) 20/07/01 10:38:28.719 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 47.0, whose tasks have all completed, from pool 20/07/01 10:38:28.719 dag-scheduler-event-loop INFO DAGScheduler: ShuffleMapStage 47 (apply at OutcomeOf.scala:85) finished in 0.023 s 20/07/01 10:38:28.719 dag-scheduler-event-loop INFO DAGScheduler: looking for newly runnable stages 20/07/01 10:38:28.720 dag-scheduler-event-loop INFO DAGScheduler: running: Set() 20/07/01 10:38:28.720 dag-scheduler-event-loop INFO DAGScheduler: waiting: Set(ResultStage 48) 20/07/01 10:38:28.720 dag-scheduler-event-loop INFO DAGScheduler: failed: Set() 20/07/01 10:38:28.720 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 48 (MapPartitionsRDD[171] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.721 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_82 stored as values in memory (estimated size 10.1 KiB, free 2.1 GiB) 20/07/01 10:38:28.723 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_82_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 2.1 GiB) 20/07/01 10:38:28.723 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_82_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.723 dag-scheduler-event-loop INFO SparkContext: Created broadcast 82 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.724 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 48 (MapPartitionsRDD[171] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:28.724 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 48.0 with 1 tasks resource profile 0 20/07/01 10:38:28.725 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 48.0 (TID 87, amp-jenkins-worker-04.amp, executor driver, partition 0, NODE_LOCAL, 7325 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.725 Executor task launch worker for task 87 INFO Executor: Running task 0.0 in stage 48.0 (TID 87) 20/07/01 10:38:28.727 Executor task launch worker for task 87 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 4 (240.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks 20/07/01 10:38:28.727 Executor task launch worker for task 87 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 20/07/01 10:38:28.729 Executor task launch worker for task 87 INFO Executor: Finished task 0.0 in stage 48.0 (TID 87). 2648 bytes result sent to driver 20/07/01 10:38:28.730 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 48.0 (TID 87) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:28.730 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 48.0, whose tasks have all completed, from pool 20/07/01 10:38:28.730 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 48 (apply at OutcomeOf.scala:85) finished in 0.009 s 20/07/01 10:38:28.731 dag-scheduler-event-loop INFO DAGScheduler: Job 40 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:28.731 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 48: Stage finished 20/07/01 10:38:28.731 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 40 finished: apply at OutcomeOf.scala:85, took 0.037225 s 20/07/01 10:38:28.738 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.739 dag-scheduler-event-loop INFO DAGScheduler: Got job 41 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:28.739 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 49 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.739 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:28.739 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:28.739 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 49 (ParallelCollectionRDD[160] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.740 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_83 stored as values in memory (estimated size 2024.0 B, free 2.1 GiB) 20/07/01 10:38:28.742 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_83_piece0 stored as bytes in memory (estimated size 1276.0 B, free 2.1 GiB) 20/07/01 10:38:28.742 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_83_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:28.742 dag-scheduler-event-loop INFO SparkContext: Created broadcast 83 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.743 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 49 (ParallelCollectionRDD[160] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:28.743 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 49.0 with 2 tasks resource profile 0 20/07/01 10:38:28.744 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 49.0 (TID 88, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7357 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.745 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 49.0 (TID 89, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8651 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.745 Executor task launch worker for task 88 INFO Executor: Running task 0.0 in stage 49.0 (TID 88) 20/07/01 10:38:28.745 Executor task launch worker for task 89 INFO Executor: Running task 1.0 in stage 49.0 (TID 89) 20/07/01 10:38:28.747 Executor task launch worker for task 89 INFO Executor: Finished task 1.0 in stage 49.0 (TID 89). 832 bytes result sent to driver 20/07/01 10:38:28.747 Executor task launch worker for task 88 INFO Executor: Finished task 0.0 in stage 49.0 (TID 88). 832 bytes result sent to driver 20/07/01 10:38:28.748 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 49.0 (TID 89) in 4 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:28.748 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 49.0 (TID 88) in 5 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:28.748 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 49.0, whose tasks have all completed, from pool 20/07/01 10:38:28.749 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 49 (apply at OutcomeOf.scala:85) finished in 0.008 s 20/07/01 10:38:28.749 dag-scheduler-event-loop INFO DAGScheduler: Job 41 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:28.749 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 49: Stage finished 20/07/01 10:38:28.749 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 41 finished: apply at OutcomeOf.scala:85, took 0.010626 s 20/07/01 10:38:28.752 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'Array data types' ===== 20/07/01 10:38:28.754 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'write with compression - sql configs' ===== 20/07/01 10:38:28.760 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:28.778 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.779 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:28.789 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:28.790 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:28.790 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:28.790 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:28.794 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:28.794 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:28.795 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:28.805 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_84 stored as values in memory (estimated size 307.9 KiB, free 2.1 GiB) 20/07/01 10:38:28.840 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_84_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:28.841 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_84_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:28.841 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 84 from apply at OutcomeOf.scala:85 20/07/01 10:38:28.842 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:28.912 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:28.913 dag-scheduler-event-loop INFO DAGScheduler: Got job 42 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:28.913 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 50 (apply at OutcomeOf.scala:85) 20/07/01 10:38:28.913 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:28.913 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:28.913 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 50 (MapPartitionsRDD[173] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:28.982 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_85 stored as values in memory (estimated size 175.5 KiB, free 2.1 GiB) 20/07/01 10:38:28.985 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_85_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:28.985 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_85_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:28.986 dag-scheduler-event-loop INFO SparkContext: Created broadcast 85 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:28.986 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 50 (MapPartitionsRDD[173] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:28.986 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 50.0 with 2 tasks resource profile 0 20/07/01 10:38:28.987 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 50.0 (TID 90, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.987 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 50.0 (TID 91, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:28.987 Executor task launch worker for task 90 INFO Executor: Running task 0.0 in stage 50.0 (TID 90) 20/07/01 10:38:28.987 Executor task launch worker for task 91 INFO Executor: Running task 1.0 in stage 50.0 (TID 91) 20/07/01 10:38:28.998 Executor task launch worker for task 90 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:28.998 Executor task launch worker for task 90 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:28.998 Executor task launch worker for task 90 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.005 Executor task launch worker for task 91 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.005 Executor task launch worker for task 91 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.005 Executor task launch worker for task 91 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.005 Executor task launch worker for task 91 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:29.016 Executor task launch worker for task 90 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:29.034 Executor task launch worker for task 91 INFO CodeGenerator: Code generated in 23.057371 ms 20/07/01 10:38:29.049 Executor task launch worker for task 91 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103828_0050_m_000001_91 20/07/01 10:38:29.050 Executor task launch worker for task 91 INFO Executor: Finished task 1.0 in stage 50.0 (TID 91). 2069 bytes result sent to driver 20/07/01 10:38:29.051 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 50.0 (TID 91) in 64 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:29.057 Executor task launch worker for task 90 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103828_0050_m_000000_90' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c7c32462-37fa-44fd-8ba1-6c710169f494/uncompress 20/07/01 10:38:29.057 Executor task launch worker for task 90 INFO SparkHadoopMapRedUtil: attempt_20200701103828_0050_m_000000_90: Committed 20/07/01 10:38:29.058 Executor task launch worker for task 90 INFO Executor: Finished task 0.0 in stage 50.0 (TID 90). 2198 bytes result sent to driver 20/07/01 10:38:29.058 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 50.0 (TID 90) in 71 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:29.058 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 50.0, whose tasks have all completed, from pool 20/07/01 10:38:29.059 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 50 (apply at OutcomeOf.scala:85) finished in 0.146 s 20/07/01 10:38:29.059 dag-scheduler-event-loop INFO DAGScheduler: Job 42 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:29.059 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 50: Stage finished 20/07/01 10:38:29.059 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 42 finished: apply at OutcomeOf.scala:85, took 0.146986 s 20/07/01 10:38:29.065 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 8b816dde-e344-4088-b14c-ce741fd140a7 committed. 20/07/01 10:38:29.066 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 8b816dde-e344-4088-b14c-ce741fd140a7. 20/07/01 10:38:29.077 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.077 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.088 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:29.088 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:29.089 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:29.089 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:29.093 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the bzip2 codec 20/07/01 10:38:29.093 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.093 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.094 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.102 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_86 stored as values in memory (estimated size 307.9 KiB, free 2.1 GiB) 20/07/01 10:38:29.142 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_86_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:29.142 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_86_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:29.143 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 86 from apply at OutcomeOf.scala:85 20/07/01 10:38:29.143 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:29.213 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:29.214 dag-scheduler-event-loop INFO DAGScheduler: Got job 43 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:29.214 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 51 (apply at OutcomeOf.scala:85) 20/07/01 10:38:29.214 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:29.214 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:29.215 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 51 (MapPartitionsRDD[177] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:29.277 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_87 stored as values in memory (estimated size 175.5 KiB, free 2.1 GiB) 20/07/01 10:38:29.280 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_87_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:29.280 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_87_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:29.280 dag-scheduler-event-loop INFO SparkContext: Created broadcast 87 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:29.281 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 51 (MapPartitionsRDD[177] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:29.281 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 51.0 with 2 tasks resource profile 0 20/07/01 10:38:29.282 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 51.0 (TID 92, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.282 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 51.0 (TID 93, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.282 Executor task launch worker for task 93 INFO Executor: Running task 1.0 in stage 51.0 (TID 93) 20/07/01 10:38:29.282 Executor task launch worker for task 92 INFO Executor: Running task 0.0 in stage 51.0 (TID 92) 20/07/01 10:38:29.294 Executor task launch worker for task 92 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.294 Executor task launch worker for task 93 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.294 Executor task launch worker for task 92 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.294 Executor task launch worker for task 93 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.294 Executor task launch worker for task 92 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.294 Executor task launch worker for task 93 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.294 Executor task launch worker for task 93 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:29.300 Executor task launch worker for task 93 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103829_0051_m_000001_93 20/07/01 10:38:29.301 Executor task launch worker for task 93 INFO Executor: Finished task 1.0 in stage 51.0 (TID 93). 2069 bytes result sent to driver 20/07/01 10:38:29.301 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 51.0 (TID 93) in 19 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:29.305 Executor task launch worker for task 92 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:29.327 Executor task launch worker for task 92 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103829_0051_m_000000_92' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c7c32462-37fa-44fd-8ba1-6c710169f494/bzip2 20/07/01 10:38:29.327 Executor task launch worker for task 92 INFO SparkHadoopMapRedUtil: attempt_20200701103829_0051_m_000000_92: Committed 20/07/01 10:38:29.327 Executor task launch worker for task 92 INFO Executor: Finished task 0.0 in stage 51.0 (TID 92). 2198 bytes result sent to driver 20/07/01 10:38:29.328 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 51.0 (TID 92) in 47 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:29.328 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 51.0, whose tasks have all completed, from pool 20/07/01 10:38:29.329 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 51 (apply at OutcomeOf.scala:85) finished in 0.114 s 20/07/01 10:38:29.329 dag-scheduler-event-loop INFO DAGScheduler: Job 43 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:29.329 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 51: Stage finished 20/07/01 10:38:29.329 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 43 finished: apply at OutcomeOf.scala:85, took 0.115533 s 20/07/01 10:38:29.334 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 6dee09ed-90b9-46f3-9776-5e362e51ca05 committed. 20/07/01 10:38:29.335 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 6dee09ed-90b9-46f3-9776-5e362e51ca05. 20/07/01 10:38:29.345 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.345 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.357 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:29.357 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:29.357 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:29.358 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:29.361 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the xz codec 20/07/01 10:38:29.362 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.362 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.362 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.371 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_88 stored as values in memory (estimated size 307.8 KiB, free 2.1 GiB) 20/07/01 10:38:29.409 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_88_piece0 stored as bytes in memory (estimated size 28.0 KiB, free 2.1 GiB) 20/07/01 10:38:29.410 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_88_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:29.410 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 88 from apply at OutcomeOf.scala:85 20/07/01 10:38:29.411 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:29.482 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:29.483 dag-scheduler-event-loop INFO DAGScheduler: Got job 44 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:29.483 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 52 (apply at OutcomeOf.scala:85) 20/07/01 10:38:29.483 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:29.483 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:29.483 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 52 (MapPartitionsRDD[181] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:29.544 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_89 stored as values in memory (estimated size 175.5 KiB, free 2.1 GiB) 20/07/01 10:38:29.546 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_89_piece0 stored as bytes in memory (estimated size 62.4 KiB, free 2.1 GiB) 20/07/01 10:38:29.546 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_89_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.4 KiB, free: 2.1 GiB) 20/07/01 10:38:29.547 dag-scheduler-event-loop INFO SparkContext: Created broadcast 89 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:29.547 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 52 (MapPartitionsRDD[181] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:29.547 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 52.0 with 2 tasks resource profile 0 20/07/01 10:38:29.548 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 52.0 (TID 94, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.548 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 52.0 (TID 95, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.548 Executor task launch worker for task 95 INFO Executor: Running task 1.0 in stage 52.0 (TID 95) 20/07/01 10:38:29.548 Executor task launch worker for task 94 INFO Executor: Running task 0.0 in stage 52.0 (TID 94) 20/07/01 10:38:29.562 Executor task launch worker for task 94 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.562 Executor task launch worker for task 94 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.562 Executor task launch worker for task 94 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.563 Executor task launch worker for task 95 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.563 Executor task launch worker for task 95 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.563 Executor task launch worker for task 95 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.563 Executor task launch worker for task 95 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:29.568 Executor task launch worker for task 95 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103829_0052_m_000001_95 20/07/01 10:38:29.569 Executor task launch worker for task 95 INFO Executor: Finished task 1.0 in stage 52.0 (TID 95). 2069 bytes result sent to driver 20/07/01 10:38:29.570 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 52.0 (TID 95) in 22 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:29.576 Executor task launch worker for task 94 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:29.633 Executor task launch worker for task 94 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103829_0052_m_000000_94' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c7c32462-37fa-44fd-8ba1-6c710169f494/xz 20/07/01 10:38:29.633 Executor task launch worker for task 94 INFO SparkHadoopMapRedUtil: attempt_20200701103829_0052_m_000000_94: Committed 20/07/01 10:38:29.633 Executor task launch worker for task 94 INFO Executor: Finished task 0.0 in stage 52.0 (TID 94). 2198 bytes result sent to driver 20/07/01 10:38:29.634 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 52.0 (TID 94) in 86 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:29.634 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 52.0, whose tasks have all completed, from pool 20/07/01 10:38:29.635 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 52 (apply at OutcomeOf.scala:85) finished in 0.150 s 20/07/01 10:38:29.635 dag-scheduler-event-loop INFO DAGScheduler: Job 44 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:29.635 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 52: Stage finished 20/07/01 10:38:29.635 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 44 finished: apply at OutcomeOf.scala:85, took 0.152707 s 20/07/01 10:38:29.640 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job c8627d5f-c4d7-426f-a6ac-636ec7f13d8e committed. 20/07/01 10:38:29.640 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job c8627d5f-c4d7-426f-a6ac-636ec7f13d8e. 20/07/01 10:38:29.651 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.651 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.661 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:29.662 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:29.662 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:29.662 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:29.666 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the deflate codec 20/07/01 10:38:29.667 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Avro compression level 9 will be used for deflate codec. 20/07/01 10:38:29.668 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.668 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.668 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.677 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_90 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:29.708 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_90_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:29.708 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_90_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:29.709 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 90 from apply at OutcomeOf.scala:85 20/07/01 10:38:29.709 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:29.771 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:29.772 dag-scheduler-event-loop INFO DAGScheduler: Got job 45 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:29.772 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 53 (apply at OutcomeOf.scala:85) 20/07/01 10:38:29.772 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:29.772 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:29.772 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 53 (MapPartitionsRDD[185] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:29.842 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_91 stored as values in memory (estimated size 175.8 KiB, free 2.1 GiB) 20/07/01 10:38:29.844 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_91_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:29.845 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_91_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:29.845 dag-scheduler-event-loop INFO SparkContext: Created broadcast 91 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:29.845 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 53 (MapPartitionsRDD[185] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:29.845 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 53.0 with 2 tasks resource profile 0 20/07/01 10:38:29.846 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 53.0 (TID 96, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.847 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 53.0 (TID 97, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:29.847 Executor task launch worker for task 96 INFO Executor: Running task 0.0 in stage 53.0 (TID 96) 20/07/01 10:38:29.847 Executor task launch worker for task 97 INFO Executor: Running task 1.0 in stage 53.0 (TID 97) 20/07/01 10:38:29.858 Executor task launch worker for task 97 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.858 Executor task launch worker for task 96 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.858 Executor task launch worker for task 97 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.858 Executor task launch worker for task 96 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.858 Executor task launch worker for task 97 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.858 Executor task launch worker for task 96 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.858 Executor task launch worker for task 97 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:29.864 Executor task launch worker for task 97 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103829_0053_m_000001_97 20/07/01 10:38:29.865 Executor task launch worker for task 97 INFO Executor: Finished task 1.0 in stage 53.0 (TID 97). 2069 bytes result sent to driver 20/07/01 10:38:29.865 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 53.0 (TID 97) in 18 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:29.869 Executor task launch worker for task 96 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:29.880 Executor task launch worker for task 96 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103829_0053_m_000000_96' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c7c32462-37fa-44fd-8ba1-6c710169f494/deflate 20/07/01 10:38:29.880 Executor task launch worker for task 96 INFO SparkHadoopMapRedUtil: attempt_20200701103829_0053_m_000000_96: Committed 20/07/01 10:38:29.881 Executor task launch worker for task 96 INFO Executor: Finished task 0.0 in stage 53.0 (TID 96). 2198 bytes result sent to driver 20/07/01 10:38:29.881 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 53.0 (TID 96) in 35 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:29.881 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 53.0, whose tasks have all completed, from pool 20/07/01 10:38:29.882 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 53 (apply at OutcomeOf.scala:85) finished in 0.110 s 20/07/01 10:38:29.882 dag-scheduler-event-loop INFO DAGScheduler: Job 45 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:29.882 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 53: Stage finished 20/07/01 10:38:29.882 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 45 finished: apply at OutcomeOf.scala:85, took 0.111073 s 20/07/01 10:38:29.890 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 0a46b0ce-3b0d-44cd-bd96-f6dcd56281b1 committed. 20/07/01 10:38:29.890 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 0a46b0ce-3b0d-44cd-bd96-f6dcd56281b1. 20/07/01 10:38:29.902 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.902 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:29.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:29.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:29.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:29.913 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:29.917 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:29.918 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:29.918 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:29.918 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:29.928 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_92 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:29.938 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_92_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:29.938 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_92_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:29.939 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 92 from apply at OutcomeOf.scala:85 20/07/01 10:38:29.939 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:29.959 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:29.960 dag-scheduler-event-loop INFO DAGScheduler: Got job 46 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:29.960 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 54 (apply at OutcomeOf.scala:85) 20/07/01 10:38:29.960 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:29.960 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:29.960 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 54 (MapPartitionsRDD[189] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.023 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_93 stored as values in memory (estimated size 175.7 KiB, free 2.1 GiB) 20/07/01 10:38:30.025 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_93_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.025 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_93_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:30.026 dag-scheduler-event-loop INFO SparkContext: Created broadcast 93 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.026 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 54 (MapPartitionsRDD[189] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.026 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 54.0 with 2 tasks resource profile 0 20/07/01 10:38:30.028 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 54.0 (TID 98, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.028 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 54.0 (TID 99, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.028 Executor task launch worker for task 99 INFO Executor: Running task 1.0 in stage 54.0 (TID 99) 20/07/01 10:38:30.028 Executor task launch worker for task 98 INFO Executor: Running task 0.0 in stage 54.0 (TID 98) 20/07/01 10:38:30.040 Executor task launch worker for task 99 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:30.040 Executor task launch worker for task 99 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:30.040 Executor task launch worker for task 99 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:30.040 Executor task launch worker for task 99 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.041 Executor task launch worker for task 98 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:30.041 Executor task launch worker for task 98 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:30.041 Executor task launch worker for task 98 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:30.046 Executor task launch worker for task 99 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103829_0054_m_000001_99 20/07/01 10:38:30.046 Executor task launch worker for task 99 INFO Executor: Finished task 1.0 in stage 54.0 (TID 99). 2069 bytes result sent to driver 20/07/01 10:38:30.047 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 54.0 (TID 99) in 19 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.053 Executor task launch worker for task 98 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.063 Executor task launch worker for task 98 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103829_0054_m_000000_98' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c7c32462-37fa-44fd-8ba1-6c710169f494/snappy 20/07/01 10:38:30.063 Executor task launch worker for task 98 INFO SparkHadoopMapRedUtil: attempt_20200701103829_0054_m_000000_98: Committed 20/07/01 10:38:30.064 Executor task launch worker for task 98 INFO Executor: Finished task 0.0 in stage 54.0 (TID 98). 2198 bytes result sent to driver 20/07/01 10:38:30.064 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 54.0 (TID 98) in 37 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.064 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 54.0, whose tasks have all completed, from pool 20/07/01 10:38:30.065 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 54 (apply at OutcomeOf.scala:85) finished in 0.104 s 20/07/01 10:38:30.065 dag-scheduler-event-loop INFO DAGScheduler: Job 46 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.065 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 54: Stage finished 20/07/01 10:38:30.065 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 46 finished: apply at OutcomeOf.scala:85, took 0.105792 s 20/07/01 10:38:30.074 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 16687580-04e1-4fb7-9a18-633cc1ef0f90 committed. 20/07/01 10:38:30.075 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 16687580-04e1-4fb7-9a18-633cc1ef0f90. 20/07/01 10:38:30.081 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'write with compression - sql configs' ===== 20/07/01 10:38:30.082 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'dsl test' ===== 20/07/01 10:38:30.087 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.096 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.096 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.105 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.105 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.105 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.106 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.112 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_94 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.121 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_94_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.121 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_94_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.122 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 94 from $anonfun$new$50 at OutcomeOf.scala:85 20/07/01 10:38:30.122 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.129 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$50 at OutcomeOf.scala:85 20/07/01 10:38:30.130 dag-scheduler-event-loop INFO DAGScheduler: Got job 47 ($anonfun$new$50 at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:30.130 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 55 ($anonfun$new$50 at OutcomeOf.scala:85) 20/07/01 10:38:30.130 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.130 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.130 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 55 (MapPartitionsRDD[194] at $anonfun$new$50 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.131 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_95 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.133 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_95_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.133 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_95_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.133 dag-scheduler-event-loop INFO SparkContext: Created broadcast 95 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.133 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 55 (MapPartitionsRDD[194] at $anonfun$new$50 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:30.133 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 55.0 with 1 tasks resource profile 0 20/07/01 10:38:30.134 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 55.0 (TID 100, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.135 Executor task launch worker for task 100 INFO Executor: Running task 0.0 in stage 55.0 (TID 100) 20/07/01 10:38:30.137 Executor task launch worker for task 100 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:30.141 Executor task launch worker for task 100 INFO Executor: Finished task 0.0 in stage 55.0 (TID 100). 1631 bytes result sent to driver 20/07/01 10:38:30.142 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 55.0 (TID 100) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:30.142 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 55.0, whose tasks have all completed, from pool 20/07/01 10:38:30.142 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 55 ($anonfun$new$50 at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:30.143 dag-scheduler-event-loop INFO DAGScheduler: Job 47 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.143 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 55: Stage finished 20/07/01 10:38:30.143 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 47 finished: $anonfun$new$50 at OutcomeOf.scala:85, took 0.013579 s 20/07/01 10:38:30.144 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'dsl test' ===== 20/07/01 10:38:30.146 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'old avro data source name works' ===== 20/07/01 10:38:30.150 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.159 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.160 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.174 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.174 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.174 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.175 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.182 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_96 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.192 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_96_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.193 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_96_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.193 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 96 from $anonfun$new$51 at OutcomeOf.scala:85 20/07/01 10:38:30.194 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.202 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$51 at OutcomeOf.scala:85 20/07/01 10:38:30.204 dag-scheduler-event-loop INFO DAGScheduler: Got job 48 ($anonfun$new$51 at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:30.204 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 56 ($anonfun$new$51 at OutcomeOf.scala:85) 20/07/01 10:38:30.204 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.204 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.204 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 56 (MapPartitionsRDD[197] at $anonfun$new$51 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.206 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_97 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.207 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_97_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.208 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_97_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.208 dag-scheduler-event-loop INFO SparkContext: Created broadcast 97 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.208 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 56 (MapPartitionsRDD[197] at $anonfun$new$51 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:30.208 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 56.0 with 1 tasks resource profile 0 20/07/01 10:38:30.210 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 56.0 (TID 101, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.210 Executor task launch worker for task 101 INFO Executor: Running task 0.0 in stage 56.0 (TID 101) 20/07/01 10:38:30.213 Executor task launch worker for task 101 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:30.216 Executor task launch worker for task 101 INFO Executor: Finished task 0.0 in stage 56.0 (TID 101). 1631 bytes result sent to driver 20/07/01 10:38:30.217 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 56.0 (TID 101) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:30.217 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 56.0, whose tasks have all completed, from pool 20/07/01 10:38:30.217 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 56 ($anonfun$new$51 at OutcomeOf.scala:85) finished in 0.012 s 20/07/01 10:38:30.217 dag-scheduler-event-loop INFO DAGScheduler: Job 48 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.218 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 56: Stage finished 20/07/01 10:38:30.218 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 48 finished: $anonfun$new$51 at OutcomeOf.scala:85, took 0.014439 s 20/07/01 10:38:30.219 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'old avro data source name works' ===== 20/07/01 10:38:30.221 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'support of various data types' ===== 20/07/01 10:38:30.225 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.234 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.234 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.241 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.242 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.242 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.242 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:30.257 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_98 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.266 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_98_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.266 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_98_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.267 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 98 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.268 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.276 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.277 dag-scheduler-event-loop INFO DAGScheduler: Got job 49 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.277 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 57 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.277 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.277 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.277 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 57 (MapPartitionsRDD[200] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.279 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_99 stored as values in memory (estimated size 8.8 KiB, free 2.1 GiB) 20/07/01 10:38:30.280 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_99_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:30.280 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_99_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:30.280 dag-scheduler-event-loop INFO SparkContext: Created broadcast 99 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.281 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 57 (MapPartitionsRDD[200] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.281 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 57.0 with 2 tasks resource profile 0 20/07/01 10:38:30.282 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 57.0 (TID 102, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.283 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 57.0 (TID 103, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.283 Executor task launch worker for task 103 INFO Executor: Running task 1.0 in stage 57.0 (TID 103) 20/07/01 10:38:30.283 Executor task launch worker for task 102 INFO Executor: Running task 0.0 in stage 57.0 (TID 102) 20/07/01 10:38:30.286 Executor task launch worker for task 102 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.286 Executor task launch worker for task 103 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.292 Executor task launch worker for task 103 INFO Executor: Finished task 1.0 in stage 57.0 (TID 103). 1357 bytes result sent to driver 20/07/01 10:38:30.293 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 57.0 (TID 103) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.293 Executor task launch worker for task 102 INFO Executor: Finished task 0.0 in stage 57.0 (TID 102). 2233 bytes result sent to driver 20/07/01 10:38:30.294 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 57.0 (TID 102) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.294 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 57.0, whose tasks have all completed, from pool 20/07/01 10:38:30.294 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 57 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:30.295 dag-scheduler-event-loop INFO DAGScheduler: Job 49 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.295 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 57: Stage finished 20/07/01 10:38:30.295 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 49 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.018741 s 20/07/01 10:38:30.347 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 42.493801 ms 20/07/01 10:38:30.354 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.364 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.364 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.374 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.375 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.375 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.375 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.380 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_100 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.389 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_100_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.389 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_100_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.390 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 100 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.390 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.396 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.397 dag-scheduler-event-loop INFO DAGScheduler: Got job 50 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.397 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 58 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.397 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.398 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.398 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 58 (MapPartitionsRDD[203] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.399 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_101 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.401 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_101_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.401 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_101_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.401 dag-scheduler-event-loop INFO SparkContext: Created broadcast 101 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.402 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 58 (MapPartitionsRDD[203] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.402 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 58.0 with 2 tasks resource profile 0 20/07/01 10:38:30.403 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 58.0 (TID 104, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.403 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 58.0 (TID 105, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.403 Executor task launch worker for task 105 INFO Executor: Running task 1.0 in stage 58.0 (TID 105) 20/07/01 10:38:30.403 Executor task launch worker for task 104 INFO Executor: Running task 0.0 in stage 58.0 (TID 104) 20/07/01 10:38:30.406 Executor task launch worker for task 105 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.406 Executor task launch worker for task 104 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.411 Executor task launch worker for task 105 INFO Executor: Finished task 1.0 in stage 58.0 (TID 105). 1357 bytes result sent to driver 20/07/01 10:38:30.411 Executor task launch worker for task 104 INFO Executor: Finished task 0.0 in stage 58.0 (TID 104). 1499 bytes result sent to driver 20/07/01 10:38:30.412 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 58.0 (TID 105) in 9 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.412 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 58.0 (TID 104) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.412 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 58.0, whose tasks have all completed, from pool 20/07/01 10:38:30.412 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 58 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.014 s 20/07/01 10:38:30.413 dag-scheduler-event-loop INFO DAGScheduler: Job 50 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.413 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 58: Stage finished 20/07/01 10:38:30.413 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 50 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.016143 s 20/07/01 10:38:30.419 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.430 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.430 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.442 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.442 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.442 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.443 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:30.450 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_102 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.459 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_102_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.460 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_102_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.460 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 102 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.461 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.468 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.468 dag-scheduler-event-loop INFO DAGScheduler: Got job 51 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.468 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 59 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.468 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.468 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.469 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 59 (MapPartitionsRDD[206] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.470 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_103 stored as values in memory (estimated size 7.6 KiB, free 2.1 GiB) 20/07/01 10:38:30.471 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_103_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.471 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_103_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.472 dag-scheduler-event-loop INFO SparkContext: Created broadcast 103 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.472 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 59 (MapPartitionsRDD[206] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.472 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 59.0 with 2 tasks resource profile 0 20/07/01 10:38:30.473 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 59.0 (TID 106, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.473 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 59.0 (TID 107, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.473 Executor task launch worker for task 107 INFO Executor: Running task 1.0 in stage 59.0 (TID 107) 20/07/01 10:38:30.473 Executor task launch worker for task 106 INFO Executor: Running task 0.0 in stage 59.0 (TID 106) 20/07/01 10:38:30.476 Executor task launch worker for task 107 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.476 Executor task launch worker for task 106 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.489 Executor task launch worker for task 107 INFO CodeGenerator: Code generated in 11.732956 ms 20/07/01 10:38:30.493 Executor task launch worker for task 107 INFO Executor: Finished task 1.0 in stage 59.0 (TID 107). 1357 bytes result sent to driver 20/07/01 10:38:30.494 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 59.0 (TID 107) in 21 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.494 Executor task launch worker for task 106 INFO Executor: Finished task 0.0 in stage 59.0 (TID 106). 1507 bytes result sent to driver 20/07/01 10:38:30.494 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 59.0 (TID 106) in 22 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.494 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 59.0, whose tasks have all completed, from pool 20/07/01 10:38:30.495 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 59 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.026 s 20/07/01 10:38:30.495 dag-scheduler-event-loop INFO DAGScheduler: Job 51 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.495 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 59: Stage finished 20/07/01 10:38:30.495 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 51 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.027388 s 20/07/01 10:38:30.518 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 18.628509 ms 20/07/01 10:38:30.524 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.534 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.534 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.544 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.544 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.544 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.545 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.549 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_104 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.558 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_104_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.559 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_104_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.559 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 104 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.560 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.567 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.567 dag-scheduler-event-loop INFO DAGScheduler: Got job 52 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.567 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 60 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.567 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.567 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.568 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 60 (MapPartitionsRDD[209] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.569 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_105 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.570 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_105_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.570 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_105_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.571 dag-scheduler-event-loop INFO SparkContext: Created broadcast 105 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.571 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 60 (MapPartitionsRDD[209] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.571 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 60.0 with 2 tasks resource profile 0 20/07/01 10:38:30.572 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 60.0 (TID 108, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.572 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 60.0 (TID 109, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.572 Executor task launch worker for task 109 INFO Executor: Running task 1.0 in stage 60.0 (TID 109) 20/07/01 10:38:30.572 Executor task launch worker for task 108 INFO Executor: Running task 0.0 in stage 60.0 (TID 108) 20/07/01 10:38:30.577 Executor task launch worker for task 109 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.578 Executor task launch worker for task 108 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.583 Executor task launch worker for task 109 INFO Executor: Finished task 1.0 in stage 60.0 (TID 109). 1357 bytes result sent to driver 20/07/01 10:38:30.584 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 60.0 (TID 109) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.585 Executor task launch worker for task 108 INFO Executor: Finished task 0.0 in stage 60.0 (TID 108). 1444 bytes result sent to driver 20/07/01 10:38:30.586 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 60.0 (TID 108) in 14 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.586 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 60.0, whose tasks have all completed, from pool 20/07/01 10:38:30.587 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 60 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:30.587 dag-scheduler-event-loop INFO DAGScheduler: Job 52 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.587 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 60: Stage finished 20/07/01 10:38:30.587 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 52 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.020357 s 20/07/01 10:38:30.595 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.606 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.606 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.617 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.622 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_106 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.631 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_106_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.631 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_106_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.632 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 106 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.632 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.639 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.639 dag-scheduler-event-loop INFO DAGScheduler: Got job 53 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.639 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 61 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.639 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.640 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.640 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 61 (MapPartitionsRDD[212] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.641 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_107 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.642 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_107_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.642 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_107_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.642 dag-scheduler-event-loop INFO SparkContext: Created broadcast 107 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.643 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 61 (MapPartitionsRDD[212] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.643 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 61.0 with 2 tasks resource profile 0 20/07/01 10:38:30.644 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 61.0 (TID 110, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.644 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 61.0 (TID 111, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.644 Executor task launch worker for task 110 INFO Executor: Running task 0.0 in stage 61.0 (TID 110) 20/07/01 10:38:30.644 Executor task launch worker for task 111 INFO Executor: Running task 1.0 in stage 61.0 (TID 111) 20/07/01 10:38:30.647 Executor task launch worker for task 111 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.647 Executor task launch worker for task 110 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.650 Executor task launch worker for task 111 INFO Executor: Finished task 1.0 in stage 61.0 (TID 111). 1357 bytes result sent to driver 20/07/01 10:38:30.650 Executor task launch worker for task 110 INFO Executor: Finished task 0.0 in stage 61.0 (TID 110). 1432 bytes result sent to driver 20/07/01 10:38:30.651 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 61.0 (TID 111) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.651 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 61.0 (TID 110) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.651 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 61.0, whose tasks have all completed, from pool 20/07/01 10:38:30.651 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 61 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:30.651 dag-scheduler-event-loop INFO DAGScheduler: Job 53 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.651 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 61: Stage finished 20/07/01 10:38:30.652 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 53 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.012602 s 20/07/01 10:38:30.658 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.668 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.668 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.677 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.677 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.678 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.678 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.682 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_108 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.691 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_108_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.691 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_108_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.692 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 108 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.693 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.700 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.701 dag-scheduler-event-loop INFO DAGScheduler: Got job 54 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.701 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 62 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.701 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.701 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.701 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 62 (MapPartitionsRDD[215] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.702 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_109 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.704 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_109_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.704 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_109_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.705 dag-scheduler-event-loop INFO SparkContext: Created broadcast 109 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.705 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 62 (MapPartitionsRDD[215] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.705 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 62.0 with 2 tasks resource profile 0 20/07/01 10:38:30.706 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 62.0 (TID 112, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.707 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 62.0 (TID 113, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.707 Executor task launch worker for task 112 INFO Executor: Running task 0.0 in stage 62.0 (TID 112) 20/07/01 10:38:30.707 Executor task launch worker for task 113 INFO Executor: Running task 1.0 in stage 62.0 (TID 113) 20/07/01 10:38:30.711 Executor task launch worker for task 113 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.711 Executor task launch worker for task 112 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.714 Executor task launch worker for task 113 INFO Executor: Finished task 1.0 in stage 62.0 (TID 113). 1357 bytes result sent to driver 20/07/01 10:38:30.714 Executor task launch worker for task 112 INFO Executor: Finished task 0.0 in stage 62.0 (TID 112). 1430 bytes result sent to driver 20/07/01 10:38:30.714 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 62.0 (TID 113) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.714 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 62.0 (TID 112) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.714 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 62.0, whose tasks have all completed, from pool 20/07/01 10:38:30.715 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 62 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:30.715 dag-scheduler-event-loop INFO DAGScheduler: Job 54 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.715 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 62: Stage finished 20/07/01 10:38:30.715 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 54 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.014908 s 20/07/01 10:38:30.724 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.736 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.736 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.746 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.747 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.747 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.747 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.752 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_110 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.761 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_110_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.761 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_110_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.761 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 110 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.762 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.769 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.770 dag-scheduler-event-loop INFO DAGScheduler: Got job 55 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.770 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 63 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.770 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.770 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.770 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 63 (MapPartitionsRDD[218] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.771 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_111 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.772 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_111_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.773 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_111_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.773 dag-scheduler-event-loop INFO SparkContext: Created broadcast 111 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.773 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 63 (MapPartitionsRDD[218] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.773 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 63.0 with 2 tasks resource profile 0 20/07/01 10:38:30.774 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 63.0 (TID 114, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.774 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 63.0 (TID 115, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.775 Executor task launch worker for task 115 INFO Executor: Running task 1.0 in stage 63.0 (TID 115) 20/07/01 10:38:30.775 Executor task launch worker for task 114 INFO Executor: Running task 0.0 in stage 63.0 (TID 114) 20/07/01 10:38:30.778 Executor task launch worker for task 114 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.778 Executor task launch worker for task 115 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.786 Executor task launch worker for task 114 INFO CodeGenerator: Code generated in 7.465322 ms 20/07/01 10:38:30.791 Executor task launch worker for task 115 INFO Executor: Finished task 1.0 in stage 63.0 (TID 115). 1357 bytes result sent to driver 20/07/01 10:38:30.791 Executor task launch worker for task 114 INFO Executor: Finished task 0.0 in stage 63.0 (TID 114). 1441 bytes result sent to driver 20/07/01 10:38:30.791 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 63.0 (TID 115) in 17 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.792 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 63.0 (TID 114) in 18 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.792 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 63.0, whose tasks have all completed, from pool 20/07/01 10:38:30.792 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 63 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.022 s 20/07/01 10:38:30.792 dag-scheduler-event-loop INFO DAGScheduler: Job 55 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.792 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 63: Stage finished 20/07/01 10:38:30.792 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 55 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.023535 s 20/07/01 10:38:30.800 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.810 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.810 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.820 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.820 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.820 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.820 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:30.826 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_112 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.834 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_112_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.835 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_112_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.835 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 112 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.836 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.842 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.843 dag-scheduler-event-loop INFO DAGScheduler: Got job 56 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.843 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 64 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.843 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.843 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.843 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 64 (MapPartitionsRDD[221] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.844 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_113 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.845 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_113_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.846 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_113_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:30.846 dag-scheduler-event-loop INFO SparkContext: Created broadcast 113 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.846 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 64 (MapPartitionsRDD[221] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.846 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 64.0 with 2 tasks resource profile 0 20/07/01 10:38:30.847 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 64.0 (TID 116, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.847 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 64.0 (TID 117, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.847 Executor task launch worker for task 117 INFO Executor: Running task 1.0 in stage 64.0 (TID 117) 20/07/01 10:38:30.847 Executor task launch worker for task 116 INFO Executor: Running task 0.0 in stage 64.0 (TID 116) 20/07/01 10:38:30.849 Executor task launch worker for task 117 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.849 Executor task launch worker for task 116 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.852 Executor task launch worker for task 117 INFO Executor: Finished task 1.0 in stage 64.0 (TID 117). 1357 bytes result sent to driver 20/07/01 10:38:30.852 Executor task launch worker for task 116 INFO Executor: Finished task 0.0 in stage 64.0 (TID 116). 1455 bytes result sent to driver 20/07/01 10:38:30.853 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 64.0 (TID 117) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.853 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 64.0 (TID 116) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.853 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 64.0, whose tasks have all completed, from pool 20/07/01 10:38:30.853 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 64 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.009 s 20/07/01 10:38:30.853 dag-scheduler-event-loop INFO DAGScheduler: Job 56 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.853 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 64: Stage finished 20/07/01 10:38:30.853 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 56 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.011123 s 20/07/01 10:38:30.860 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.871 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.872 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.881 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.881 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.881 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.881 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:30.887 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_114 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.896 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_114_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.896 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_114_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.897 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 114 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.898 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.906 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.906 dag-scheduler-event-loop INFO DAGScheduler: Got job 57 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.906 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 65 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.906 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.907 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.907 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 65 (MapPartitionsRDD[224] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.908 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_115 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:30.909 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_115_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.909 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_115_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.909 dag-scheduler-event-loop INFO SparkContext: Created broadcast 115 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:30.910 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 65 (MapPartitionsRDD[224] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:30.910 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 65.0 with 2 tasks resource profile 0 20/07/01 10:38:30.911 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 65.0 (TID 118, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.911 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 65.0 (TID 119, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:30.911 Executor task launch worker for task 119 INFO Executor: Running task 1.0 in stage 65.0 (TID 119) 20/07/01 10:38:30.911 Executor task launch worker for task 118 INFO Executor: Running task 0.0 in stage 65.0 (TID 118) 20/07/01 10:38:30.915 Executor task launch worker for task 119 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:30.915 Executor task launch worker for task 118 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:30.924 Executor task launch worker for task 118 INFO CodeGenerator: Code generated in 7.744257 ms 20/07/01 10:38:30.931 Executor task launch worker for task 119 INFO Executor: Finished task 1.0 in stage 65.0 (TID 119). 1357 bytes result sent to driver 20/07/01 10:38:30.931 Executor task launch worker for task 118 INFO Executor: Finished task 0.0 in stage 65.0 (TID 118). 1637 bytes result sent to driver 20/07/01 10:38:30.932 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 65.0 (TID 119) in 21 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:30.932 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 65.0 (TID 118) in 22 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:30.932 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 65.0, whose tasks have all completed, from pool 20/07/01 10:38:30.932 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 65 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.025 s 20/07/01 10:38:30.933 dag-scheduler-event-loop INFO DAGScheduler: Job 57 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:30.933 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 65: Stage finished 20/07/01 10:38:30.933 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 57 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.027023 s 20/07/01 10:38:30.943 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 7.24939 ms 20/07/01 10:38:30.948 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:30.957 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.958 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:30.967 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:30.967 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:30.967 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:30.967 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct> 20/07/01 10:38:30.972 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_116 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:30.981 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_116_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:30.982 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_116_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:30.982 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 116 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.983 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:30.989 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:30.997 dag-scheduler-event-loop INFO DAGScheduler: Got job 58 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:30.997 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 66 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:30.998 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:30.998 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:30.998 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 66 (MapPartitionsRDD[227] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:30.999 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_117 stored as values in memory (estimated size 7.6 KiB, free 2.1 GiB) 20/07/01 10:38:31.000 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_117_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.000 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_117_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.001 dag-scheduler-event-loop INFO SparkContext: Created broadcast 117 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.001 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 66 (MapPartitionsRDD[227] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.001 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 66.0 with 2 tasks resource profile 0 20/07/01 10:38:31.002 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 66.0 (TID 120, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.002 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 66.0 (TID 121, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.002 Executor task launch worker for task 121 INFO Executor: Running task 1.0 in stage 66.0 (TID 121) 20/07/01 10:38:31.002 Executor task launch worker for task 120 INFO Executor: Running task 0.0 in stage 66.0 (TID 120) 20/07/01 10:38:31.006 Executor task launch worker for task 121 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:31.006 Executor task launch worker for task 120 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.014 Executor task launch worker for task 120 INFO CodeGenerator: Code generated in 6.92433 ms 20/07/01 10:38:31.019 Executor task launch worker for task 121 INFO Executor: Finished task 1.0 in stage 66.0 (TID 121). 1357 bytes result sent to driver 20/07/01 10:38:31.019 Executor task launch worker for task 120 INFO Executor: Finished task 0.0 in stage 66.0 (TID 120). 1451 bytes result sent to driver 20/07/01 10:38:31.019 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 66.0 (TID 121) in 17 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.020 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 66.0 (TID 120) in 19 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.020 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 66.0, whose tasks have all completed, from pool 20/07/01 10:38:31.020 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 66 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.022 s 20/07/01 10:38:31.020 dag-scheduler-event-loop INFO DAGScheduler: Job 58 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.020 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 66: Stage finished 20/07/01 10:38:31.020 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 58 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.023160 s 20/07/01 10:38:31.034 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 11.716563 ms 20/07/01 10:38:31.040 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:31.049 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.050 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.060 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.061 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.061 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.061 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:31.066 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_118 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.075 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_118_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.075 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_118_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.076 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 118 from $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:31.076 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.084 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$52 at OutcomeOf.scala:85 20/07/01 10:38:31.085 dag-scheduler-event-loop INFO DAGScheduler: Got job 59 ($anonfun$new$52 at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.085 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 67 ($anonfun$new$52 at OutcomeOf.scala:85) 20/07/01 10:38:31.085 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.085 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.085 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 67 (MapPartitionsRDD[230] at $anonfun$new$52 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.087 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_119 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:31.088 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_119_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.089 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_119_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.089 dag-scheduler-event-loop INFO SparkContext: Created broadcast 119 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.089 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 67 (MapPartitionsRDD[230] at $anonfun$new$52 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.089 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 67.0 with 2 tasks resource profile 0 20/07/01 10:38:31.090 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 67.0 (TID 122, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.090 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 67.0 (TID 123, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.091 Executor task launch worker for task 123 INFO Executor: Running task 1.0 in stage 67.0 (TID 123) 20/07/01 10:38:31.091 Executor task launch worker for task 122 INFO Executor: Running task 0.0 in stage 67.0 (TID 122) 20/07/01 10:38:31.093 Executor task launch worker for task 123 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:31.093 Executor task launch worker for task 122 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.096 Executor task launch worker for task 123 INFO Executor: Finished task 1.0 in stage 67.0 (TID 123). 1357 bytes result sent to driver 20/07/01 10:38:31.096 Executor task launch worker for task 122 INFO Executor: Finished task 0.0 in stage 67.0 (TID 122). 1447 bytes result sent to driver 20/07/01 10:38:31.096 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 67.0 (TID 123) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.096 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 67.0 (TID 122) in 6 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.097 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 67.0, whose tasks have all completed, from pool 20/07/01 10:38:31.097 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 67 ($anonfun$new$52 at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:31.097 dag-scheduler-event-loop INFO DAGScheduler: Job 59 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.097 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 67: Stage finished 20/07/01 10:38:31.097 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 59 finished: $anonfun$new$52 at OutcomeOf.scala:85, took 0.013294 s 20/07/01 10:38:31.100 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'support of various data types' ===== 20/07/01 10:38:31.101 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'sql test' ===== 20/07/01 10:38:31.111 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.111 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.126 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:31.136 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.136 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.146 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.147 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.147 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.147 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:31.153 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_120 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.162 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_120_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.162 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_120_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.163 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 120 from $anonfun$new$65 at OutcomeOf.scala:85 20/07/01 10:38:31.163 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.171 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: $anonfun$new$65 at OutcomeOf.scala:85 20/07/01 10:38:31.172 dag-scheduler-event-loop INFO DAGScheduler: Got job 60 ($anonfun$new$65 at OutcomeOf.scala:85) with 1 output partitions 20/07/01 10:38:31.172 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 68 ($anonfun$new$65 at OutcomeOf.scala:85) 20/07/01 10:38:31.172 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.172 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.172 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 68 (MapPartitionsRDD[233] at $anonfun$new$65 at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.173 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_121 stored as values in memory (estimated size 7.7 KiB, free 2.1 GiB) 20/07/01 10:38:31.175 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_121_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:31.175 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_121_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:31.176 dag-scheduler-event-loop INFO SparkContext: Created broadcast 121 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.176 dag-scheduler-event-loop INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 68 (MapPartitionsRDD[233] at $anonfun$new$65 at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) 20/07/01 10:38:31.176 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 68.0 with 1 tasks resource profile 0 20/07/01 10:38:31.177 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 68.0 (TID 124, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7813 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.177 Executor task launch worker for task 124 INFO Executor: Running task 0.0 in stage 68.0 (TID 124) 20/07/01 10:38:31.180 Executor task launch worker for task 124 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/episodes.avro, range: 0-597, partition values: [empty row] 20/07/01 10:38:31.184 Executor task launch worker for task 124 INFO Executor: Finished task 0.0 in stage 68.0 (TID 124). 1796 bytes result sent to driver 20/07/01 10:38:31.185 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 68.0 (TID 124) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/1) 20/07/01 10:38:31.185 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 68.0, whose tasks have all completed, from pool 20/07/01 10:38:31.185 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 68 ($anonfun$new$65 at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:31.186 dag-scheduler-event-loop INFO DAGScheduler: Job 60 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.186 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 68: Stage finished 20/07/01 10:38:31.186 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 60 finished: $anonfun$new$65 at OutcomeOf.scala:85, took 0.014692 s 20/07/01 10:38:31.188 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'sql test' ===== 20/07/01 10:38:31.189 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'conversion to avro and back' ===== 20/07/01 10:38:31.199 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:31.213 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.213 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.224 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.224 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.224 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.224 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:31.228 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:31.228 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.228 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.228 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.241 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_122 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.252 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_122_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.252 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_122_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.253 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 122 from apply at OutcomeOf.scala:85 20/07/01 10:38:31.254 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.274 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:31.275 dag-scheduler-event-loop INFO DAGScheduler: Got job 61 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.275 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 69 (apply at OutcomeOf.scala:85) 20/07/01 10:38:31.275 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.275 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.275 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 69 (MapPartitionsRDD[235] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.345 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_123 stored as values in memory (estimated size 175.7 KiB, free 2.1 GiB) 20/07/01 10:38:31.348 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_123_piece0 stored as bytes in memory (estimated size 62.5 KiB, free 2.1 GiB) 20/07/01 10:38:31.348 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_123_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.348 dag-scheduler-event-loop INFO SparkContext: Created broadcast 123 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.348 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 69 (MapPartitionsRDD[235] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.348 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 69.0 with 2 tasks resource profile 0 20/07/01 10:38:31.349 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 69.0 (TID 125, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.350 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 69.0 (TID 126, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.350 Executor task launch worker for task 125 INFO Executor: Running task 0.0 in stage 69.0 (TID 125) 20/07/01 10:38:31.350 Executor task launch worker for task 126 INFO Executor: Running task 1.0 in stage 69.0 (TID 126) 20/07/01 10:38:31.367 Executor task launch worker for task 126 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.367 Executor task launch worker for task 126 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.367 Executor task launch worker for task 126 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.367 Executor task launch worker for task 126 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:31.368 Executor task launch worker for task 125 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.368 Executor task launch worker for task 125 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.369 Executor task launch worker for task 125 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.374 Executor task launch worker for task 126 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103831_0069_m_000001_126 20/07/01 10:38:31.375 Executor task launch worker for task 126 INFO Executor: Finished task 1.0 in stage 69.0 (TID 126). 2069 bytes result sent to driver 20/07/01 10:38:31.376 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 69.0 (TID 126) in 26 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.379 Executor task launch worker for task 125 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.389 Executor task launch worker for task 125 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103831_0069_m_000000_125' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c6cb66a1-284c-4570-b924-76ba3b06c1f1/avro 20/07/01 10:38:31.390 Executor task launch worker for task 125 INFO SparkHadoopMapRedUtil: attempt_20200701103831_0069_m_000000_125: Committed 20/07/01 10:38:31.390 Executor task launch worker for task 125 INFO Executor: Finished task 0.0 in stage 69.0 (TID 125). 2198 bytes result sent to driver 20/07/01 10:38:31.391 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 69.0 (TID 125) in 42 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.391 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 69.0, whose tasks have all completed, from pool 20/07/01 10:38:31.392 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 69 (apply at OutcomeOf.scala:85) finished in 0.116 s 20/07/01 10:38:31.392 dag-scheduler-event-loop INFO DAGScheduler: Job 61 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.392 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 69: Stage finished 20/07/01 10:38:31.392 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 61 finished: apply at OutcomeOf.scala:85, took 0.117844 s 20/07/01 10:38:31.398 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job d6f96564-45c1-4a77-9932-4944eab4a72e committed. 20/07/01 10:38:31.399 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job d6f96564-45c1-4a77-9932-4944eab4a72e. 20/07/01 10:38:31.405 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:31.411 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.411 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.418 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.418 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.419 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.419 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:31.443 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_124 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.454 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_124_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.455 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_124_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.455 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 124 from apply at OutcomeOf.scala:85 20/07/01 10:38:31.456 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.465 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:31.466 dag-scheduler-event-loop INFO DAGScheduler: Got job 62 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.466 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 70 (apply at OutcomeOf.scala:85) 20/07/01 10:38:31.466 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.466 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.466 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 70 (MapPartitionsRDD[240] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.468 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_125 stored as values in memory (estimated size 8.8 KiB, free 2.1 GiB) 20/07/01 10:38:31.470 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_125_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:31.470 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_125_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.471 dag-scheduler-event-loop INFO SparkContext: Created broadcast 125 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.471 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 70 (MapPartitionsRDD[240] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.471 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 70.0 with 2 tasks resource profile 0 20/07/01 10:38:31.472 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 70.0 (TID 127, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.472 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 70.0 (TID 128, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.472 Executor task launch worker for task 128 INFO Executor: Running task 1.0 in stage 70.0 (TID 128) 20/07/01 10:38:31.472 Executor task launch worker for task 127 INFO Executor: Running task 0.0 in stage 70.0 (TID 127) 20/07/01 10:38:31.475 Executor task launch worker for task 128 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:31.475 Executor task launch worker for task 127 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.481 Executor task launch worker for task 128 INFO Executor: Finished task 1.0 in stage 70.0 (TID 128). 1357 bytes result sent to driver 20/07/01 10:38:31.482 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 70.0 (TID 128) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.484 Executor task launch worker for task 127 INFO Executor: Finished task 0.0 in stage 70.0 (TID 127). 2233 bytes result sent to driver 20/07/01 10:38:31.485 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 70.0 (TID 127) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.485 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 70.0, whose tasks have all completed, from pool 20/07/01 10:38:31.485 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 70 (apply at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:31.485 dag-scheduler-event-loop INFO DAGScheduler: Job 62 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.485 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 70: Stage finished 20/07/01 10:38:31.486 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 62 finished: apply at OutcomeOf.scala:85, took 0.020175 s 20/07/01 10:38:31.504 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:31.515 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.515 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.526 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.527 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.527 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.527 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:31.539 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.539 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.561 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.561 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.561 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.562 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:31.564 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_126 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.579 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_126_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.579 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_126_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.579 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 126 from apply at OutcomeOf.scala:85 20/07/01 10:38:31.580 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.647 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:31.647 dag-scheduler-event-loop INFO DAGScheduler: Got job 63 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.647 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 71 (apply at OutcomeOf.scala:85) 20/07/01 10:38:31.647 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.648 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.648 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 71 (MapPartitionsRDD[245] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.673 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_127 stored as values in memory (estimated size 19.5 KiB, free 2.1 GiB) 20/07/01 10:38:31.674 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_127_piece0 stored as bytes in memory (estimated size 8.6 KiB, free 2.1 GiB) 20/07/01 10:38:31.674 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_127_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 8.6 KiB, free: 2.1 GiB) 20/07/01 10:38:31.675 dag-scheduler-event-loop INFO SparkContext: Created broadcast 127 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.675 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 71 (MapPartitionsRDD[245] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.675 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 71.0 with 2 tasks resource profile 0 20/07/01 10:38:31.676 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 71.0 (TID 129, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.676 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 71.0 (TID 130, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.677 Executor task launch worker for task 129 INFO Executor: Running task 0.0 in stage 71.0 (TID 129) 20/07/01 10:38:31.677 Executor task launch worker for task 130 INFO Executor: Running task 1.0 in stage 71.0 (TID 130) 20/07/01 10:38:31.678 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_93_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.679 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_79_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 69.8 KiB, free: 2.1 GiB) 20/07/01 10:38:31.680 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_122_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.681 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_118_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.681 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_116_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.682 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_89_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.4 KiB, free: 2.1 GiB) 20/07/01 10:38:31.682 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_92_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.683 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_94_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.684 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_90_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.685 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_85_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.686 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_87_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.686 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_107_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.687 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_114_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.688 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_112_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.689 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_104_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.690 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_106_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.691 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_95_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.691 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_82_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 5.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.692 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_109_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.692 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_115_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.693 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_99_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.693 Executor task launch worker for task 129 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c6cb66a1-284c-4570-b924-76ba3b06c1f1/avro/part-00000-7d6b766c-c485-48b3-8a09-04ec02a4740d-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.693 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_101_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.694 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_103_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.694 Executor task launch worker for task 130 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c6cb66a1-284c-4570-b924-76ba3b06c1f1/avro/part-00000-7d6b766c-c485-48b3-8a09-04ec02a4740d-c000.avro, range: 1024-1388, partition values: [empty row] 20/07/01 10:38:31.695 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_84_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.696 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_100_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.696 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_81_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 6.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.697 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_121_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:31.697 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_102_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.698 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_83_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 1276.0 B, free: 2.1 GiB) 20/07/01 10:38:31.698 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_125_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.699 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_108_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.700 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_91_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.700 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_80_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.701 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_117_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.701 Executor task launch worker for task 130 INFO Executor: Finished task 1.0 in stage 71.0 (TID 130). 1511 bytes result sent to driver 20/07/01 10:38:31.701 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_86_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.702 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 71.0 (TID 130) in 26 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.702 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_105_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.702 Executor task launch worker for task 129 INFO Executor: Finished task 0.0 in stage 71.0 (TID 129). 1554 bytes result sent to driver 20/07/01 10:38:31.703 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_110_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.703 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 71.0 (TID 129) in 27 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.703 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 71.0, whose tasks have all completed, from pool 20/07/01 10:38:31.703 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 71 (apply at OutcomeOf.scala:85) finished in 0.055 s 20/07/01 10:38:31.703 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_124_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.703 dag-scheduler-event-loop INFO DAGScheduler: Job 63 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.703 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 71: Stage finished 20/07/01 10:38:31.703 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 63 finished: apply at OutcomeOf.scala:85, took 0.056746 s 20/07/01 10:38:31.704 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_96_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.705 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_88_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.705 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_119_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.706 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_111_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.706 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_120_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.707 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_113_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.708 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_123_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 62.5 KiB, free: 2.1 GiB) 20/07/01 10:38:31.709 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_97_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:31.709 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Removed broadcast_98_piece0 on amp-jenkins-worker-04.amp:35025 in memory (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.717 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_128 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.727 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_128_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.727 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_128_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.728 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 128 from apply at OutcomeOf.scala:85 20/07/01 10:38:31.729 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.739 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:31.740 dag-scheduler-event-loop INFO DAGScheduler: Got job 64 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.740 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 72 (apply at OutcomeOf.scala:85) 20/07/01 10:38:31.740 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.740 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.740 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 72 (MapPartitionsRDD[248] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.741 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_129 stored as values in memory (estimated size 8.8 KiB, free 2.1 GiB) 20/07/01 10:38:31.743 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_129_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:31.743 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_129_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.744 dag-scheduler-event-loop INFO SparkContext: Created broadcast 129 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.744 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 72 (MapPartitionsRDD[248] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.744 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 72.0 with 2 tasks resource profile 0 20/07/01 10:38:31.745 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 72.0 (TID 131, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.745 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 72.0 (TID 132, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.746 Executor task launch worker for task 131 INFO Executor: Running task 0.0 in stage 72.0 (TID 131) 20/07/01 10:38:31.746 Executor task launch worker for task 132 INFO Executor: Running task 1.0 in stage 72.0 (TID 132) 20/07/01 10:38:31.748 Executor task launch worker for task 131 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c6cb66a1-284c-4570-b924-76ba3b06c1f1/avro/part-00000-7d6b766c-c485-48b3-8a09-04ec02a4740d-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.748 Executor task launch worker for task 132 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-c6cb66a1-284c-4570-b924-76ba3b06c1f1/avro/part-00000-7d6b766c-c485-48b3-8a09-04ec02a4740d-c000.avro, range: 1024-1388, partition values: [empty row] 20/07/01 10:38:31.755 Executor task launch worker for task 132 INFO Executor: Finished task 1.0 in stage 72.0 (TID 132). 1357 bytes result sent to driver 20/07/01 10:38:31.756 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 72.0 (TID 132) in 11 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.757 Executor task launch worker for task 131 INFO Executor: Finished task 0.0 in stage 72.0 (TID 131). 2233 bytes result sent to driver 20/07/01 10:38:31.757 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 72.0 (TID 131) in 12 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.757 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 72.0, whose tasks have all completed, from pool 20/07/01 10:38:31.758 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 72 (apply at OutcomeOf.scala:85) finished in 0.017 s 20/07/01 10:38:31.758 dag-scheduler-event-loop INFO DAGScheduler: Job 64 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.758 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 72: Stage finished 20/07/01 10:38:31.758 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 64 finished: apply at OutcomeOf.scala:85, took 0.019161 s 20/07/01 10:38:31.775 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'conversion to avro and back' ===== 20/07/01 10:38:31.777 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'conversion to avro and back with namespace' ===== 20/07/01 10:38:31.783 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:31.800 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.800 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:31.811 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:31.812 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:31.812 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:31.812 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:31.816 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:31.816 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.816 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.816 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.828 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_130 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:31.841 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_130_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.842 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_130_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:31.843 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 130 from apply at OutcomeOf.scala:85 20/07/01 10:38:31.843 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:31.861 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:31.862 dag-scheduler-event-loop INFO DAGScheduler: Got job 65 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:31.862 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 73 (apply at OutcomeOf.scala:85) 20/07/01 10:38:31.862 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:31.862 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:31.862 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 73 (MapPartitionsRDD[250] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:31.948 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_131 stored as values in memory (estimated size 176.1 KiB, free 2.1 GiB) 20/07/01 10:38:31.950 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_131_piece0 stored as bytes in memory (estimated size 62.7 KiB, free 2.1 GiB) 20/07/01 10:38:31.951 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_131_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 62.7 KiB, free: 2.1 GiB) 20/07/01 10:38:31.951 dag-scheduler-event-loop INFO SparkContext: Created broadcast 131 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:31.952 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 73 (MapPartitionsRDD[250] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:31.952 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 73.0 with 2 tasks resource profile 0 20/07/01 10:38:31.953 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 73.0 (TID 133, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.953 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 73.0 (TID 134, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:31.953 Executor task launch worker for task 133 INFO Executor: Running task 0.0 in stage 73.0 (TID 133) 20/07/01 10:38:31.953 Executor task launch worker for task 134 INFO Executor: Running task 1.0 in stage 73.0 (TID 134) 20/07/01 10:38:31.968 Executor task launch worker for task 134 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.968 Executor task launch worker for task 133 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:31.968 Executor task launch worker for task 134 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.969 Executor task launch worker for task 133 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:31.969 Executor task launch worker for task 134 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.969 Executor task launch worker for task 133 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:31.969 Executor task launch worker for task 134 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:31.974 Executor task launch worker for task 134 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20200701103831_0073_m_000001_134 20/07/01 10:38:31.975 Executor task launch worker for task 134 INFO Executor: Finished task 1.0 in stage 73.0 (TID 134). 2069 bytes result sent to driver 20/07/01 10:38:31.975 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 73.0 (TID 134) in 22 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:31.980 Executor task launch worker for task 133 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:31.989 Executor task launch worker for task 133 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103831_0073_m_000000_133' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro 20/07/01 10:38:31.989 Executor task launch worker for task 133 INFO SparkHadoopMapRedUtil: attempt_20200701103831_0073_m_000000_133: Committed 20/07/01 10:38:31.989 Executor task launch worker for task 133 INFO Executor: Finished task 0.0 in stage 73.0 (TID 133). 2198 bytes result sent to driver 20/07/01 10:38:31.990 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 73.0 (TID 133) in 38 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:31.990 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 73.0, whose tasks have all completed, from pool 20/07/01 10:38:31.990 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 73 (apply at OutcomeOf.scala:85) finished in 0.127 s 20/07/01 10:38:31.990 dag-scheduler-event-loop INFO DAGScheduler: Job 65 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:31.991 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 73: Stage finished 20/07/01 10:38:31.991 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 65 finished: apply at OutcomeOf.scala:85, took 0.129499 s 20/07/01 10:38:31.996 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job c2ce7327-4945-4b4b-a83e-2f9f4df711b0 committed. 20/07/01 10:38:31.996 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job c2ce7327-4945-4b4b-a83e-2f9f4df711b0. 20/07/01 10:38:32.002 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:32.008 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.009 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.015 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.016 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.016 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.016 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:32.028 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_132 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.037 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_132_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.037 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_132_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.038 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 132 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.038 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.045 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.045 dag-scheduler-event-loop INFO DAGScheduler: Got job 66 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.046 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 74 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.046 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.046 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.046 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 74 (MapPartitionsRDD[255] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.047 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_133 stored as values in memory (estimated size 8.8 KiB, free 2.1 GiB) 20/07/01 10:38:32.049 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_133_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:32.049 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_133_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:32.050 dag-scheduler-event-loop INFO SparkContext: Created broadcast 133 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.050 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 74 (MapPartitionsRDD[255] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.050 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 74.0 with 2 tasks resource profile 0 20/07/01 10:38:32.051 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 74.0 (TID 135, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.051 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 74.0 (TID 136, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7809 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.052 Executor task launch worker for task 135 INFO Executor: Running task 0.0 in stage 74.0 (TID 135) 20/07/01 10:38:32.052 Executor task launch worker for task 136 INFO Executor: Running task 1.0 in stage 74.0 (TID 136) 20/07/01 10:38:32.054 Executor task launch worker for task 136 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 1024-1365, partition values: [empty row] 20/07/01 10:38:32.054 Executor task launch worker for task 135 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/external/avro/target/scala-2.12/test-classes/test.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:32.060 Executor task launch worker for task 136 INFO Executor: Finished task 1.0 in stage 74.0 (TID 136). 1357 bytes result sent to driver 20/07/01 10:38:32.061 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 74.0 (TID 136) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.063 Executor task launch worker for task 135 INFO Executor: Finished task 0.0 in stage 74.0 (TID 135). 2233 bytes result sent to driver 20/07/01 10:38:32.064 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 74.0 (TID 135) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.064 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 74.0, whose tasks have all completed, from pool 20/07/01 10:38:32.064 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 74 (apply at OutcomeOf.scala:85) finished in 0.018 s 20/07/01 10:38:32.064 dag-scheduler-event-loop INFO DAGScheduler: Job 66 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.065 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 74: Stage finished 20/07/01 10:38:32.065 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 66 finished: apply at OutcomeOf.scala:85, took 0.019836 s 20/07/01 10:38:32.077 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:32.083 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.083 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.090 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.090 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.091 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.091 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:32.103 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.103 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.119 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct, complex_map: map>, union_string_null: string, union_int_long_null: bigint ... 10 more fields> 20/07/01 10:38:32.122 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_134 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.130 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_134_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.131 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_134_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.131 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 134 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.132 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.141 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.142 dag-scheduler-event-loop INFO DAGScheduler: Got job 67 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.142 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 75 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.142 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.142 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.142 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 75 (MapPartitionsRDD[260] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.158 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_135 stored as values in memory (estimated size 19.5 KiB, free 2.1 GiB) 20/07/01 10:38:32.160 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_135_piece0 stored as bytes in memory (estimated size 8.6 KiB, free 2.1 GiB) 20/07/01 10:38:32.160 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_135_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 8.6 KiB, free: 2.1 GiB) 20/07/01 10:38:32.160 dag-scheduler-event-loop INFO SparkContext: Created broadcast 135 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.160 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 75 (MapPartitionsRDD[260] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.161 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 75.0 with 2 tasks resource profile 0 20/07/01 10:38:32.162 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 75.0 (TID 137, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7876 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.162 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 75.0 (TID 138, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7876 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.162 Executor task launch worker for task 137 INFO Executor: Running task 0.0 in stage 75.0 (TID 137) 20/07/01 10:38:32.162 Executor task launch worker for task 138 INFO Executor: Running task 1.0 in stage 75.0 (TID 138) 20/07/01 10:38:32.176 Executor task launch worker for task 138 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro, range: 1024-1434, partition values: [empty row] 20/07/01 10:38:32.176 Executor task launch worker for task 137 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:32.182 Executor task launch worker for task 138 INFO Executor: Finished task 1.0 in stage 75.0 (TID 138). 1511 bytes result sent to driver 20/07/01 10:38:32.182 task-result-getter-1 INFO TaskSetManager: Finished task 1.0 in stage 75.0 (TID 138) in 20 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.183 Executor task launch worker for task 137 INFO Executor: Finished task 0.0 in stage 75.0 (TID 137). 1554 bytes result sent to driver 20/07/01 10:38:32.184 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 75.0 (TID 137) in 23 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.184 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 75.0, whose tasks have all completed, from pool 20/07/01 10:38:32.184 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 75 (apply at OutcomeOf.scala:85) finished in 0.041 s 20/07/01 10:38:32.184 dag-scheduler-event-loop INFO DAGScheduler: Job 67 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.184 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 75: Stage finished 20/07/01 10:38:32.184 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 67 finished: apply at OutcomeOf.scala:85, took 0.042945 s 20/07/01 10:38:32.199 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_136 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.208 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_136_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.208 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_136_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.209 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 136 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.209 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.215 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.216 dag-scheduler-event-loop INFO DAGScheduler: Got job 68 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.216 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 76 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.216 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.216 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.216 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 76 (MapPartitionsRDD[263] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.218 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_137 stored as values in memory (estimated size 8.8 KiB, free 2.1 GiB) 20/07/01 10:38:32.219 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_137_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2.1 GiB) 20/07/01 10:38:32.219 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_137_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.7 KiB, free: 2.1 GiB) 20/07/01 10:38:32.219 dag-scheduler-event-loop INFO SparkContext: Created broadcast 137 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.220 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 76 (MapPartitionsRDD[263] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.220 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 76.0 with 2 tasks resource profile 0 20/07/01 10:38:32.221 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 76.0 (TID 139, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7876 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.221 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 76.0 (TID 140, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7876 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.221 Executor task launch worker for task 140 INFO Executor: Running task 1.0 in stage 76.0 (TID 140) 20/07/01 10:38:32.221 Executor task launch worker for task 139 INFO Executor: Running task 0.0 in stage 76.0 (TID 139) 20/07/01 10:38:32.223 Executor task launch worker for task 140 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro, range: 1024-1434, partition values: [empty row] 20/07/01 10:38:32.223 Executor task launch worker for task 139 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro, range: 0-1024, partition values: [empty row] 20/07/01 10:38:32.230 Executor task launch worker for task 140 INFO Executor: Finished task 1.0 in stage 76.0 (TID 140). 1357 bytes result sent to driver 20/07/01 10:38:32.231 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 76.0 (TID 140) in 10 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.232 Executor task launch worker for task 139 INFO Executor: Finished task 0.0 in stage 76.0 (TID 139). 2233 bytes result sent to driver 20/07/01 10:38:32.233 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 76.0 (TID 139) in 13 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.233 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 76.0, whose tasks have all completed, from pool 20/07/01 10:38:32.233 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 76 (apply at OutcomeOf.scala:85) finished in 0.016 s 20/07/01 10:38:32.233 dag-scheduler-event-loop INFO DAGScheduler: Job 68 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.233 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 76: Stage finished 20/07/01 10:38:32.233 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 68 finished: apply at OutcomeOf.scala:85, took 0.017763 s 20/07/01 10:38:32.247 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_138 stored as values in memory (estimated size 294.6 KiB, free 2.1 GiB) 20/07/01 10:38:32.256 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_138_piece0 stored as bytes in memory (estimated size 27.2 KiB, free 2.1 GiB) 20/07/01 10:38:32.256 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_138_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 27.2 KiB, free: 2.1 GiB) 20/07/01 10:38:32.257 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 138 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.316 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileInputFormat: Total input files to process : 1 20/07/01 10:38:32.333 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.334 dag-scheduler-event-loop INFO DAGScheduler: Got job 69 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.334 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 77 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.334 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.334 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.334 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 77 (/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro MapPartitionsRDD[265] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.337 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_139 stored as values in memory (estimated size 4.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.338 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_139_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 2.1 GiB) 20/07/01 10:38:32.338 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_139_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 2.4 KiB, free: 2.1 GiB) 20/07/01 10:38:32.339 dag-scheduler-event-loop INFO SparkContext: Created broadcast 139 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.339 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 77 (/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro MapPartitionsRDD[265] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.339 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 77.0 with 2 tasks resource profile 0 20/07/01 10:38:32.347 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 77.0 (TID 141, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7522 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.348 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 77.0 (TID 142, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7522 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.348 Executor task launch worker for task 141 INFO Executor: Running task 0.0 in stage 77.0 (TID 141) 20/07/01 10:38:32.348 Executor task launch worker for task 142 INFO Executor: Running task 1.0 in stage 77.0 (TID 142) 20/07/01 10:38:32.356 Executor task launch worker for task 141 INFO HadoopRDD: Input split: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro:0+717 20/07/01 10:38:32.356 Executor task launch worker for task 142 INFO HadoopRDD: Input split: file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-7f20d1c0-26d1-42bb-9028-79d50bcfdf44/namedAvro/part-00000-833ee533-652a-47a3-8c23-f167df45af59-c000.avro:717+717 20/07/01 10:38:32.369 Executor task launch worker for task 141 INFO Executor: Finished task 0.0 in stage 77.0 (TID 141). 1880 bytes result sent to driver 20/07/01 10:38:32.369 Executor task launch worker for task 142 INFO Executor: Finished task 1.0 in stage 77.0 (TID 142). 1533 bytes result sent to driver 20/07/01 10:38:32.369 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 77.0 (TID 141) in 29 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.369 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 77.0 (TID 142) in 22 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.370 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 77.0, whose tasks have all completed, from pool 20/07/01 10:38:32.370 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 77 (apply at OutcomeOf.scala:85) finished in 0.034 s 20/07/01 10:38:32.370 dag-scheduler-event-loop INFO DAGScheduler: Job 69 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.370 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 77: Stage finished 20/07/01 10:38:32.370 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 69 finished: apply at OutcomeOf.scala:85, took 0.037087 s 20/07/01 10:38:32.373 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== FINISHED o.a.s.sql.avro.AvroV1Suite: 'conversion to avro and back with namespace' ===== 20/07/01 10:38:32.374 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroV1Suite: ===== TEST OUTPUT FOR o.a.s.sql.avro.AvroV1Suite: 'converting some specific sparkSQL types to avro' ===== 20/07/01 10:38:32.385 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.386 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.400 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO AvroUtils: Compressing Avro output using the snappy codec 20/07/01 10:38:32.400 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:32.400 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:32.400 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:32.423 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 6.535877 ms 20/07/01 10:38:32.440 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.441 dag-scheduler-event-loop INFO DAGScheduler: Got job 70 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.441 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 78 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.441 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.441 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.442 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 78 (MapPartitionsRDD[268] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.522 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_140 stored as values in memory (estimated size 180.7 KiB, free 2.1 GiB) 20/07/01 10:38:32.523 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_140_piece0 stored as bytes in memory (estimated size 64.6 KiB, free 2.1 GiB) 20/07/01 10:38:32.524 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_140_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 64.6 KiB, free: 2.1 GiB) 20/07/01 10:38:32.524 dag-scheduler-event-loop INFO SparkContext: Created broadcast 140 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.524 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 78 (MapPartitionsRDD[268] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.524 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 78.0 with 2 tasks resource profile 0 20/07/01 10:38:32.525 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 78.0 (TID 143, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7608 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.526 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 78.0 (TID 144, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 8497 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.526 Executor task launch worker for task 144 INFO Executor: Running task 1.0 in stage 78.0 (TID 144) 20/07/01 10:38:32.526 Executor task launch worker for task 143 INFO Executor: Running task 0.0 in stage 78.0 (TID 143) 20/07/01 10:38:32.555 Executor task launch worker for task 143 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:32.555 Executor task launch worker for task 144 INFO FileOutputCommitter: File Output Committer Algorithm version is 2 20/07/01 10:38:32.555 Executor task launch worker for task 143 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:32.555 Executor task launch worker for task 144 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 20/07/01 10:38:32.555 Executor task launch worker for task 143 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:32.555 Executor task launch worker for task 144 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/07/01 10:38:32.585 Executor task launch worker for task 144 INFO CodeGenerator: Code generated in 24.537658 ms 20/07/01 10:38:32.590 Executor task launch worker for task 143 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103832_0078_m_000000_143' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro 20/07/01 10:38:32.590 Executor task launch worker for task 143 INFO SparkHadoopMapRedUtil: attempt_20200701103832_0078_m_000000_143: Committed 20/07/01 10:38:32.591 Executor task launch worker for task 143 INFO Executor: Finished task 0.0 in stage 78.0 (TID 143). 2184 bytes result sent to driver 20/07/01 10:38:32.592 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 78.0 (TID 143) in 67 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.599 Executor task launch worker for task 144 INFO FileOutputCommitter: Saved output of task 'attempt_20200701103832_0078_m_000001_144' to file:/home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro 20/07/01 10:38:32.599 Executor task launch worker for task 144 INFO SparkHadoopMapRedUtil: attempt_20200701103832_0078_m_000001_144: Committed 20/07/01 10:38:32.600 Executor task launch worker for task 144 INFO Executor: Finished task 1.0 in stage 78.0 (TID 144). 2184 bytes result sent to driver 20/07/01 10:38:32.601 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 78.0 (TID 144) in 75 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.601 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 78.0, whose tasks have all completed, from pool 20/07/01 10:38:32.601 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 78 (apply at OutcomeOf.scala:85) finished in 0.159 s 20/07/01 10:38:32.601 dag-scheduler-event-loop INFO DAGScheduler: Job 70 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.601 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 78: Stage finished 20/07/01 10:38:32.602 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 70 finished: apply at OutcomeOf.scala:85, took 0.160907 s 20/07/01 10:38:32.607 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Write Job 768ae725-12fc-40f1-b576-0820790bad98 committed. 20/07/01 10:38:32.607 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileFormatWriter: Finished processing stats for write job 768ae725-12fc-40f1-b576-0820790bad98. 20/07/01 10:38:32.613 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 0 ms to list leaf files for 1 paths. 20/07/01 10:38:32.619 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.619 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.626 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.626 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.626 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.626 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:32.632 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_141 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.643 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_141_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.643 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_141_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.644 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 141 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.644 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.652 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.653 dag-scheduler-event-loop INFO DAGScheduler: Got job 71 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.653 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 79 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.653 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.653 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.653 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 79 (MapPartitionsRDD[273] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.655 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_142 stored as values in memory (estimated size 8.3 KiB, free 2.1 GiB) 20/07/01 10:38:32.656 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_142_piece0 stored as bytes in memory (estimated size 4.4 KiB, free 2.1 GiB) 20/07/01 10:38:32.656 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_142_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.4 KiB, free: 2.1 GiB) 20/07/01 10:38:32.657 dag-scheduler-event-loop INFO SparkContext: Created broadcast 142 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.657 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 79 (MapPartitionsRDD[273] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.657 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 79.0 with 2 tasks resource profile 0 20/07/01 10:38:32.658 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 79.0 (TID 145, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.658 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 79.0 (TID 146, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.659 Executor task launch worker for task 145 INFO Executor: Running task 0.0 in stage 79.0 (TID 145) 20/07/01 10:38:32.659 Executor task launch worker for task 146 INFO Executor: Running task 1.0 in stage 79.0 (TID 146) 20/07/01 10:38:32.663 Executor task launch worker for task 145 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00001-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-543, partition values: [empty row] 20/07/01 10:38:32.663 Executor task launch worker for task 146 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00000-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-525, partition values: [empty row] 20/07/01 10:38:32.674 Executor task launch worker for task 145 INFO CodeGenerator: Code generated in 9.674843 ms 20/07/01 10:38:32.680 Executor task launch worker for task 145 INFO Executor: Finished task 0.0 in stage 79.0 (TID 145). 1502 bytes result sent to driver 20/07/01 10:38:32.680 Executor task launch worker for task 146 INFO Executor: Finished task 1.0 in stage 79.0 (TID 146). 1464 bytes result sent to driver 20/07/01 10:38:32.681 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 79.0 (TID 145) in 23 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.681 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 79.0 (TID 146) in 23 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.681 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 79.0, whose tasks have all completed, from pool 20/07/01 10:38:32.682 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 79 (apply at OutcomeOf.scala:85) finished in 0.028 s 20/07/01 10:38:32.682 dag-scheduler-event-loop INFO DAGScheduler: Job 71 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.682 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 79: Stage finished 20/07/01 10:38:32.682 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 71 finished: apply at OutcomeOf.scala:85, took 0.029709 s 20/07/01 10:38:32.698 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 13.284893 ms 20/07/01 10:38:32.704 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:32.713 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.713 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.724 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.725 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.725 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.725 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:32.730 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_143 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.740 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_143_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.740 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_143_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.741 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 143 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.741 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.749 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.749 dag-scheduler-event-loop INFO DAGScheduler: Got job 72 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.750 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 80 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.750 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.750 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.750 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 80 (MapPartitionsRDD[276] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.752 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_144 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:32.753 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_144_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.753 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_144_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:32.754 dag-scheduler-event-loop INFO SparkContext: Created broadcast 144 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.754 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 80 (MapPartitionsRDD[276] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.754 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 80.0 with 2 tasks resource profile 0 20/07/01 10:38:32.755 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 80.0 (TID 147, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.755 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 80.0 (TID 148, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.755 Executor task launch worker for task 148 INFO Executor: Running task 1.0 in stage 80.0 (TID 148) 20/07/01 10:38:32.755 Executor task launch worker for task 147 INFO Executor: Running task 0.0 in stage 80.0 (TID 147) 20/07/01 10:38:32.759 Executor task launch worker for task 147 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00001-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-543, partition values: [empty row] 20/07/01 10:38:32.759 Executor task launch worker for task 148 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00000-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-525, partition values: [empty row] 20/07/01 10:38:32.762 Executor task launch worker for task 147 INFO Executor: Finished task 0.0 in stage 80.0 (TID 147). 1426 bytes result sent to driver 20/07/01 10:38:32.762 Executor task launch worker for task 148 INFO Executor: Finished task 1.0 in stage 80.0 (TID 148). 1417 bytes result sent to driver 20/07/01 10:38:32.763 task-result-getter-3 INFO TaskSetManager: Finished task 0.0 in stage 80.0 (TID 147) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.763 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 80.0 (TID 148) in 8 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.763 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 80.0, whose tasks have all completed, from pool 20/07/01 10:38:32.764 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 80 (apply at OutcomeOf.scala:85) finished in 0.013 s 20/07/01 10:38:32.764 dag-scheduler-event-loop INFO DAGScheduler: Job 72 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.764 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 80: Stage finished 20/07/01 10:38:32.764 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 72 finished: apply at OutcomeOf.scala:85, took 0.014946 s 20/07/01 10:38:32.771 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:32.779 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.779 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.789 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.789 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.789 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.789 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:32.794 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_145 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.803 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_145_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.803 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_145_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.804 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 145 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.804 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.811 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.812 dag-scheduler-event-loop INFO DAGScheduler: Got job 73 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.812 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 81 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.812 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.812 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.812 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 81 (MapPartitionsRDD[279] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.814 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_146 stored as values in memory (estimated size 7.8 KiB, free 2.1 GiB) 20/07/01 10:38:32.815 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_146_piece0 stored as bytes in memory (estimated size 4.2 KiB, free 2.1 GiB) 20/07/01 10:38:32.815 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_146_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.2 KiB, free: 2.1 GiB) 20/07/01 10:38:32.815 dag-scheduler-event-loop INFO SparkContext: Created broadcast 146 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.816 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 81 (MapPartitionsRDD[279] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.816 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 81.0 with 2 tasks resource profile 0 20/07/01 10:38:32.817 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 0.0 in stage 81.0 (TID 149, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.817 dispatcher-event-loop-1 INFO TaskSetManager: Starting task 1.0 in stage 81.0 (TID 150, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.817 Executor task launch worker for task 149 INFO Executor: Running task 0.0 in stage 81.0 (TID 149) 20/07/01 10:38:32.817 Executor task launch worker for task 150 INFO Executor: Running task 1.0 in stage 81.0 (TID 150) 20/07/01 10:38:32.820 Executor task launch worker for task 149 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00001-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-543, partition values: [empty row] 20/07/01 10:38:32.820 Executor task launch worker for task 150 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00000-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-525, partition values: [empty row] 20/07/01 10:38:32.829 Executor task launch worker for task 150 INFO CodeGenerator: Code generated in 7.399777 ms 20/07/01 10:38:32.834 Executor task launch worker for task 149 INFO Executor: Finished task 0.0 in stage 81.0 (TID 149). 1422 bytes result sent to driver 20/07/01 10:38:32.834 Executor task launch worker for task 150 INFO Executor: Finished task 1.0 in stage 81.0 (TID 150). 1411 bytes result sent to driver 20/07/01 10:38:32.834 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 81.0 (TID 149) in 18 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.835 task-result-getter-2 INFO TaskSetManager: Finished task 1.0 in stage 81.0 (TID 150) in 17 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.835 task-result-getter-2 INFO TaskSchedulerImpl: Removed TaskSet 81.0, whose tasks have all completed, from pool 20/07/01 10:38:32.835 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 81 (apply at OutcomeOf.scala:85) finished in 0.022 s 20/07/01 10:38:32.835 dag-scheduler-event-loop INFO DAGScheduler: Job 73 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.835 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 81: Stage finished 20/07/01 10:38:32.835 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 73 finished: apply at OutcomeOf.scala:85, took 0.023605 s 20/07/01 10:38:32.843 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO CodeGenerator: Code generated in 5.483467 ms 20/07/01 10:38:32.849 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:32.857 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.857 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.866 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.867 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.867 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.867 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:32.871 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_147 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.880 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_147_piece0 stored as bytes in memory (estimated size 28.1 KiB, free 2.1 GiB) 20/07/01 10:38:32.881 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_147_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 28.1 KiB, free: 2.1 GiB) 20/07/01 10:38:32.881 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Created broadcast 147 from apply at OutcomeOf.scala:85 20/07/01 10:38:32.882 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceScanExec: Planning scan with bin packing, max size: 1024 bytes, open cost is considered as scanning 4194304 bytes. 20/07/01 10:38:32.889 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO SparkContext: Starting job: apply at OutcomeOf.scala:85 20/07/01 10:38:32.890 dag-scheduler-event-loop INFO DAGScheduler: Got job 74 (apply at OutcomeOf.scala:85) with 2 output partitions 20/07/01 10:38:32.890 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 82 (apply at OutcomeOf.scala:85) 20/07/01 10:38:32.890 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List() 20/07/01 10:38:32.890 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List() 20/07/01 10:38:32.890 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 82 (MapPartitionsRDD[282] at apply at OutcomeOf.scala:85), which has no missing parents 20/07/01 10:38:32.891 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_148 stored as values in memory (estimated size 7.5 KiB, free 2.1 GiB) 20/07/01 10:38:32.892 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_148_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.893 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_148_piece0 in memory on amp-jenkins-worker-04.amp:35025 (size: 4.0 KiB, free: 2.1 GiB) 20/07/01 10:38:32.893 dag-scheduler-event-loop INFO SparkContext: Created broadcast 148 from broadcast at DAGScheduler.scala:1293 20/07/01 10:38:32.893 dag-scheduler-event-loop INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 82 (MapPartitionsRDD[282] at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0, 1)) 20/07/01 10:38:32.893 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 82.0 with 2 tasks resource profile 0 20/07/01 10:38:32.894 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 0.0 in stage 82.0 (TID 151, amp-jenkins-worker-04.amp, executor driver, partition 0, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.894 dispatcher-event-loop-0 INFO TaskSetManager: Starting task 1.0 in stage 82.0 (TID 152, amp-jenkins-worker-04.amp, executor driver, partition 1, PROCESS_LOCAL, 7871 bytes) taskResourceAssignments Map() 20/07/01 10:38:32.895 Executor task launch worker for task 152 INFO Executor: Running task 1.0 in stage 82.0 (TID 152) 20/07/01 10:38:32.895 Executor task launch worker for task 151 INFO Executor: Running task 0.0 in stage 82.0 (TID 151) 20/07/01 10:38:32.897 Executor task launch worker for task 152 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00000-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-525, partition values: [empty row] 20/07/01 10:38:32.897 Executor task launch worker for task 151 INFO FileScanRDD: Reading File path: file:///home/jenkins/workspace/NewSparkPullRequestBuilder/target/tmp/spark-29f72501-e3a5-48e6-ad76-bbafc617500f/avro/part-00001-89929a99-41ea-45b5-91d1-66ca7590f024-c000.avro, range: 0-543, partition values: [empty row] 20/07/01 10:38:32.901 Executor task launch worker for task 152 INFO Executor: Finished task 1.0 in stage 82.0 (TID 152). 1417 bytes result sent to driver 20/07/01 10:38:32.901 Executor task launch worker for task 151 INFO Executor: Finished task 0.0 in stage 82.0 (TID 151). 1422 bytes result sent to driver 20/07/01 10:38:32.901 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 82.0 (TID 152) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (1/2) 20/07/01 10:38:32.901 task-result-getter-0 INFO TaskSetManager: Finished task 0.0 in stage 82.0 (TID 151) in 7 ms on amp-jenkins-worker-04.amp (executor driver) (2/2) 20/07/01 10:38:32.902 task-result-getter-0 INFO TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool 20/07/01 10:38:32.902 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 82 (apply at OutcomeOf.scala:85) finished in 0.011 s 20/07/01 10:38:32.902 dag-scheduler-event-loop INFO DAGScheduler: Job 74 is finished. Cancelling potential speculative or zombie tasks for this job 20/07/01 10:38:32.902 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 82: Stage finished 20/07/01 10:38:32.902 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO DAGScheduler: Job 74 finished: apply at OutcomeOf.scala:85, took 0.012959 s 20/07/01 10:38:32.909 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO InMemoryFileIndex: It took 1 ms to list leaf files for 1 paths. 20/07/01 10:38:32.917 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.917 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO BaseSessionStateBuilder$$anon$2: Optimization rule 'org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation' is excluded from the optimizer. 20/07/01 10:38:32.926 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pruning directories with: 20/07/01 10:38:32.926 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Pushed Filters: 20/07/01 10:38:32.926 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Post-Scan Filters: 20/07/01 10:38:32.926 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO FileSourceStrategy: Output Data Schema: struct 20/07/01 10:38:32.930 pool-1-thread-1-ScalaTest-running-AvroV1Suite INFO MemoryStore: Block broadcast_149 stored as values in memory (estimated size 308.0 KiB, free 2.1 GiB) 20/07/01 10:38:32.939 pool-1-thread-1-ScalaTest-running-AvroV1Sui