Started by upstream project "quinine" build number 3934 originally caused by: Started by upstream project "ADAM" build number 4197 originally caused by: Started by timer [EnvInject] - Loading node environment variables. Building remotely on amp-jenkins-worker-01 (spark-packaging spark-compile spark-experimental-docker spark-release spark-docs centos) in workspace /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/quinine.git # timeout=10 Fetching upstream changes from https://github.com/bigdatagenomics/quinine.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/quinine.git +refs/heads/*:refs/remotes/origin/* Checking out Revision 31b91e2d60ebedf732f8cf34d3dc1bf78bb428af (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 31b91e2d60ebedf732f8cf34d3dc1bf78bb428af > git rev-list 31b91e2d60ebedf732f8cf34d3dc1bf78bb428af # timeout=10 [centos] $ /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.1.1/bin/mvn -Dlabel=centos -Dhadoop.version=1.0.4 clean package [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.bdgenomics.quinine:quinine-core_2.10:jar:0.0.3-SNAPSHOT [WARNING] 'dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 370, column 25 [WARNING] 'dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 374, column 25 [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 370, column 25 [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 374, column 25 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.bdgenomics.quinine:quinine-cli_2.10:jar:0.0.3-SNAPSHOT [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 370, column 25 [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ org.bdgenomics.quinine:quinine-parent_2.10:0.0.3-SNAPSHOT, /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/pom.xml, line 374, column 25 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.bdgenomics.quinine:quinine-parent_2.10:pom:0.0.3-SNAPSHOT [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ line 370, column 25 [WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.apache.hadoop:hadoop-client:jar with value '*' does not match a valid id pattern. @ line 374, column 25 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] quinine [INFO] quinine: Core [INFO] quinine: CLI [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building quinine 0.0.3-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ quinine-parent_2.10 --- [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-versions) @ quinine-parent_2.10 --- [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ quinine-parent_2.10 --- [INFO] [INFO] --- scalariform-maven-plugin:0.1.4:format (default-cli) @ quinine-parent_2.10 --- [INFO] Modified 0 of 54 .scala files [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ quinine-parent_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ quinine-parent_2.10 --- [INFO] No sources to compile [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building quinine: Core 0.0.3-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ quinine-core_2.10 --- [INFO] Deleting /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-versions) @ quinine-core_2.10 --- [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ quinine-core_2.10 --- [INFO] [INFO] --- build-helper-maven-plugin:1.9.1:add-source (add-source) @ quinine-core_2.10 --- [INFO] Source directory: /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/src/main/scala added. [INFO] [INFO] --- scalariform-maven-plugin:0.1.4:format (default-cli) @ quinine-core_2.10 --- [INFO] Modified 0 of 41 .scala files [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ quinine-core_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/src/main/resources [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ quinine-core_2.10 --- [WARNING] Expected all dependencies to require Scala version: 2.10.4 [WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-remote_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-actor_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-slf4j_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] org.apache.spark:spark-core_2.10:1.6.1 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/src/main/scala:-1: info: compiling [INFO] Compiling 24 source files to /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/classes at 1551676874516 [WARNING] warning: there were 2 feature warning(s); re-run with -feature for details [WARNING] one warning found [INFO] prepare-compile in 0 s [INFO] compile in 13 s [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ quinine-core_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- build-helper-maven-plugin:1.9.1:add-test-source (add-test-source) @ quinine-core_2.10 --- [INFO] Test Source directory: /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/src/test/scala added. [INFO] [INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ quinine-core_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ quinine-core_2.10 --- [WARNING] Expected all dependencies to require Scala version: 2.10.4 [WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-remote_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-actor_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] com.typesafe.akka:akka-slf4j_2.10:2.3.11 requires scala version: 2.10.4 [WARNING] org.apache.spark:spark-core_2.10:1.6.1 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version: 2.10.4 [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: 2.10.0 [WARNING] Multiple versions of scala libraries detected! [INFO] /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/src/test/scala:-1: info: compiling [INFO] Compiling 17 source files to /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes at 1551676888335 [INFO] prepare-compile in 0 s [INFO] compile in 15 s [INFO] [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ quinine-core_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ quinine-core_2.10 --- [INFO] Tests are skipped. [INFO] [INFO] --- scalatest-maven-plugin:1.0:test (test) @ quinine-core_2.10 --- Discovery starting. Discovery completed in 521 milliseconds. Run starting. Expected test count is: 74 RNASeqPanelStatsSuite: 19/03/03 21:21:45 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:45 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:45 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:45 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:46 INFO util.Utils: Successfully started service 'sparkDriver' on port 54227. 19/03/03 21:21:46 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:46 INFO Remoting: Starting remoting 19/03/03 21:21:46 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.10.21:54228] 19/03/03 21:21:46 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 54228. 19/03/03 21:21:46 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:46 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:46 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-7a7a9949-a985-4418-b5c2-a078a48664b3 19/03/03 21:21:46 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB 19/03/03 21:21:47 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:47 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:47 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34084. 19/03/03 21:21:47 INFO netty.NettyBlockTransferService: Server created on 34084 19/03/03 21:21:47 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:47 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:34084 with 511.1 MB RAM, BlockManagerId(driver, localhost, 34084) 19/03/03 21:21:47 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:47 INFO spark.SparkContext: Starting job: reduce at RNASeqReadCounts.scala:80 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Got job 0 (reduce at RNASeqReadCounts.scala:80) with 4 output partitions 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79), which has no missing parents 19/03/03 21:21:47 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 2.0 KB) 19/03/03 21:21:47 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1324.0 B, free 3.3 KB) 19/03/03 21:21:47 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:34084 (size: 1324.0 B, free: 511.1 MB) 19/03/03 21:21:47 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:47 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79) 19/03/03 21:21:47 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:47 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2023 bytes) 19/03/03 21:21:47 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:47 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:47 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:47 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:47 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:47 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:47 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_0_3 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_0_0 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_0_2 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_0_1 not found, computing it 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_0_0 stored as values in memory (estimated size 16.0 B, free 3.4 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_0_0 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_0_3 stored as values in memory (estimated size 352.0 B, free 3.7 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_0_2 stored as values in memory (estimated size 352.0 B, free 4.0 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_0_3 in memory on localhost:34084 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_0_1 stored as values in memory (estimated size 352.0 B, free 4.4 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_0_2 in memory on localhost:34084 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_0_1 in memory on localhost:34084 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1488 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1548 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1548 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1548 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 140 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 106 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 125 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 115 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) finished in 0.164 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Job 0 finished: reduce at RNASeqReadCounts.scala:80, took 0.483611 s 19/03/03 21:21:48 INFO spark.SparkContext: Starting job: collect at RightOuterJoin.scala:77 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Registering RDD 7 (keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Got job 1 (collect at RightOuterJoin.scala:77) with 4 output partitions 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (collect at RightOuterJoin.scala:77) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.0 KB, free 8.4 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.1 KB, free 10.5 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:34084 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:34084 in memory (size: 1324.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.ContextCleaner: Cleaned accumulator 1 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_1_0 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_1_1 not found, computing it 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_1_1 stored as values in memory (estimated size 16.0 B, free 7.2 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_1_0 stored as values in memory (estimated size 16.0 B, free 7.2 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_1_1 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_1_0 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_1_2 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_1_3 not found, computing it 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_1_2 stored as values in memory (estimated size 16.0 B, free 7.2 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_1_2 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_1_3 stored as values in memory (estimated size 4.7 KB, free 11.9 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_1_3 in memory on localhost:34084 (size: 4.7 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1670 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1670 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1670 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 95 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 76 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 69 ms on localhost (3/4) 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1670 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 96 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at RightOuterJoin.scala:75) finished in 0.131 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:48 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.8 KB, free 16.7 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 19.1 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:34084 (size: 2.5 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1228 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1124 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1124 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1124 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 75 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 77 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 74 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 76 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ResultStage 2 (collect at RightOuterJoin.scala:77) finished in 0.079 s 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Job 1 finished: collect at RightOuterJoin.scala:77, took 0.298965 s 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 216.0 B, free 19.4 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 124.0 B, free 19.5 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:34084 (size: 124.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 3 from broadcast at RightOuterJoin.scala:86 19/03/03 21:21:48 INFO spark.SparkContext: Starting job: reduce at RNASeqAlignmentCounts.scala:72 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Registering RDD 9 (map at RightOuterJoin.scala:90) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Registering RDD 10 (flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Got job 2 (reduce at RNASeqAlignmentCounts.scala:72) with 4 output partitions 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.4 KB, free 22.9 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2039.0 B, free 24.8 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:34084 (size: 2039.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.7 KB, free 28.5 KB) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 30.6 KB) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:34084 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_0_0 locally 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 76 ms on localhost (1/4) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_0_1 locally 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 94 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 100 ms on localhost (3/4) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_0_2 locally 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 88 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (map at RightOuterJoin.scala:90) finished in 0.107 s 19/03/03 21:21:48 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:48 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 50 ms on localhost (1/4) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_0_3 locally 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 63 ms on localhost (2/4) 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 62 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 58 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (flatMap at RightOuterJoin.scala:94) finished in 0.122 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:48 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.8 KB, free 34.5 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2047.0 B, free 36.5 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:34084 (size: 2047.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 20) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 22) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 21) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 23) 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_14_0 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_14_2 not found, computing it 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_14_3 not found, computing it 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO spark.CacheManager: Partition rdd_14_1 not found, computing it 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_14_0 stored as values in memory (estimated size 16.0 B, free 36.5 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_14_2 stored as values in memory (estimated size 16.0 B, free 36.5 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_14_0 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_14_2 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_14_3 stored as values in memory (estimated size 16.0 B, free 36.5 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_14_3 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 20). 1704 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 22). 1704 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 23). 1704 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 20) in 47 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 22) in 45 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 23) in 48 ms on localhost (3/4) 19/03/03 21:21:48 INFO storage.MemoryStore: Block rdd_14_1 stored as values in memory (estimated size 5.7 KB, free 42.2 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added rdd_14_1 in memory on localhost:34084 (size: 5.7 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 21). 1767 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 21) in 78 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) finished in 0.080 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Job 2 finished: reduce at RNASeqAlignmentCounts.scala:72, took 0.261326 s 19/03/03 21:21:48 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:43 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Got job 3 (count at TranscriptCounts.scala:43) with 4 output partitions 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at TranscriptCounts.scala:43) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 1808.0 B, free 44.0 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1186.0 B, free 45.1 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:34084 (size: 1186.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 27) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 24) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 26) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 25) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 27). 2057 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 26). 2057 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 25). 2057 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 24). 2057 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 27) in 22 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 26) in 27 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 24) in 40 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 25) in 35 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ResultStage 6 (count at TranscriptCounts.scala:43) finished in 0.040 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Job 3 finished: count at TranscriptCounts.scala:43, took 0.051520 s 19/03/03 21:21:48 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:51 19/03/03 21:21:48 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 162 bytes 19/03/03 21:21:48 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 157 bytes 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Registering RDD 19 (distinct at TranscriptCounts.scala:50) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Got job 4 (count at TranscriptCounts.scala:51) with 4 output partitions 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (count at TranscriptCounts.scala:51) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 9) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 9) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.0 KB, free 49.1 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.1 KB, free 51.2 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:34084 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 28, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 29, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 9.0 (TID 30, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 9.0 (TID 31, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 9.0 (TID 30) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 28) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 9.0 (TID 31) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 29) 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_14_3 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_14_2 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_14_0 locally 19/03/03 21:21:48 INFO storage.BlockManager: Found block rdd_14_1 locally 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 9.0 (TID 31). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 9.0 (TID 30). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 28). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 9.0 (TID 29). 2240 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 9.0 (TID 31) in 33 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 9.0 (TID 30) in 36 ms on localhost (2/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 28) in 37 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 29) in 37 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ShuffleMapStage 9 (distinct at TranscriptCounts.scala:50) finished in 0.038 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:48 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 10) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.1 KB, free 54.3 KB) 19/03/03 21:21:48 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1856.0 B, free 56.1 KB) 19/03/03 21:21:48 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:34084 (size: 1856.0 B, free: 511.1 MB) 19/03/03 21:21:48 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Adding task set 10.0 with 4 tasks 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 10.0 (TID 32, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 33, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 10.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 10.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:48 INFO executor.Executor: Running task 1.0 in stage 10.0 (TID 32) 19/03/03 21:21:48 INFO executor.Executor: Running task 2.0 in stage 10.0 (TID 34) 19/03/03 21:21:48 INFO executor.Executor: Running task 3.0 in stage 10.0 (TID 35) 19/03/03 21:21:48 INFO executor.Executor: Running task 0.0 in stage 10.0 (TID 33) 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:48 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:48 INFO executor.Executor: Finished task 3.0 in stage 10.0 (TID 35). 1123 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 2.0 in stage 10.0 (TID 34). 1123 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 10.0 (TID 35) in 19 ms on localhost (1/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 10.0 (TID 34) in 20 ms on localhost (2/4) 19/03/03 21:21:48 INFO executor.Executor: Finished task 0.0 in stage 10.0 (TID 33). 1123 bytes result sent to driver 19/03/03 21:21:48 INFO executor.Executor: Finished task 1.0 in stage 10.0 (TID 32). 1123 bytes result sent to driver 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 33) in 26 ms on localhost (3/4) 19/03/03 21:21:48 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 10.0 (TID 32) in 27 ms on localhost (4/4) 19/03/03 21:21:48 INFO scheduler.DAGScheduler: ResultStage 10 (count at TranscriptCounts.scala:51) finished in 0.028 s 19/03/03 21:21:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 19/03/03 21:21:48 INFO scheduler.DAGScheduler: Job 4 finished: count at TranscriptCounts.scala:51, took 0.104200 s 19/03/03 21:21:48 INFO rdd.ParallelCollectionRDD: Removing RDD 0 from persistence list 19/03/03 21:21:48 INFO storage.BlockManager: Removing RDD 0 19/03/03 21:21:49 INFO spark.SparkContext: Starting job: collect at BroadcastRegionJoin.scala:100 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 28 (keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Got job 5 (collect at BroadcastRegionJoin.scala:100) with 4 output partitions 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 11) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 11) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.2 KB, free 59.3 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.2 KB, free 61.5 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:34084 (size: 2.2 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 36, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 37, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 38, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 11.0 (TID 39, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 36) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 11.0 (TID 39) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 38) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 37) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:34084 in memory (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:34084 in memory (size: 1856.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 9 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:34084 in memory (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 8 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:34084 in memory (size: 1186.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 7 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:34084 in memory (size: 2047.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 37). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 6 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 5 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 38). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 36). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:34084 in memory (size: 2039.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 4 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 36) in 62 ms on localhost (1/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 38) in 56 ms on localhost (2/4) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:34084 in memory (size: 2.5 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 37) in 60 ms on localhost (3/4) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 3 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:34084 in memory (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned accumulator 2 19/03/03 21:21:49 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 11.0 (TID 39). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 11.0 (TID 39) in 60 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 11 (keyBy at BroadcastRegionJoin.scala:98) finished in 0.072 s 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 12) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 5.0 KB, free 22.2 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.5 KB, free 24.7 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:34084 (size: 2.5 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 40, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 41, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 42, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 43, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 40) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 42) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 41) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 43) 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 43). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 40). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 41). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 42). 1228 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 43) in 18 ms on localhost (1/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 40) in 20 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 41) in 22 ms on localhost (3/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 42) in 21 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) finished in 0.024 s 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Job 5 finished: collect at BroadcastRegionJoin.scala:100, took 0.120794 s 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 216.0 B, free 24.9 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 124.0 B, free 25.0 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:34084 (size: 124.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 12 from broadcast at BroadcastRegionJoin.scala:109 19/03/03 21:21:49 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 22 (flatMap at ReadCoverage.scala:59) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 32 (flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 30 (map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 39 (map at CoverageStats.scala:172) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Got job 6 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (collect at CoverageStats.scala:174) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 2.7 KB, free 27.8 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 1677.0 B, free 29.4 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:34084 (size: 1677.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 3.6 KB, free 33.0 KB) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 44, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 13.0 (TID 45, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.1 KB, free 35.1 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:34084 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 13.0 (TID 46, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 13.0 (TID 47, localhost, partition 3,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 44) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 13.0 (TID 45) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 13.0 (TID 47) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 13.0 (TID 46) 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 44). 1090 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 48) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 44) in 31 ms on localhost (1/4) 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 48). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 49) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 48) in 17 ms on localhost (1/4) 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 49). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 50) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 49) in 17 ms on localhost (2/4) 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 50). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 15.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 15.0 (TID 51) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 50) in 17 ms on localhost (3/4) 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 15.0 (TID 51). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 15.0 (TID 51) in 23 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 15 (map at BroadcastRegionJoin.scala:113) finished in 0.069 s 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 13) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17, ShuffleMapStage 14) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 13.0 (TID 47). 1090 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 13.0 (TID 46). 1090 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 13.0 (TID 45). 1090 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 13.0 (TID 47) in 130 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 13.0 (TID 45) in 136 ms on localhost (3/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 13.0 (TID 46) in 134 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 13 (flatMap at ReadCoverage.scala:59) finished in 0.142 s 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17, ShuffleMapStage 14) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 4.3 KB, free 39.4 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KB, free 41.8 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:34084 (size: 2.4 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 14.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 14.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 14.0 (TID 53, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 14.0 (TID 54, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 14.0 (TID 55, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 14.0 (TID 52) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 14.0 (TID 55) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 14.0 (TID 53) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 14.0 (TID 54) 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_23_0 not found, computing it 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_23_1 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_23_3 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_23_2 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_23_1 stored as values in memory (estimated size 16.0 B, free 41.8 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_23_1 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_23_3 stored as values in memory (estimated size 16.0 B, free 41.9 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_23_3 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 14.0 (TID 54). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 14.0 (TID 55). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 14.0 (TID 54) in 17 ms on localhost (1/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 14.0 (TID 55) in 16 ms on localhost (2/4) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_23_0 stored as values in memory (estimated size 17.1 KB, free 59.0 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_23_0 in memory on localhost:34084 (size: 17.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_23_2 stored as values in memory (estimated size 17.1 KB, free 76.1 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_23_2 in memory on localhost:34084 (size: 17.1 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 14.0 (TID 52). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 14.0 (TID 52) in 130 ms on localhost (3/4) 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 14.0 (TID 53). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 14.0 (TID 53) in 180 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 14 (flatMap at BroadcastRegionJoin.scala:118) finished in 0.182 s 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.1 KB, free 80.2 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.2 KB, free 82.3 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:34084 (size: 2.2 KB, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 56) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 59) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 58) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 57) 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_37_3 not found, computing it 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_37_1 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_37_2 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO spark.CacheManager: Partition rdd_37_0 not found, computing it 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_37_2 stored as values in memory (estimated size 16.0 B, free 82.4 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_37_2 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_37_3 stored as values in memory (estimated size 16.0 B, free 82.4 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_37_3 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_37_0 stored as values in memory (estimated size 16.0 B, free 82.4 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_37_0 in memory on localhost:34084 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 58). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 58) in 21 ms on localhost (1/4) 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 59). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 56). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 59) in 25 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 56) in 27 ms on localhost (3/4) 19/03/03 21:21:49 INFO storage.MemoryStore: Block rdd_37_1 stored as values in memory (estimated size 45.8 KB, free 128.2 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added rdd_37_1 in memory on localhost:34084 (size: 45.8 KB, free: 511.0 MB) 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 57). 1886 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 57) in 110 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (map at CoverageStats.scala:172) finished in 0.111 s 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 2.5 KB, free 130.8 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 1554.0 B, free 132.3 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:34084 (size: 1554.0 B, free: 511.0 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 17.0 (TID 60, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 61, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 17.0 (TID 62, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 17.0 (TID 63, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 17.0 (TID 60) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 17.0 (TID 63) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 61) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 17.0 (TID 62) 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 17.0 (TID 63). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 61). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 17.0 (TID 62). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 17.0 (TID 60). 1131 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 17.0 (TID 63) in 21 ms on localhost (1/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 17.0 (TID 60) in 27 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 17.0 (TID 62) in 23 ms on localhost (3/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 61) in 25 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ResultStage 17 (collect at CoverageStats.scala:174) finished in 0.027 s 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Job 6 finished: collect at CoverageStats.scala:174, took 0.509833 s 19/03/03 21:21:49 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:49 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 8 is 158 bytes 19/03/03 21:21:49 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 161 bytes 19/03/03 21:21:49 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 157 bytes 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Registering RDD 42 (map at CoverageStats.scala:172) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Got job 7 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 22 (collect at CoverageStats.scala:174) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 21) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 21) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 4.2 KB, free 136.5 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.2 KB, free 138.7 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:34084 (size: 2.2 KB, free: 511.0 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 64, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 21.0 (TID 65, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 21.0 (TID 66, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 21.0 (TID 67, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 64) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 21.0 (TID 65) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 21.0 (TID 66) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 21.0 (TID 67) 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_37_0 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_37_1 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_37_3 locally 19/03/03 21:21:49 INFO storage.BlockManager: Found block rdd_37_2 locally 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 64). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 21.0 (TID 67). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 64) in 23 ms on localhost (1/4) 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 21.0 (TID 66). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 21.0 (TID 67) in 18 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 21.0 (TID 66) in 23 ms on localhost (3/4) 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 21.0 (TID 65). 2240 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 21.0 (TID 65) in 29 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ShuffleMapStage 21 (map at CoverageStats.scala:172) finished in 0.031 s 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:49 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 22) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 2.5 KB, free 141.2 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 1554.0 B, free 142.7 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:34084 (size: 1554.0 B, free: 511.0 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 22.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 22.0 (TID 68, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 22.0 (TID 69, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 22.0 (TID 70, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 22.0 (TID 71, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 22.0 (TID 68) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 22.0 (TID 71) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 22.0 (TID 69) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 22.0 (TID 70) 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:49 INFO executor.Executor: Finished task 3.0 in stage 22.0 (TID 71). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 1.0 in stage 22.0 (TID 68). 1131 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 0.0 in stage 22.0 (TID 69). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO executor.Executor: Finished task 2.0 in stage 22.0 (TID 70). 1124 bytes result sent to driver 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 22.0 (TID 71) in 11 ms on localhost (1/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 22.0 (TID 68) in 13 ms on localhost (2/4) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 22.0 (TID 70) in 12 ms on localhost (3/4) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: ResultStage 22 (collect at CoverageStats.scala:174) finished in 0.014 s 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 22.0 (TID 69) in 14 ms on localhost (4/4) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Job 7 finished: collect at CoverageStats.scala:174, took 0.068007 s 19/03/03 21:21:49 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:49 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:49 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:49 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:49 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:49 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:49 INFO spark.SparkContext: Running Spark version 1.6.1 - single transcript, with reads at the start, middle, and end 19/03/03 21:21:49 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:49 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:49 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:49 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:49 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:49 INFO util.Utils: Successfully started service 'sparkDriver' on port 58166. 19/03/03 21:21:49 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:49 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:49 INFO Remoting: Starting remoting 19/03/03 21:21:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58167] 19/03/03 21:21:49 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58167. 19/03/03 21:21:49 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:49 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:49 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-53067c76-dc9d-401d-9935-1e8e884e2e92 19/03/03 21:21:49 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB 19/03/03 21:21:49 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:49 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:49 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40733. 19/03/03 21:21:49 INFO netty.NettyBlockTransferService: Server created on 40733 19/03/03 21:21:49 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:49 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:40733 with 511.1 MB RAM, BlockManagerId(driver, localhost, 40733) 19/03/03 21:21:49 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:49 INFO spark.SparkContext: Starting job: reduce at RNASeqReadCounts.scala:80 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Got job 0 (reduce at RNASeqReadCounts.scala:80) with 4 output partitions 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79), which has no missing parents 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 2.0 KB) 19/03/03 21:21:49 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1324.0 B, free 3.3 KB) 19/03/03 21:21:49 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:40733 (size: 1324.0 B, free: 511.1 MB) 19/03/03 21:21:49 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:49 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79) 19/03/03 21:21:49 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2157 bytes) 19/03/03 21:21:49 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:49 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:49 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:49 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_0_3 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_0_1 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_0_2 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_0_0 not found, computing it 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_0_3 stored as values in memory (estimated size 632.0 B, free 4.0 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_0_3 in memory on localhost:40733 (size: 632.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_0_1 stored as values in memory (estimated size 352.0 B, free 4.3 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_0_0 stored as values in memory (estimated size 352.0 B, free 4.6 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_0_1 in memory on localhost:40733 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_0_2 stored as values in memory (estimated size 352.0 B, free 5.0 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_0_0 in memory on localhost:40733 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_0_2 in memory on localhost:40733 (size: 352.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1548 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1548 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1548 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1548 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 58 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 27 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 56 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 25 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) finished in 0.063 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 0 finished: reduce at RNASeqReadCounts.scala:80, took 0.073960 s 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: collect at RightOuterJoin.scala:77 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 7 (keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 1 (collect at RightOuterJoin.scala:77) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (collect at RightOuterJoin.scala:77) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.0 KB, free 9.0 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.1 KB, free 11.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:40733 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_1_2 not found, computing it 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_1_2 stored as values in memory (estimated size 16.0 B, free 11.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_1_2 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_1_1 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_1_0 not found, computing it 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_1_0 stored as values in memory (estimated size 16.0 B, free 11.1 KB) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_1_3 not found, computing it 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_1_0 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_1_1 stored as values in memory (estimated size 4.7 KB, free 15.9 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_1_1 in memory on localhost:40733 (size: 4.7 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_1_3 stored as values in memory (estimated size 4.7 KB, free 20.6 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_1_3 in memory on localhost:40733 (size: 4.7 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1670 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1670 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 16 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 22 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1670 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1670 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 29 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 26 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at RightOuterJoin.scala:75) finished in 0.033 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.8 KB, free 25.3 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 27.8 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:40733 (size: 2.5 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 13 ms on localhost (1/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 16 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1239 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 14 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 20 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 2 (collect at RightOuterJoin.scala:77) finished in 0.024 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 1 finished: collect at RightOuterJoin.scala:77, took 0.079883 s 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 232.0 B, free 28.0 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 128.0 B, free 28.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:40733 (size: 128.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 3 from broadcast at RightOuterJoin.scala:86 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: reduce at RNASeqAlignmentCounts.scala:72 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 9 (map at RightOuterJoin.scala:90) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 10 (flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 2 (reduce at RNASeqAlignmentCounts.scala:72) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.4 KB, free 31.5 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2039.0 B, free 33.5 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:40733 (size: 2039.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94), which has no missing parents 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.7 KB, free 37.2 KB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 39.3 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:40733 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_0_0 locally 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 25 ms on localhost (1/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_0_1 locally 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 41 ms on localhost (2/4) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_0_2 locally 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2146 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 32 ms on localhost (3/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 43 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (map at RightOuterJoin.scala:90) finished in 0.048 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_0_3 locally 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 23 ms on localhost (1/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 23 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 21 ms on localhost (3/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 24 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (flatMap at RightOuterJoin.scala:94) finished in 0.049 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.8 KB, free 43.1 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2047.0 B, free 45.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:40733 (size: 2047.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 22) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 20) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 21) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 23) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_14_3 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_14_2 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_14_0 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_14_1 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_14_2 stored as values in memory (estimated size 16.0 B, free 45.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_14_2 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_14_3 stored as values in memory (estimated size 16.0 B, free 45.2 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_14_3 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 22). 1704 bytes result sent to driver 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_14_0 stored as values in memory (estimated size 392.0 B, free 45.5 KB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 23). 1704 bytes result sent to driver 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_14_0 in memory on localhost:40733 (size: 392.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 22) in 20 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 23) in 20 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 20). 1767 bytes result sent to driver 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_14_1 stored as values in memory (estimated size 6.0 KB, free 51.5 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_14_1 in memory on localhost:40733 (size: 6.0 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 20) in 30 ms on localhost (3/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 21). 1767 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 21) in 32 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) finished in 0.034 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 2 finished: reduce at RNASeqAlignmentCounts.scala:72, took 0.130485 s 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:43 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 3 (count at TranscriptCounts.scala:43) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at TranscriptCounts.scala:43) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 1808.0 B, free 53.3 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1186.0 B, free 54.5 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:40733 (size: 1186.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 27) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 25) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 24) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 26) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 27). 2057 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 26). 2057 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 24). 2057 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 25). 2057 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 27) in 12 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 26) in 14 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 24) in 21 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 25) in 18 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 6 (count at TranscriptCounts.scala:43) finished in 0.021 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 3 finished: count at TranscriptCounts.scala:43, took 0.033134 s 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:51 19/03/03 21:21:50 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:50 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 165 bytes 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 19 (distinct at TranscriptCounts.scala:50) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 4 (count at TranscriptCounts.scala:51) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (count at TranscriptCounts.scala:51) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 9) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 9) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.0 KB, free 58.4 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.1 KB, free 60.5 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:40733 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 28, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 29, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 9.0 (TID 30, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 9.0 (TID 31, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 9.0 (TID 30) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 9.0 (TID 31) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 29) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 28) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_14_3 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_14_2 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_14_0 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_14_1 locally 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 9.0 (TID 31). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 9.0 (TID 31) in 10 ms on localhost (1/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 28). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 9.0 (TID 30). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 9.0 (TID 29). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 9.0 (TID 30) in 13 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 29) in 16 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 28) in 18 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 9 (distinct at TranscriptCounts.scala:50) finished in 0.018 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 10) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.1 KB, free 63.6 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1856.0 B, free 65.5 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:40733 (size: 1856.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 10.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 10.0 (TID 32, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 33, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 10.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 10.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 10.0 (TID 32) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 10.0 (TID 35) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 10.0 (TID 34) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 10.0 (TID 33) 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 10.0 (TID 33). 1123 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 10.0 (TID 35). 1123 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 10.0 (TID 34). 1123 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 10.0 (TID 32). 1123 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 33) in 13 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 10.0 (TID 32) in 14 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 10.0 (TID 35) in 12 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 10.0 (TID 34) in 14 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 10 (count at TranscriptCounts.scala:51) finished in 0.015 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 4 finished: count at TranscriptCounts.scala:51, took 0.059588 s 19/03/03 21:21:50 INFO rdd.ParallelCollectionRDD: Removing RDD 0 from persistence list 19/03/03 21:21:50 INFO storage.BlockManager: Removing RDD 0 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: collect at BroadcastRegionJoin.scala:100 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 28 (keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 5 (collect at BroadcastRegionJoin.scala:100) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 11) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 11) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.2 KB, free 68.0 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.2 KB, free 70.2 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:40733 (size: 2.2 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 36, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 37, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 38, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 11.0 (TID 39, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 37) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 38) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 11.0 (TID 39) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 36) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 36). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 38). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 36) in 27 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 38) in 17 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 37). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 11.0 (TID 39). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 37) in 29 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 11.0 (TID 39) in 23 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 11 (keyBy at BroadcastRegionJoin.scala:98) finished in 0.036 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 12) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 5.0 KB, free 75.2 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.5 KB, free 77.7 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:40733 (size: 2.5 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 40, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 41, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 42, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 43, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 40) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 43) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 42) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 41) 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 43). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 40). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 41). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 42). 1239 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 43) in 12 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 40) in 16 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 41) in 15 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) finished in 0.022 s 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 42) in 19 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 5 finished: collect at BroadcastRegionJoin.scala:100, took 0.082871 s 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 232.0 B, free 77.9 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 128.0 B, free 78.0 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:40733 (size: 128.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 12 from broadcast at BroadcastRegionJoin.scala:109 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 22 (flatMap at ReadCoverage.scala:59) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 32 (flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 30 (map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 39 (map at CoverageStats.scala:172) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 6 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (collect at CoverageStats.scala:174) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 2.7 KB, free 80.8 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 1677.0 B, free 82.4 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:40733 (size: 1677.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 3.6 KB, free 86.0 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.1 KB, free 88.1 KB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 44, localhost, partition 0,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:40733 (size: 2.1 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 13.0 (TID 45, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 13.0 (TID 46, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 13.0 (TID 47, localhost, partition 3,PROCESS_LOCAL, 2146 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 44) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 13.0 (TID 47) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 13.0 (TID 45) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 13.0 (TID 46) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 44). 1090 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 13.0 (TID 46). 1090 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 13.0 (TID 45). 1090 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 48) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 49) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 44) in 41 ms on localhost (1/4) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 50) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 48). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 13.0 (TID 46) in 35 ms on localhost (2/4) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 13.0 (TID 47). 1090 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 15.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 15.0 (TID 51) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 13.0 (TID 45) in 41 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 13.0 (TID 47) in 37 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 13 (flatMap at ReadCoverage.scala:59) finished in 0.049 s 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 50). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 15) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17, ShuffleMapStage 14) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 48) in 12 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118), which has no missing parents 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 50) in 11 ms on localhost (2/4) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 4.3 KB, free 92.4 KB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 49). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KB, free 94.8 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:40733 (size: 2.4 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 14.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 49) in 18 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 14.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 14.0 (TID 53, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 14.0 (TID 54, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 14.0 (TID 52) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 14.0 (TID 54) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 15.0 (TID 51). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 14.0 (TID 53) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 14.0 (TID 55, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 14.0 (TID 55) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_23_0 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_23_1 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_23_2 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_23_3 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 15.0 (TID 51) in 18 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 15 (map at BroadcastRegionJoin.scala:113) finished in 0.048 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 14) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_23_1 stored as values in memory (estimated size 16.0 B, free 94.8 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_23_1 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_23_3 stored as values in memory (estimated size 16.0 B, free 94.9 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_23_3 in memory on localhost:40733 (size: 16.0 B, free: 511.1 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 14.0 (TID 54). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 14.0 (TID 55). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 14.0 (TID 54) in 11 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 14.0 (TID 55) in 11 ms on localhost (2/4) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_23_2 stored as values in memory (estimated size 22.8 KB, free 117.6 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_23_0 stored as values in memory (estimated size 22.8 KB, free 140.4 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_23_2 in memory on localhost:40733 (size: 22.8 KB, free: 511.1 MB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_23_0 in memory on localhost:40733 (size: 22.8 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 14.0 (TID 52). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 14.0 (TID 53). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 14.0 (TID 52) in 76 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 14.0 (TID 53) in 76 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 14 (flatMap at BroadcastRegionJoin.scala:118) finished in 0.077 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.1 KB, free 144.5 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.2 KB, free 146.7 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:40733 (size: 2.2 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 57) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 59) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 58) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 56) 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_37_0 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_37_1 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_37_2 not found, computing it 19/03/03 21:21:50 INFO spark.CacheManager: Partition rdd_37_3 not found, computing it 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_37_0 stored as values in memory (estimated size 16.0 B, free 146.7 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_37_2 stored as values in memory (estimated size 16.0 B, free 146.7 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_37_0 in memory on localhost:40733 (size: 16.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_37_3 stored as values in memory (estimated size 16.0 B, free 146.7 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_37_2 in memory on localhost:40733 (size: 16.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_37_3 in memory on localhost:40733 (size: 16.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 56). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 58). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 59). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 56) in 22 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 59) in 20 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 58) in 21 ms on localhost (3/4) 19/03/03 21:21:50 INFO storage.MemoryStore: Block rdd_37_1 stored as values in memory (estimated size 45.8 KB, free 192.6 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added rdd_37_1 in memory on localhost:40733 (size: 45.8 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 57). 1886 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 57) in 67 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (map at CoverageStats.scala:172) finished in 0.068 s 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 2.5 KB, free 195.1 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 1554.0 B, free 196.6 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:40733 (size: 1554.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 17.0 (TID 60, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 17.0 (TID 61, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 62, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 17.0 (TID 63, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 17.0 (TID 60) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 62) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 17.0 (TID 63) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 17.0 (TID 61) 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 62). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 17.0 (TID 63). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 17.0 (TID 61). 1131 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 17.0 (TID 60). 1131 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 62) in 8 ms on localhost (1/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 17.0 (TID 63) in 8 ms on localhost (2/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 17.0 (TID 60) in 9 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 17.0 (TID 61) in 10 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 17 (collect at CoverageStats.scala:174) finished in 0.010 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 6 finished: collect at CoverageStats.scala:174, took 0.240735 s 19/03/03 21:21:50 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:50 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 8 is 159 bytes 19/03/03 21:21:50 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 161 bytes 19/03/03 21:21:50 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 161 bytes 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Registering RDD 42 (map at CoverageStats.scala:172) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Got job 7 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 22 (collect at CoverageStats.scala:174) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 21) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 21) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 4.2 KB, free 200.9 KB) 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.2 KB, free 203.1 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:40733 (size: 2.2 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 4 tasks 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 64, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 21.0 (TID 65, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 21.0 (TID 66, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 21.0 (TID 67, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 64) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 21.0 (TID 65) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 21.0 (TID 66) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 21.0 (TID 67) 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_37_1 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_37_3 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_37_0 locally 19/03/03 21:21:50 INFO storage.BlockManager: Found block rdd_37_2 locally 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_17_piece0 on localhost:40733 in memory (size: 1554.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 21.0 (TID 66). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 21.0 (TID 67). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 64). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 34 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_16_piece0 on localhost:40733 in memory (size: 2.2 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 33 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 21.0 (TID 67) in 33 ms on localhost (1/4) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_15_piece0 on localhost:40733 in memory (size: 2.4 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 64) in 35 ms on localhost (2/4) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 32 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 21.0 (TID 66) in 34 ms on localhost (3/4) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:40733 in memory (size: 2.1 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 31 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:40733 in memory (size: 1677.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 30 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned shuffle 7 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:40733 in memory (size: 2.5 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 29 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:40733 in memory (size: 2.2 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 21.0 (TID 65). 2240 bytes result sent to driver 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 28 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:40733 in memory (size: 1856.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 27 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:40733 in memory (size: 2.1 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 21.0 (TID 65) in 41 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ShuffleMapStage 21 (map at CoverageStats.scala:172) finished in 0.042 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 26 19/03/03 21:21:50 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:50 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 22) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:40733 in memory (size: 1186.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 25 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 2.5 KB, free 150.7 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:40733 in memory (size: 2047.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 24 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:40733 in memory (size: 2.1 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 23 19/03/03 21:21:50 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 1554.0 B, free 140.6 KB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:40733 (size: 1554.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:40733 in memory (size: 2039.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 22 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Adding task set 22.0 with 4 tasks 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:40733 in memory (size: 2.5 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 22.0 (TID 68, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 21 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 22.0 (TID 69, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 22.0 (TID 70, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:40733 in memory (size: 2.1 KB, free: 511.0 MB) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 22.0 (TID 71, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:50 INFO executor.Executor: Running task 2.0 in stage 22.0 (TID 70) 19/03/03 21:21:50 INFO executor.Executor: Running task 0.0 in stage 22.0 (TID 69) 19/03/03 21:21:50 INFO executor.Executor: Running task 1.0 in stage 22.0 (TID 68) 19/03/03 21:21:50 INFO executor.Executor: Running task 3.0 in stage 22.0 (TID 71) 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 20 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:50 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:40733 in memory (size: 1324.0 B, free: 511.0 MB) 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:50 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:50 INFO spark.ContextCleaner: Cleaned accumulator 19 19/03/03 21:21:50 INFO executor.Executor: Finished task 2.0 in stage 22.0 (TID 70). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO executor.Executor: Finished task 0.0 in stage 22.0 (TID 69). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 22.0 (TID 70) in 9 ms on localhost (1/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 3.0 in stage 22.0 (TID 71). 1124 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 22.0 (TID 69) in 9 ms on localhost (2/4) 19/03/03 21:21:50 INFO executor.Executor: Finished task 1.0 in stage 22.0 (TID 68). 1131 bytes result sent to driver 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 22.0 (TID 71) in 11 ms on localhost (3/4) 19/03/03 21:21:50 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 22.0 (TID 68) in 13 ms on localhost (4/4) 19/03/03 21:21:50 INFO scheduler.DAGScheduler: ResultStage 22 (collect at CoverageStats.scala:174) finished in 0.013 s 19/03/03 21:21:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 19/03/03 21:21:50 INFO scheduler.DAGScheduler: Job 7 finished: collect at CoverageStats.scala:174, took 0.072736 s 19/03/03 21:21:50 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:50 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:50 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:50 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:50 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:50 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:50 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:50 INFO spark.SparkContext: Successfully stopped SparkContext - two transcripts, one fully covered, one not covered at all, one random read 19/03/03 21:21:50 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:50 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:50 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:50 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:50 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:50 INFO util.Utils: Successfully started service 'sparkDriver' on port 58655. 19/03/03 21:21:50 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:50 INFO Remoting: Starting remoting 19/03/03 21:21:50 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58656] 19/03/03 21:21:50 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58656. 19/03/03 21:21:50 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:50 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:50 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-a6173517-18da-4d05-9105-3fc2c0c36811 19/03/03 21:21:50 INFO storage.MemoryStore: MemoryStore started with capacity 492.0 MB 19/03/03 21:21:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:51 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46387. 19/03/03 21:21:51 INFO netty.NettyBlockTransferService: Server created on 46387 19/03/03 21:21:51 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:51 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:46387 with 492.0 MB RAM, BlockManagerId(driver, localhost, 46387) 19/03/03 21:21:51 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: reduce at RNASeqReadCounts.scala:80 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 0 (reduce at RNASeqReadCounts.scala:80) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 2.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1324.0 B, free 3.3 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:46387 (size: 1324.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at map at RNASeqReadCounts.scala:79) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2157 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_0_3 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_0_3 stored as values in memory (estimated size 632.0 B, free 4.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_0_3 in memory on localhost:46387 (size: 632.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_0_2 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_0_2 stored as values in memory (estimated size 352.0 B, free 4.3 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_0_2 in memory on localhost:46387 (size: 352.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_0_0 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_0_1 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_0_0 stored as values in memory (estimated size 352.0 B, free 4.6 KB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1548 bytes result sent to driver 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_0_1 stored as values in memory (estimated size 352.0 B, free 5.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_0_0 in memory on localhost:46387 (size: 352.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_0_1 in memory on localhost:46387 (size: 352.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1548 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1548 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1548 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 15 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 16 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 24 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 20 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) finished in 0.026 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 0 finished: reduce at RNASeqReadCounts.scala:80, took 0.033715 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: collect at RightOuterJoin.scala:77 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 7 (keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 1 (collect at RightOuterJoin.scala:77) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (collect at RightOuterJoin.scala:77) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.0 KB, free 9.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.1 KB, free 11.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:46387 (size: 2.1 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at RightOuterJoin.scala:75) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_1_2 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_1_2 stored as values in memory (estimated size 16.0 B, free 11.1 KB) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_1_0 not found, computing it 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_1_2 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_1_0 stored as values in memory (estimated size 16.0 B, free 11.1 KB) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_1_3 not found, computing it 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_1_0 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_1_1 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_1_3 stored as values in memory (estimated size 4.7 KB, free 15.9 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_1_3 in memory on localhost:46387 (size: 4.7 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_1_1 stored as values in memory (estimated size 4.7 KB, free 20.6 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_1_1 in memory on localhost:46387 (size: 4.7 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1670 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1670 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 11 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 18 ms on localhost (2/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1670 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 20 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1670 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 22 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at RightOuterJoin.scala:75) finished in 0.030 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.8 KB, free 25.3 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 27.8 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:46387 (size: 2.5 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[8] at groupByKey at RightOuterJoin.scala:76) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 13 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 12 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 13 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1239 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 18 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 2 (collect at RightOuterJoin.scala:77) finished in 0.019 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 1 finished: collect at RightOuterJoin.scala:77, took 0.067610 s 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 232.0 B, free 28.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 128.0 B, free 28.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:46387 (size: 128.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 3 from broadcast at RightOuterJoin.scala:86 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: reduce at RNASeqAlignmentCounts.scala:72 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 9 (map at RightOuterJoin.scala:90) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 10 (flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 2 (reduce at RNASeqAlignmentCounts.scala:72) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 3, ShuffleMapStage 4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.4 KB, free 31.5 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2039.0 B, free 33.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:46387 (size: 2039.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[9] at map at RightOuterJoin.scala:90) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94), which has no missing parents 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.7 KB, free 37.2 KB) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 39.3 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:46387 (size: 2.1 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[10] at flatMap at RightOuterJoin.scala:94) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_0_0 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 15 ms on localhost (1/4) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_0_1 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 22 ms on localhost (2/4) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2146 bytes) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_0_2 locally 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 18 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 22 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (map at RightOuterJoin.scala:90) finished in 0.025 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_0_3 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 14 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 14 ms on localhost (2/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 13 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 14 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (flatMap at RightOuterJoin.scala:94) finished in 0.028 s 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.8 KB, free 43.1 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2047.0 B, free 45.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:46387 (size: 2047.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[16] at flatMap at RNASeqAlignmentCounts.scala:71) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 20) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 22) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 21) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 23) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_14_3 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_14_0 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_14_2 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_14_1 not found, computing it 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_14_2 stored as values in memory (estimated size 16.0 B, free 45.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_14_2 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_14_0 stored as values in memory (estimated size 392.0 B, free 45.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_14_0 in memory on localhost:46387 (size: 392.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_14_3 stored as values in memory (estimated size 16.0 B, free 45.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_14_3 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 22). 1704 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 20). 1767 bytes result sent to driver 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_14_1 stored as values in memory (estimated size 6.0 KB, free 51.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_14_1 in memory on localhost:46387 (size: 6.0 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 23). 1704 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 22) in 13 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 23) in 14 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 20) in 15 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 21). 1767 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 21) in 18 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 5 (reduce at RNASeqAlignmentCounts.scala:72) finished in 0.019 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 2 finished: reduce at RNASeqAlignmentCounts.scala:72, took 0.070016 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:43 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 3 (count at TranscriptCounts.scala:43) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at TranscriptCounts.scala:43) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 1808.0 B, free 53.3 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1186.0 B, free 54.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:46387 (size: 1186.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[17] at filter at TranscriptCounts.scala:42) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 24) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 25) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 26) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 27) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 24). 2057 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 27). 2057 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 26). 2057 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 25). 2057 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 24) in 18 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 27) in 9 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 26) in 13 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 25) in 17 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 6 (count at TranscriptCounts.scala:43) finished in 0.020 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 3 finished: count at TranscriptCounts.scala:43, took 0.028925 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: count at TranscriptCounts.scala:51 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 165 bytes 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 19 (distinct at TranscriptCounts.scala:50) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 4 (count at TranscriptCounts.scala:51) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (count at TranscriptCounts.scala:51) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 9) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 9) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.0 KB, free 58.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.1 KB, free 60.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:46387 (size: 2.1 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[19] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 28, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 29, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 9.0 (TID 30, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 9.0 (TID 31, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 29) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 28) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 9.0 (TID 30) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 9.0 (TID 31) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_14_3 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_14_0 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_14_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_14_2 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 9.0 (TID 31). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 9.0 (TID 30). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 28). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 9.0 (TID 29). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 9.0 (TID 31) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 9.0 (TID 30) in 8 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 28) in 9 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 29) in 9 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 9 (distinct at TranscriptCounts.scala:50) finished in 0.010 s 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 10) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.1 KB, free 63.6 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1856.0 B, free 65.5 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:46387 (size: 1856.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 10 (MapPartitionsRDD[21] at distinct at TranscriptCounts.scala:50) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 10.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 10.0 (TID 32, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 33, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 10.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 10.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 10.0 (TID 32) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 10.0 (TID 33) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 10.0 (TID 35) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 10.0 (TID 34) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 10.0 (TID 32). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 10.0 (TID 34). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 10.0 (TID 32) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 10.0 (TID 35). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 10.0 (TID 33). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 10.0 (TID 34) in 7 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 10.0 (TID 35) in 8 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 33) in 9 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 10 (count at TranscriptCounts.scala:51) finished in 0.009 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 4 finished: count at TranscriptCounts.scala:51, took 0.034049 s 19/03/03 21:21:51 INFO rdd.ParallelCollectionRDD: Removing RDD 0 from persistence list 19/03/03 21:21:51 INFO storage.BlockManager: Removing RDD 0 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: collect at BroadcastRegionJoin.scala:100 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 28 (keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 5 (collect at BroadcastRegionJoin.scala:100) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 11) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 11) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.2 KB, free 68.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.2 KB, free 70.2 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:46387 (size: 2.2 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 11 (MapPartitionsRDD[28] at keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 36, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 37, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 38, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 11.0 (TID 39, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 37) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 11.0 (TID 39) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 38) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 36) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 36). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 38). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 38) in 9 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 36) in 13 ms on localhost (2/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 11.0 (TID 39). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 37). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 11.0 (TID 39) in 13 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 37) in 16 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 11 (keyBy at BroadcastRegionJoin.scala:98) finished in 0.018 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 12) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 5.0 KB, free 75.2 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.5 KB, free 77.7 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:46387 (size: 2.5 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 12 (ShuffledRDD[29] at groupByKey at BroadcastRegionJoin.scala:99) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 40, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 41, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 42, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 43, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 43) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 42) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 41) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 40) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 41). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 40). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 43). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 42). 1239 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 41) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 40) in 8 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 43) in 7 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 42) in 10 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 12 (collect at BroadcastRegionJoin.scala:100) finished in 0.011 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 5 finished: collect at BroadcastRegionJoin.scala:100, took 0.040919 s 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 232.0 B, free 77.9 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 128.0 B, free 78.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:46387 (size: 128.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 12 from broadcast at BroadcastRegionJoin.scala:109 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 22 (flatMap at ReadCoverage.scala:59) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 32 (flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 30 (map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 39 (map at CoverageStats.scala:172) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 6 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (collect at CoverageStats.scala:174) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 2.7 KB, free 80.8 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 1677.0 B, free 82.4 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:46387 (size: 1677.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[22] at flatMap at ReadCoverage.scala:59) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 3.6 KB, free 86.0 KB) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 44, localhost, partition 0,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 13.0 (TID 45, localhost, partition 1,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.1 KB, free 88.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:46387 (size: 2.1 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[30] at map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 13.0 (TID 46, localhost, partition 2,PROCESS_LOCAL, 2079 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 13.0 (TID 47, localhost, partition 3,PROCESS_LOCAL, 2146 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 13.0 (TID 46) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 44) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 13.0 (TID 45) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 13.0 (TID 47) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 13.0 (TID 46). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 13.0 (TID 45). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 44). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 48) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 13.0 (TID 47). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 49) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_0 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 13.0 (TID 46) in 21 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 48). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_1 locally 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 50) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 13.0 (TID 45) in 24 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 44) in 26 ms on localhost (3/4) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_2 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 15.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2071 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 15.0 (TID 51) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 13.0 (TID 47) in 24 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 13 (flatMap at ReadCoverage.scala:59) finished in 0.029 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 15) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17, ShuffleMapStage 14) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 50). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 48) in 9 ms on localhost (1/4) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_1_3 locally 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118), which has no missing parents 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 50) in 7 ms on localhost (2/4) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 4.3 KB, free 92.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KB, free 94.8 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:46387 (size: 2.4 KB, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 49). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 14 (MapPartitionsRDD[32] at flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 14.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 14.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 14.0 (TID 53, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 15.0 (TID 51). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 14.0 (TID 54, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 14.0 (TID 52) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 49) in 15 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 14.0 (TID 53) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 14.0 (TID 54) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 14.0 (TID 55, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 14.0 (TID 55) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_23_0 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_23_2 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_23_1 not found, computing it 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_23_3 not found, computing it 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 15.0 (TID 51) in 12 ms on localhost (4/4) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 15 (map at BroadcastRegionJoin.scala:113) finished in 0.032 s 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 14) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_23_1 stored as values in memory (estimated size 16.0 B, free 94.8 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_23_1 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_23_3 stored as values in memory (estimated size 16.0 B, free 94.9 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_23_3 in memory on localhost:46387 (size: 16.0 B, free: 492.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 14.0 (TID 54). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 14.0 (TID 55). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 14.0 (TID 54) in 8 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 14.0 (TID 55) in 8 ms on localhost (2/4) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_23_0 stored as values in memory (estimated size 22.8 KB, free 117.6 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_23_0 in memory on localhost:46387 (size: 22.8 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_23_2 stored as values in memory (estimated size 22.8 KB, free 140.4 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_23_2 in memory on localhost:46387 (size: 22.8 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 14.0 (TID 52). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 14.0 (TID 53). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 14.0 (TID 52) in 50 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 14.0 (TID 53) in 50 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 14 (flatMap at BroadcastRegionJoin.scala:118) finished in 0.051 s 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 16, ResultStage 17) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.1 KB, free 144.5 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.2 KB, free 146.7 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:46387 (size: 2.2 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:46387 in memory (size: 2.1 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[39] at map at CoverageStats.scala:172) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:46387 in memory (size: 1677.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 56) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 58) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 59) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:46387 in memory (size: 2.5 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 57) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 47 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_37_2 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_37_0 not found, computing it 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:46387 in memory (size: 2.2 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_37_1 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_37_3 not found, computing it 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 46 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:46387 in memory (size: 1856.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 45 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:46387 in memory (size: 2.1 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 44 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:46387 in memory (size: 1186.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 43 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:46387 in memory (size: 2047.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 42 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_37_0 stored as values in memory (estimated size 16.0 B, free 103.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_37_0 in memory on localhost:46387 (size: 16.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_37_2 stored as values in memory (estimated size 16.0 B, free 101.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_37_3 stored as values in memory (estimated size 16.0 B, free 101.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:46387 in memory (size: 2.1 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_37_2 in memory on localhost:46387 (size: 16.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 41 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_37_3 in memory on localhost:46387 (size: 16.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:46387 in memory (size: 2039.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 40 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:46387 in memory (size: 2.5 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 39 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 58). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 56). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:46387 in memory (size: 2.1 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 38 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 59). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:46387 in memory (size: 1324.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.ContextCleaner: Cleaned accumulator 37 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 58) in 13 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 59) in 14 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 56) in 15 ms on localhost (3/4) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_37_1 stored as values in memory (estimated size 45.8 KB, free 121.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_37_1 in memory on localhost:46387 (size: 45.8 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 57). 1886 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 57) in 61 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (map at CoverageStats.scala:172) finished in 0.061 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 2.5 KB, free 123.6 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 1554.0 B, free 125.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:46387 (size: 1554.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 17 (ShuffledRDD[40] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 17.0 (TID 60, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 17.0 (TID 61, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 62, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 17.0 (TID 63, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 17.0 (TID 63) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 17.0 (TID 60) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 62) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 17.0 (TID 61) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 17.0 (TID 63). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 17.0 (TID 60). 1131 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 17.0 (TID 61). 1131 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 17.0 (TID 63) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 62). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 17.0 (TID 60) in 8 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 17.0 (TID 61) in 8 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 62) in 9 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 17 (collect at CoverageStats.scala:174) finished in 0.010 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 6 finished: collect at CoverageStats.scala:174, took 0.201682 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: collect at CoverageStats.scala:174 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 8 is 159 bytes 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 161 bytes 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 161 bytes 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 42 (map at CoverageStats.scala:172) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 7 (collect at CoverageStats.scala:174) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 22 (collect at CoverageStats.scala:174) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 21) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 21) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 4.2 KB, free 129.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.2 KB, free 131.6 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:46387 (size: 2.2 KB, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[42] at map at CoverageStats.scala:172) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 64, localhost, partition 0,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 21.0 (TID 65, localhost, partition 1,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 21.0 (TID 66, localhost, partition 2,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 21.0 (TID 67, localhost, partition 3,PROCESS_LOCAL, 1956 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 64) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 21.0 (TID 67) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 21.0 (TID 65) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 21.0 (TID 66) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_37_3 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_37_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_37_2 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_37_0 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 21.0 (TID 67). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 21.0 (TID 66). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 64). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 21.0 (TID 67) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 21.0 (TID 66) in 7 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 64) in 7 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 21.0 (TID 65). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 21.0 (TID 65) in 13 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 21 (map at CoverageStats.scala:172) finished in 0.014 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 22) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 2.5 KB, free 134.1 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 1554.0 B, free 135.6 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:46387 (size: 1554.0 B, free: 491.9 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 22 (ShuffledRDD[43] at reduceByKey at CoverageStats.scala:173) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 22.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 22.0 (TID 68, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 22.0 (TID 69, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 22.0 (TID 70, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 22.0 (TID 71, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 22.0 (TID 71) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 22.0 (TID 69) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 22.0 (TID 68) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 22.0 (TID 70) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 22.0 (TID 68). 1131 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 22.0 (TID 70). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 22.0 (TID 71). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 22.0 (TID 69). 1124 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 22.0 (TID 68) in 7 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 22.0 (TID 70) in 7 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 22.0 (TID 69) in 8 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 22.0 (TID 71) in 8 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 22 (collect at CoverageStats.scala:174) finished in 0.008 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 7 finished: collect at CoverageStats.scala:174, took 0.036024 s 19/03/03 21:21:51 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:51 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:51 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:51 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:51 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:51 INFO spark.SparkContext: Successfully stopped SparkContext - two SO transcripts, one fully covered, one not covered at all, one random read 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:51 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:51 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:51 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:51 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'sparkDriver' on port 58842. 19/03/03 21:21:51 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:51 INFO Remoting: Starting remoting 19/03/03 21:21:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58843] 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58843. 19/03/03 21:21:51 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:51 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:51 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-aa18ddc0-278c-437d-8821-138a77999ea5 19/03/03 21:21:51 INFO storage.MemoryStore: MemoryStore started with capacity 501.0 MB 19/03/03 21:21:51 INFO spark.SparkEnv: Registering OutputCommitCoordinator PanelBasesSuite: 19/03/03 21:21:51 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34780. 19/03/03 21:21:51 INFO netty.NettyBlockTransferService: Server created on 34780 19/03/03 21:21:51 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:51 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:34780 with 501.0 MB RAM, BlockManagerId(driver, localhost, 34780) 19/03/03 21:21:51 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: count at PanelBases.scala:67 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 0 (count at PanelBases.scala:67) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at PanelBases.scala:67) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1840.0 B, free 1840.0 B) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1207.0 B, free 3.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:34780 (size: 1207.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2050 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_2_2 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_2_0 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_2_3 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_2_1 not found, computing it 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_2_0 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_2_2 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_2_0 in memory on localhost:34780 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_2_1 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_2_2 in memory on localhost:34780 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_2_1 in memory on localhost:34780 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1487 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1487 bytes result sent to driver 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_2_3 stored as values in memory (estimated size 5.5 KB, free 8.5 KB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1487 bytes result sent to driver 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_2_3 in memory on localhost:34780 (size: 5.5 KB, free: 501.0 MB) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 30 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1488 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 28 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 29 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 24 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 0 (count at PanelBases.scala:67) finished in 0.033 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 0 finished: count at PanelBases.scala:67, took 0.041945 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: count at PanelBases.scala:68 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 4 (distinct at PanelBases.scala:59) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 1 (count at PanelBases.scala:68) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (count at PanelBases.scala:68) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 11.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1761.0 B, free 13.1 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:34780 (size: 1761.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 11 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 16 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 15 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 15 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (distinct at PanelBases.scala:59) finished in 0.022 s 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 16.2 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1840.0 B, free 18.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:34780 (size: 1840.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_6_0 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_6_2 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_6_3 not found, computing it 19/03/03 21:21:51 INFO spark.CacheManager: Partition rdd_6_1 not found, computing it 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_6_1 stored as values in memory (estimated size 2.5 KB, free 20.5 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_6_2 stored as values in memory (estimated size 3.2 KB, free 23.7 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_6_1 in memory on localhost:34780 (size: 2.5 KB, free: 501.0 MB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_6_3 stored as values in memory (estimated size 2.7 KB, free 26.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block rdd_6_0 stored as values in memory (estimated size 3.4 KB, free 29.8 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_6_2 in memory on localhost:34780 (size: 3.2 KB, free: 501.0 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_6_3 in memory on localhost:34780 (size: 2.7 KB, free: 501.0 MB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added rdd_6_0 in memory on localhost:34780 (size: 3.4 KB, free: 501.0 MB) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1703 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1703 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1703 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1703 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 10 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 10 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 12 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 11 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 2 (count at PanelBases.scala:68) finished in 0.012 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 1 finished: count at PanelBases.scala:68, took 0.050256 s 19/03/03 21:21:51 INFO spark.SparkContext: Starting job: count at PanelBases.scala:69 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 2 (flatMap at PanelBases.scala:46) 19/03/03 21:21:51 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Registering RDD 6 (distinct at PanelBases.scala:59) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Got job 2 (count at PanelBases.scala:69) with 4 output partitions 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at PanelBases.scala:69) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.6 KB, free 32.3 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1633.0 B, free 33.9 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:34780 (size: 1633.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.2 KB, free 37.1 KB) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1931.0 B, free 39.0 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:34780 (size: 1931.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_2_0 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_2_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_2_2 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_2_3 locally 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 16) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 17) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_6_0 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 12 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 18) 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_6_1 locally 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_6_2 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 11 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 10 ms on localhost (3/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 19) 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO storage.BlockManager: Found block rdd_6_3 locally 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 17) in 11 ms on localhost (1/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 16) in 12 ms on localhost (2/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 18) in 11 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 18 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (flatMap at PanelBases.scala:46) finished in 0.022 s 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 5) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 19) in 9 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (distinct at PanelBases.scala:59) finished in 0.023 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:51 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64), which has no missing parents 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.4 KB, free 42.4 KB) 19/03/03 21:21:51 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1980.0 B, free 44.4 KB) 19/03/03 21:21:51 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:34780 (size: 1980.0 B, free: 501.0 MB) 19/03/03 21:21:51 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64) 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:51 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 20) 19/03/03 21:21:51 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 21) 19/03/03 21:21:51 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 23) 19/03/03 21:21:51 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 22) 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:51 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:51 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 23). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 23) in 9 ms on localhost (1/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 21). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 21) in 10 ms on localhost (2/4) 19/03/03 21:21:51 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 22). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 20). 1123 bytes result sent to driver 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 22) in 14 ms on localhost (3/4) 19/03/03 21:21:51 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 20) in 14 ms on localhost (4/4) 19/03/03 21:21:51 INFO scheduler.DAGScheduler: ResultStage 6 (count at PanelBases.scala:69) finished in 0.016 s 19/03/03 21:21:51 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:51 INFO scheduler.DAGScheduler: Job 2 finished: count at PanelBases.scala:69, took 0.059532 s 19/03/03 21:21:51 INFO rdd.MapPartitionsRDD: Removing RDD 2 from persistence list 19/03/03 21:21:51 INFO storage.BlockManager: Removing RDD 2 19/03/03 21:21:51 INFO rdd.MapPartitionsRDD: Removing RDD 6 from persistence list 19/03/03 21:21:51 INFO storage.BlockManager: Removing RDD 6 19/03/03 21:21:51 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:51 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:51 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:51 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:51 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:51 INFO spark.SparkContext: Successfully stopped SparkContext - compute coverage for a panel with one bait and one target 19/03/03 21:21:51 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:51 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:51 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:51 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:51 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'sparkDriver' on port 51003. 19/03/03 21:21:51 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:51 INFO Remoting: Starting remoting 19/03/03 21:21:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:51004] 19/03/03 21:21:51 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 51004. 19/03/03 21:21:51 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:52 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-5b1ce11e-6ece-425c-b15d-ed600e17efeb 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore started with capacity 501.0 MB 19/03/03 21:21:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:52 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41889. 19/03/03 21:21:52 INFO netty.NettyBlockTransferService: Server created on 41889 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:52 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41889 with 501.0 MB RAM, BlockManagerId(driver, localhost, 41889) 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:67 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 0 (count at PanelBases.scala:67) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at PanelBases.scala:67) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1840.0 B, free 1840.0 B) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1207.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:41889 (size: 1207.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2050 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_1 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_0 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_2 not found, computing it 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_1 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_3 not found, computing it 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_1 in memory on localhost:41889 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_0 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_2 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_0 in memory on localhost:41889 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_2 in memory on localhost:41889 (size: 16.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_3 stored as values in memory (estimated size 5.5 KB, free 8.5 KB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_3 in memory on localhost:41889 (size: 5.5 KB, free: 501.0 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 11 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1488 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 14 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 11 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 11 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 0 (count at PanelBases.scala:67) finished in 0.015 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 0 finished: count at PanelBases.scala:67, took 0.024278 s 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:68 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 4 (distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 1 (count at PanelBases.scala:68) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (count at PanelBases.scala:68) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 11.4 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1761.0 B, free 13.1 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41889 (size: 1761.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 15 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 13 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 12 ms on localhost (3/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 18 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (distinct at PanelBases.scala:59) finished in 0.020 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 16.2 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1840.0 B, free 18.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41889 (size: 1840.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_2 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_3 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_0 not found, computing it 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_1 not found, computing it 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_2 stored as values in memory (estimated size 3.2 KB, free 21.2 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_2 in memory on localhost:41889 (size: 3.2 KB, free: 501.0 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_1 stored as values in memory (estimated size 2.5 KB, free 23.7 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_1 in memory on localhost:41889 (size: 2.5 KB, free: 501.0 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_3 stored as values in memory (estimated size 2.7 KB, free 26.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_3 in memory on localhost:41889 (size: 2.7 KB, free: 501.0 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_0 stored as values in memory (estimated size 3.4 KB, free 29.8 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_0 in memory on localhost:41889 (size: 3.4 KB, free: 501.0 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 10 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 12 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 13 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 13 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 2 (count at PanelBases.scala:68) finished in 0.015 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 1 finished: count at PanelBases.scala:68, took 0.048958 s 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:69 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 2 (flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 165 bytes 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 6 (distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 2 (count at PanelBases.scala:69) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at PanelBases.scala:69) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.6 KB, free 32.3 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1633.0 B, free 33.9 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:41889 (size: 1633.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.2 KB, free 37.1 KB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1931.0 B, free 39.0 KB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:41889 (size: 1931.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_0 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_3 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_2 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_1 locally 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 16) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_0 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 12 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 17) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_1 locally 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 18) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 19) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 13 ms on localhost (2/4) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_2 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 15 ms on localhost (3/4) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_3 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 12 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (flatMap at PanelBases.scala:46) finished in 0.016 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 5) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 16) in 11 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 18) in 12 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 17) in 13 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 19) in 12 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (distinct at PanelBases.scala:59) finished in 0.021 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.4 KB, free 42.4 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1980.0 B, free 44.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:41889 (size: 1980.0 B, free: 501.0 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 20) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 22) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 23) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 21) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 23). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 21). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 23) in 13 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 22). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 21) in 14 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 20). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 22) in 15 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 20) in 16 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 6 (count at PanelBases.scala:69) finished in 0.016 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 2 finished: count at PanelBases.scala:69, took 0.058580 s 19/03/03 21:21:52 INFO rdd.MapPartitionsRDD: Removing RDD 2 from persistence list 19/03/03 21:21:52 INFO storage.BlockManager: Removing RDD 2 19/03/03 21:21:52 INFO rdd.MapPartitionsRDD: Removing RDD 6 from persistence list 19/03/03 21:21:52 INFO storage.BlockManager: Removing RDD 6 19/03/03 21:21:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:52 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:52 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:52 INFO spark.SparkContext: Successfully stopped SparkContext - compute coverage for a panel with two baits and one target 19/03/03 21:21:52 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:52 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 58288. 19/03/03 21:21:52 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:52 INFO Remoting: Starting remoting 19/03/03 21:21:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58289] 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58289. 19/03/03 21:21:52 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:52 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-bd426d99-ad93-4c51-bc84-58fc055df922 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore started with capacity 500.3 MB 19/03/03 21:21:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:52 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34249. 19/03/03 21:21:52 INFO netty.NettyBlockTransferService: Server created on 34249 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:52 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:34249 with 500.3 MB RAM, BlockManagerId(driver, localhost, 34249) 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:67 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 0 (count at PanelBases.scala:67) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at PanelBases.scala:67) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1840.0 B, free 1840.0 B) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1207.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:34249 (size: 1207.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2015 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2050 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_1 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_0 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_2 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_2_3 not found, computing it 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_1 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_1 in memory on localhost:34249 (size: 16.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_0 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_2 stored as values in memory (estimated size 16.0 B, free 3.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_0 in memory on localhost:34249 (size: 16.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_2 in memory on localhost:34249 (size: 16.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_2_3 stored as values in memory (estimated size 5.5 KB, free 8.5 KB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_2_3 in memory on localhost:34249 (size: 5.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1487 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1488 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 12 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 14 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 11 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 11 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 0 (count at PanelBases.scala:67) finished in 0.015 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 0 finished: count at PanelBases.scala:67, took 0.021479 s 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:68 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 4 (distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 1 (count at PanelBases.scala:68) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (count at PanelBases.scala:68) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 11.4 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1761.0 B, free 13.1 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:34249 (size: 1761.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[4] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 9 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 11 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 13 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 12 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (distinct at PanelBases.scala:59) finished in 0.016 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 16.2 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1840.0 B, free 18.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:34249 (size: 1840.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_1 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_0 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_3 not found, computing it 19/03/03 21:21:52 INFO spark.CacheManager: Partition rdd_6_2 not found, computing it 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_1 stored as values in memory (estimated size 2.5 KB, free 20.5 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_1 in memory on localhost:34249 (size: 2.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_0 stored as values in memory (estimated size 3.4 KB, free 23.9 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_0 in memory on localhost:34249 (size: 3.4 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_2 stored as values in memory (estimated size 3.2 KB, free 27.1 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_2 in memory on localhost:34249 (size: 3.2 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block rdd_6_3 stored as values in memory (estimated size 2.7 KB, free 29.8 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added rdd_6_3 in memory on localhost:34249 (size: 2.7 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 9 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1703 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 10 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 10 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 11 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 2 (count at PanelBases.scala:68) finished in 0.013 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 1 finished: count at PanelBases.scala:68, took 0.039509 s 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: count at PanelBases.scala:69 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 2 (flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 165 bytes 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 6 (distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 2 (count at PanelBases.scala:69) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (count at PanelBases.scala:69) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 5, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.6 KB, free 32.3 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1633.0 B, free 33.9 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:34249 (size: 1633.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[2] at flatMap at PanelBases.scala:46) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.2 KB, free 37.1 KB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1931.0 B, free 39.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:34249 (size: 1931.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[6] at distinct at PanelBases.scala:59) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2004 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2039 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_3 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_2 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_1 locally 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_2_0 locally 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 16) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 17) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_0 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_1 locally 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 18) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 11 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 10 ms on localhost (2/4) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_2 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 13 ms on localhost (3/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 19) 19/03/03 21:21:52 INFO storage.BlockManager: Found block rdd_6_3 locally 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 13 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (flatMap at PanelBases.scala:46) finished in 0.018 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 5) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 17). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 16). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 17) in 11 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 16) in 13 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 18). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 19). 2240 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 18) in 13 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 19) in 10 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (distinct at PanelBases.scala:59) finished in 0.020 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.4 KB, free 42.4 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1980.0 B, free 44.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:34249 (size: 1980.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 6 (MapPartitionsRDD[10] at filter at PanelBases.scala:64) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 20) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 23) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 22) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 21) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 23). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 23) in 8 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 21). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 21) in 11 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 20). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 20) in 15 ms on localhost (3/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 22). 1123 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 22) in 16 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 6 (count at PanelBases.scala:69) finished in 0.017 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 2 finished: count at PanelBases.scala:69, took 0.055288 s 19/03/03 21:21:52 INFO rdd.MapPartitionsRDD: Removing RDD 2 from persistence list 19/03/03 21:21:52 INFO storage.BlockManager: Removing RDD 2 19/03/03 21:21:52 INFO rdd.MapPartitionsRDD: Removing RDD 6 from persistence list 19/03/03 21:21:52 INFO storage.BlockManager: Removing RDD 6 19/03/03 21:21:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:52 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:52 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:52 INFO spark.SparkContext: Successfully stopped SparkContext - compute coverage for a panel with two overlapping baits and one target 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:52 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:52 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 56026. RNASeqReadCountsSuite: - check stats for an unaligned read - check stats for an unique read, mapped on the forward strand - check stats for an unique read, mapped on the reverse strand - check stats for a duplicated read, mapped on the forward strand - check stats for a duplicated read, mapped on the reverse strand 19/03/03 21:21:52 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:52 INFO Remoting: Starting remoting 19/03/03 21:21:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:56027] 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 56027. 19/03/03 21:21:52 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:52 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-f5afdb69-9be9-4efe-839c-3d2b35a803e5 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore started with capacity 500.3 MB 19/03/03 21:21:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:52 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42852. 19/03/03 21:21:52 INFO netty.NettyBlockTransferService: Server created on 42852 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:52 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:42852 with 500.3 MB RAM, BlockManagerId(driver, localhost, 42852) 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: reduce at RNASeqReadCounts.scala:80 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 0 (reduce at RNASeqReadCounts.scala:80) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at RNASeqReadCounts.scala:79), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.0 KB, free 2.0 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1324.0 B, free 3.3 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:42852 (size: 1324.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at RNASeqReadCounts.scala:79) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2090 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2157 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 934 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 934 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 934 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 934 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 12 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 10 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 14 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 12 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at RNASeqReadCounts.scala:80) finished in 0.015 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 0 finished: reduce at RNASeqReadCounts.scala:80, took 0.020308 s 19/03/03 21:21:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:52 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:52 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:52 INFO spark.SparkContext: Successfully stopped SparkContext - run simple roll up 19/03/03 21:21:52 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:52 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) ComparisonTraversalEngineSuite: 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 57182. 19/03/03 21:21:52 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:52 INFO Remoting: Starting remoting 19/03/03 21:21:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:57183] 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 57183. 19/03/03 21:21:52 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:52 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-29f6c94c-01f5-4d0e-96c2-8c11081cafb1 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore started with capacity 500.3 MB 19/03/03 21:21:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:52 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41106. 19/03/03 21:21:52 INFO netty.NettyBlockTransferService: Server created on 41106 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:52 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41106 with 500.3 MB RAM, BlockManagerId(driver, localhost, 41106) 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: collect at ComparisonTraversalEngineSuite.scala:66 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 2 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 6 (keyBy at ComparisonTraversalEngine.scala:43) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 7 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 11 (keyBy at ComparisonTraversalEngine.scala:45) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 0 (collect at ComparisonTraversalEngineSuite.scala:66) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (collect at ComparisonTraversalEngineSuite.scala:66) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[2] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1913.0 B, free 5.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:41106 (size: 1913.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[2] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[7] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.5 KB, free 8.9 KB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1915.0 B, free 10.7 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41106 (size: 1915.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[7] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 5) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 18 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 6) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 7) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 19 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 10 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 19 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 22 ms on localhost (4/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.026 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 2) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 1, ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[6] at keyBy at ComparisonTraversalEngine.scala:43), which has no missing parents 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 6) in 6 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.1 KB, free 15.8 KB) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 5) in 14 ms on localhost (3/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 18.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41106 (size: 2.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[6] at keyBy at ComparisonTraversalEngine.scala:43) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 7) in 13 ms on localhost (4/4) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 11) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 10) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 8) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 9) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (groupBy at SingleReadBucket.scala:34) finished in 0.028 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[11] at keyBy at ComparisonTraversalEngine.scala:45), which has no missing parents 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 5.1 KB, free 23.5 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.5 KB, free 26.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:41106 (size: 2.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 11). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[11] at keyBy at ComparisonTraversalEngine.scala:45) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 10). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 11) in 8 ms on localhost (1/4) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 10) in 8 ms on localhost (2/4) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 9). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 8). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 9) in 18 ms on localhost (3/4) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 8) in 20 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at ComparisonTraversalEngine.scala:43) finished in 0.020 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 13 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 14 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 6 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 8 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (keyBy at ComparisonTraversalEngine.scala:45) finished in 0.017 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[15] at map at ComparisonTraversalEngine.scala:76), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.6 KB, free 29.6 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.0 KB, free 31.6 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:41106 (size: 2.0 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 4 (MapPartitionsRDD[15] at map at ComparisonTraversalEngine.scala:76) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 1124 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 1124 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 8 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 10 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 1139 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 1139 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 12 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 12 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 4 (collect at ComparisonTraversalEngineSuite.scala:66) finished in 0.013 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 0 finished: collect at ComparisonTraversalEngineSuite.scala:66, took 0.082685 s 19/03/03 21:21:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:52 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:52 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:52 INFO spark.SparkContext: Successfully stopped SparkContext - generate works on a simple RDD 19/03/03 21:21:52 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:52 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:52 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 55159. 19/03/03 21:21:52 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:52 INFO Remoting: Starting remoting 19/03/03 21:21:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55160] 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55160. 19/03/03 21:21:52 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:52 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-9e05115e-b0fe-42a6-8097-fe3a89299782 19/03/03 21:21:52 INFO storage.MemoryStore: MemoryStore started with capacity 500.3 MB 19/03/03 21:21:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:52 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38918. 19/03/03 21:21:52 INFO netty.NettyBlockTransferService: Server created on 38918 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:52 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:38918 with 500.3 MB RAM, BlockManagerId(driver, localhost, 38918) 19/03/03 21:21:52 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:52 INFO spark.SparkContext: Starting job: aggregate at ComparisonTraversalEngine.scala:88 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 2 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 6 (keyBy at ComparisonTraversalEngine.scala:43) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 7 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Registering RDD 11 (keyBy at ComparisonTraversalEngine.scala:45) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Got job 0 (aggregate at ComparisonTraversalEngine.scala:88) with 4 output partitions 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (aggregate at ComparisonTraversalEngine.scala:88) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1, ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[2] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1913.0 B, free 5.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:38918 (size: 1913.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[2] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[7] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.5 KB, free 8.9 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1915.0 B, free 10.7 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:38918 (size: 1915.0 B, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[7] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 18 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 5) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 9 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 6) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 7) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 20 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 39 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 38 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.043 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 2) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 1, ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[6] at keyBy at ComparisonTraversalEngine.scala:43), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.1 KB, free 15.8 KB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 18.4 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:38918 (size: 2.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[6] at keyBy at ComparisonTraversalEngine.scala:43) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 5) in 28 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 8) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 9) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 10) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 6) in 28 ms on localhost (3/4) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 10). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 11) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 7) in 31 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (groupBy at SingleReadBucket.scala:34) finished in 0.048 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 1) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 10) in 7 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[11] at keyBy at ComparisonTraversalEngine.scala:45), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 5.1 KB, free 23.5 KB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 11). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.5 KB, free 26.0 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:38918 (size: 2.5 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 8). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[11] at keyBy at ComparisonTraversalEngine.scala:45) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 9). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 8) in 15 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 11) in 8 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1883 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 9) in 16 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at ComparisonTraversalEngine.scala:43) finished in 0.017 s 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 3) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 8 ms on localhost (1/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 8 ms on localhost (2/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 1306 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 14 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 16 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (keyBy at ComparisonTraversalEngine.scala:45) finished in 0.016 s 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:52 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:52 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[15] at map at ComparisonTraversalEngine.scala:76), which has no missing parents 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.3 KB, free 30.3 KB) 19/03/03 21:21:52 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 32.6 KB) 19/03/03 21:21:52 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:38918 (size: 2.3 KB, free: 500.2 MB) 19/03/03 21:21:52 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 4 (MapPartitionsRDD[15] at map at ComparisonTraversalEngine.scala:76) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:52 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:52 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:52 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:52 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:52 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 1206 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 1171 bytes result sent to driver 19/03/03 21:21:52 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 1171 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 12 ms on localhost (1/4) 19/03/03 21:21:52 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 1171 bytes result sent to driver 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 14 ms on localhost (2/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 14 ms on localhost (3/4) 19/03/03 21:21:52 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 14 ms on localhost (4/4) 19/03/03 21:21:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:52 INFO scheduler.DAGScheduler: ResultStage 4 (aggregate at ComparisonTraversalEngine.scala:88) finished in 0.015 s 19/03/03 21:21:52 INFO scheduler.DAGScheduler: Job 0 finished: aggregate at ComparisonTraversalEngine.scala:88, took 0.106421 s 19/03/03 21:21:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:53 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:53 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:53 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:53 INFO spark.SparkContext: Successfully stopped SparkContext - combine works on a simple RDD 19/03/03 21:21:53 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:53 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) AFFromVCFSuite: 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriver' on port 55680. 19/03/03 21:21:53 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:53 INFO Remoting: Starting remoting 19/03/03 21:21:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55681] 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55681. 19/03/03 21:21:53 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:53 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:53 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-932ac30f-c387-42d8-b685-740fafc32431 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:53 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:53 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37694. 19/03/03 21:21:53 INFO netty.NettyBlockTransferService: Server created on 37694 19/03/03 21:21:53 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:37694 with 521.6 MB RAM, BlockManagerId(driver, localhost, 37694) 19/03/03 21:21:53 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 33.1 KB, free 33.1 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.7 KB, free 36.8 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:37694 (size: 3.7 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 0 from newAPIHadoopFile at AFFromVCF.scala:40 19/03/03 21:21:53 INFO input.FileInputFormat: Total input paths to process : 1 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: count at AFFromVCFSuite.scala:27 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 0 (count at AFFromVCFSuite.scala:27) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at AFFromVCFSuite.scala:27) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at flatMap at AFFromVCF.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.7 KB, free 39.5 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1633.0 B, free 41.1 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:37694 (size: 1633.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at flatMap at AFFromVCF.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2257 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:53 INFO rdd.NewHadoopRDD: Input split: file:/home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/af.vcf:0+365 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2057 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 49 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 0 (count at AFFromVCFSuite.scala:27) finished in 0.050 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 0 finished: count at AFFromVCFSuite.scala:27, took 0.060299 s 19/03/03 21:21:53 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:53 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:53 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:53 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:53 INFO spark.SparkContext: Successfully stopped SparkContext - converting variant contexts 19/03/03 21:21:53 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:53 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) ComparisonsSuite: 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriver' on port 55487. 19/03/03 21:21:53 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:53 INFO Remoting: Starting remoting 19/03/03 21:21:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55488] 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55488. 19/03/03 21:21:53 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:53 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:53 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-2c6f2550-8220-477c-85c7-340463e01854 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:53 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:53 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45558. 19/03/03 21:21:53 INFO netty.NettyBlockTransferService: Server created on 45558 19/03/03 21:21:53 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:45558 with 521.6 MB RAM, BlockManagerId(driver, localhost, 45558) 19/03/03 21:21:53 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 1 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 0 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1915.0 B, free 5.4 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:45558 (size: 1915.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 13 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 13 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 11 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 15 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.019 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.6 KB, free 9.9 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 12.2 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 6 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 1 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 0 finished: first at ComparisonsSuite.scala:40, took 0.043004 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 1 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 6) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 7) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 5) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 5). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 7). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 6 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 7) in 6 ms on localhost (2/3) 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 6). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 9 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 3 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 1 finished: first at ComparisonsSuite.scala:40, took 0.015535 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 5 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 2 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 4) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1912.0 B, free 24.5 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:45558 (size: 1912.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 9) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 11) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 8) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 10) 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 8 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 9 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 10 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 11) in 13 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (groupBy at SingleReadBucket.scala:34) finished in 0.016 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 12) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 12). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 6 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 5 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 2 finished: first at ComparisonsSuite.scala:40, took 0.032446 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 3 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 6) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 13) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 14) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 15) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 13). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 13) in 6 ms on localhost (1/3) 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 15). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 15) in 7 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 14) in 9 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 7 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 3 finished: first at ComparisonsSuite.scala:40, took 0.014744 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 9 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 4 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 8) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1915.0 B, free 43.6 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:45558 (size: 1915.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 8.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 8.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 8.0 (TID 17) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 8.0 (TID 19) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 16) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 8.0 (TID 18) 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 8.0 (TID 18). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 8.0 (TID 18) in 8 ms on localhost (1/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 8.0 (TID 17). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 16). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 16) in 13 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 17) in 12 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 8.0 (TID 19). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 8.0 (TID 19) in 13 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 20, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 20) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 20). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 20) in 5 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 9 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 4 finished: first at ComparisonsSuite.scala:40, took 0.033886 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 5 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 11 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 10) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 21, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 22, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 23, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 21) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 23) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 22) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 21). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 23). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 22). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 21) in 9 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 23) in 8 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 22) in 11 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 11 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 5 finished: first at ComparisonsSuite.scala:40, took 0.021628 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 13 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 6 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 12) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 12) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.5 KB, free 60.9 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1914.0 B, free 62.7 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:45558 (size: 1914.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 24) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 26) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 25) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 27) 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 24). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 26). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 25). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 24) in 16 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 25) in 14 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 26) in 11 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 27). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 27) in 14 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 12 (groupBy at SingleReadBucket.scala:34) finished in 0.022 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 13) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.6 KB, free 67.3 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.3 KB, free 69.6 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 28, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 28) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 28). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 28) in 6 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 13 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 6 finished: first at ComparisonsSuite.scala:40, took 0.040721 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 7 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 14) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.6 KB, free 74.2 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 76.5 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 29, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 30, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 31, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 29) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 30) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 31) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 31). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 29). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 30). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 31) in 7 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 29) in 7 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 30) in 9 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 15 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 7 finished: first at ComparisonsSuite.scala:40, took 0.019109 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 17 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 8 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 3.5 KB, free 80.0 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1913.0 B, free 81.9 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:45558 (size: 1913.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 33) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 34) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 35) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 32) 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 33). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 34). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 32). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 8 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 34) in 7 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 10 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 35). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 35) in 12 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (groupBy at SingleReadBucket.scala:34) finished in 0.015 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.6 KB, free 86.4 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.3 KB, free 88.7 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 36, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 36) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 36). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 36) in 8 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 17 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 8 finished: first at ComparisonsSuite.scala:40, took 0.033988 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 9 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 19 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 18) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.6 KB, free 93.3 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.3 KB, free 95.6 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 37, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 19.0 (TID 38, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 19.0 (TID 39, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 37) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 19.0 (TID 39) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 19.0 (TID 38) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 19.0 (TID 39). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 37). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 19.0 (TID 39) in 7 ms on localhost (1/3) 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 19.0 (TID 38). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 37) in 9 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 19.0 (TID 38) in 10 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 19 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 9 finished: first at ComparisonsSuite.scala:40, took 0.017822 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 21 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 10 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 21 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 20) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 20) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 3.5 KB, free 99.1 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 1914.0 B, free 101.0 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:45558 (size: 1914.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 20.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 20.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 20.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 20.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 20.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 20.0 (TID 40) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 20.0 (TID 42) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 20.0 (TID 43) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 20.0 (TID 41) 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 20.0 (TID 40). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 20.0 (TID 42). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 20.0 (TID 41). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 20.0 (TID 40) in 18 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 20.0 (TID 42) in 11 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 20.0 (TID 41) in 16 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 20.0 (TID 43). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 20.0 (TID 43) in 15 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 20 (groupBy at SingleReadBucket.scala:34) finished in 0.025 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 21) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.6 KB, free 105.5 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.3 KB, free 107.8 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 1 tasks 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 44, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 44) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 89 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:45558 in memory (size: 1912.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 88 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned shuffle 1 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 87 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 44). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 86 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 44) in 6 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 21 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:45558 in memory (size: 1915.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 10 finished: first at ComparisonsSuite.scala:40, took 0.056982 s 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 85 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 99 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 98 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on localhost:45558 in memory (size: 1913.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 97 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 96 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 95 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:45558 in memory (size: 1914.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 94 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 93 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 11 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 23 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 22) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 92 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:45558 in memory (size: 1915.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 91 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned shuffle 2 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.6 KB, free 30.6 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.ContextCleaner: Cleaned accumulator 90 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:45558 in memory (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 23.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 23.0 (TID 45, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 23.0 (TID 46, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 23.0 (TID 47, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 23.0 (TID 45) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 23.0 (TID 47) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 23.0 (TID 46) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 23.0 (TID 47). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 23.0 (TID 45). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 23.0 (TID 46). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 23.0 (TID 47) in 6 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 23.0 (TID 45) in 7 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 23.0 (TID 46) in 7 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 23 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 11 finished: first at ComparisonsSuite.scala:40, took 0.013607 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 25 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 12 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 25 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 24) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 24) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1915.0 B, free 24.5 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:45558 (size: 1915.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 24.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 24.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 24.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 24.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 24.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 24.0 (TID 48) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 24.0 (TID 49) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 24.0 (TID 50) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 24.0 (TID 51) 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 24.0 (TID 48). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 24.0 (TID 49). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 24.0 (TID 50). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 24.0 (TID 48) in 11 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 24.0 (TID 49) in 10 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 24.0 (TID 50) in 9 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 24.0 (TID 51). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 24.0 (TID 51) in 12 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 24 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 25) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 52) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 52). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 52) in 10 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 25 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 12 finished: first at ComparisonsSuite.scala:40, took 0.040362 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 13 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 27 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 26) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 27.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 27.0 (TID 53, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 27.0 (TID 54, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 27.0 (TID 55, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 27.0 (TID 55) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 27.0 (TID 53) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 27.0 (TID 54) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 27.0 (TID 53). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 27.0 (TID 55). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 27.0 (TID 54). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 27.0 (TID 53) in 12 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 27.0 (TID 55) in 13 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 27.0 (TID 54) in 15 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 27 (first at ComparisonsSuite.scala:40) finished in 0.016 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 13 finished: first at ComparisonsSuite.scala:40, took 0.023074 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Registering RDD 29 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 14 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 29 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 28) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 28) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 1913.0 B, free 43.6 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:45558 (size: 1913.0 B, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 28.0 with 4 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 28.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 28.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 28.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 28.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 28.0 (TID 56) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 28.0 (TID 58) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 28.0 (TID 57) 19/03/03 21:21:53 INFO executor.Executor: Running task 3.0 in stage 28.0 (TID 59) 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 28.0 (TID 56). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 28.0 (TID 57). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 28.0 (TID 58). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 28.0 (TID 56) in 18 ms on localhost (1/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 28.0 (TID 58) in 13 ms on localhost (2/4) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 28.0 (TID 57) in 16 ms on localhost (3/4) 19/03/03 21:21:53 INFO executor.Executor: Finished task 3.0 in stage 28.0 (TID 59). 1090 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 28.0 (TID 59) in 17 ms on localhost (4/4) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ShuffleMapStage 28 (groupBy at SingleReadBucket.scala:34) finished in 0.025 s 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:53 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 29) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 29.0 with 1 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 29.0 (TID 60, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 29.0 (TID 60) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 29.0 (TID 60). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 29.0 (TID 60) in 6 ms on localhost (1/1) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 29 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 14 finished: first at ComparisonsSuite.scala:40, took 0.047326 s 19/03/03 21:21:53 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:53 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 7 is 158 bytes 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Got job 15 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 31 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 30) 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:53 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:53 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:45558 (size: 2.3 KB, free: 521.6 MB) 19/03/03 21:21:53 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 3 tasks 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 61, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 62, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 31.0 (TID 63, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:53 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 61) 19/03/03 21:21:53 INFO executor.Executor: Running task 2.0 in stage 31.0 (TID 63) 19/03/03 21:21:53 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 62) 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:53 INFO executor.Executor: Finished task 2.0 in stage 31.0 (TID 63). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 61). 1171 bytes result sent to driver 19/03/03 21:21:53 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 62). 1306 bytes result sent to driver 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 61) in 9 ms on localhost (1/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 31.0 (TID 63) in 9 ms on localhost (2/3) 19/03/03 21:21:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 62) in 10 ms on localhost (3/3) 19/03/03 21:21:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 19/03/03 21:21:53 INFO scheduler.DAGScheduler: ResultStage 31 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:53 INFO scheduler.DAGScheduler: Job 15 finished: first at ComparisonsSuite.scala:40, took 0.017112 s 19/03/03 21:21:53 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:53 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:53 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:53 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:53 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:53 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:53 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:53 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriver' on port 51584. 19/03/03 21:21:53 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:53 INFO Remoting: Starting remoting 19/03/03 21:21:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:51585] 19/03/03 21:21:53 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 51585. 19/03/03 21:21:53 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:53 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:53 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-68840c3a-8336-4eb1-80b6-fe444641c731 19/03/03 21:21:53 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:53 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:54 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42204. 19/03/03 21:21:54 INFO netty.NettyBlockTransferService: Server created on 42204 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:54 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:42204 with 522.0 MB RAM, BlockManagerId(driver, localhost, 42204) 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:54 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:54 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:54 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:54 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:54 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:54 INFO spark.SparkContext: Running Spark version 1.6.1 - Dupe mismatches found 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:54 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriver' on port 55106. 19/03/03 21:21:54 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:54 INFO Remoting: Starting remoting 19/03/03 21:21:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55107] 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55107. 19/03/03 21:21:54 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:54 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:54 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-2c0f5947-01e9-4019-99a7-e77fae98ca33 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:54 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:54 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40283. 19/03/03 21:21:54 INFO netty.NettyBlockTransferService: Server created on 40283 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:54 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:40283 with 522.0 MB RAM, BlockManagerId(driver, localhost, 40283) 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 1 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 0 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1915.0 B, free 5.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:40283 (size: 1915.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 20 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 24 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 28 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 28 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.038 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.6 KB, free 9.9 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 12.2 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 8 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 1 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 0 finished: first at ComparisonsSuite.scala:40, took 0.058594 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 1 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 5) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 7) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 6) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 5). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 7). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 6). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 7 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 7) in 7 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 9 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 3 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 1 finished: first at ComparisonsSuite.scala:40, took 0.016081 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 5 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 2 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1912.0 B, free 24.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:40283 (size: 1912.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 8) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 10) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 11) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 9) 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 11 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 8 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 8 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 11) in 20 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (groupBy at SingleReadBucket.scala:34) finished in 0.025 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 12) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 12). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 7 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 5 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 2 finished: first at ComparisonsSuite.scala:40, took 0.044136 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 3 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 6) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 14) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 13) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 15) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 15). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 15) in 7 ms on localhost (1/3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 13). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 14) in 8 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 13) in 10 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 7 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 3 finished: first at ComparisonsSuite.scala:40, took 0.016678 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 9 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 4 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 8) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1915.0 B, free 43.6 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:40283 (size: 1915.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 8.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 8.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 16) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 8.0 (TID 17) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 8.0 (TID 18) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 8.0 (TID 19) 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 16). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 8.0 (TID 17). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 8.0 (TID 18). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 17) in 17 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 16) in 20 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 8.0 (TID 18) in 15 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 8.0 (TID 19). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 8.0 (TID 19) in 17 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (groupBy at SingleReadBucket.scala:34) finished in 0.026 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 20, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 20) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 20). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 20) in 9 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 9 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 4 finished: first at ComparisonsSuite.scala:40, took 0.045922 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 5 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 11 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 10) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 21, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 22, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 23, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 23) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 22) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 21) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 21). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 23). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 22). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 23) in 11 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 21) in 12 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 22) in 12 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 11 (first at ComparisonsSuite.scala:40) finished in 0.013 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 5 finished: first at ComparisonsSuite.scala:40, took 0.024267 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 13 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 6 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 12) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 12) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.5 KB, free 60.9 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1914.0 B, free 62.7 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:40283 (size: 1914.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 24) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 25) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 26) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 27) 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 24). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 25). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 25) in 10 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 24) in 11 ms on localhost (2/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 26). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 26) in 11 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 27). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 27) in 16 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 12 (groupBy at SingleReadBucket.scala:34) finished in 0.020 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 13) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.6 KB, free 67.3 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.3 KB, free 69.6 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 28, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 28) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 28). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 28) in 9 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 13 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 6 finished: first at ComparisonsSuite.scala:40, took 0.041573 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 7 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 14) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.6 KB, free 74.2 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 76.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 29, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 30, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 31, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 29) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 31) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 30) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 31). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 30). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 29). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 31) in 11 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 30) in 12 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 29) in 13 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 15 (first at ComparisonsSuite.scala:40) finished in 0.013 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 7 finished: first at ComparisonsSuite.scala:40, took 0.019602 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 17 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 8 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 3.5 KB, free 80.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1913.0 B, free 81.9 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:40283 (size: 1913.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 33) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 32) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 34) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 35) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 33). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 34). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 32). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 15 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 34) in 12 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 19 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 35). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 35) in 15 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (groupBy at SingleReadBucket.scala:34) finished in 0.024 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.6 KB, free 86.4 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.3 KB, free 88.7 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 36, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 36) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 36). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 36) in 7 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 17 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 8 finished: first at ComparisonsSuite.scala:40, took 0.046339 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 9 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 19 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 18) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.6 KB, free 93.3 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.3 KB, free 95.6 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 37, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 19.0 (TID 38, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 19.0 (TID 39, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 37) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 19.0 (TID 39) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 19.0 (TID 38) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 37). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 19.0 (TID 39). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 37) in 9 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 19.0 (TID 39) in 8 ms on localhost (2/3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 19.0 (TID 38). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 19.0 (TID 38) in 13 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 19 (first at ComparisonsSuite.scala:40) finished in 0.014 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 9 finished: first at ComparisonsSuite.scala:40, took 0.019661 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 21 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 10 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 21 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 20) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 20) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 3.5 KB, free 99.1 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 1914.0 B, free 101.0 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:40283 (size: 1914.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 20.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 20.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 20.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 20.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 20.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 20.0 (TID 41) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 20.0 (TID 40) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 20.0 (TID 43) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 20.0 (TID 42) 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 20.0 (TID 42). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 20.0 (TID 41). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 20.0 (TID 40). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 20.0 (TID 42) in 11 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 20.0 (TID 41) in 12 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 20.0 (TID 40) in 14 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 20.0 (TID 43). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 20.0 (TID 43) in 15 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 20 (groupBy at SingleReadBucket.scala:34) finished in 0.019 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 21) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.6 KB, free 105.5 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.3 KB, free 107.8 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 44, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 44) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 113 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:40283 in memory (size: 1912.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 112 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned shuffle 1 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 111 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 110 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:40283 in memory (size: 1915.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 109 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 44). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 123 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 44) in 6 ms on localhost (1/1) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 122 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 21 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 10 finished: first at ComparisonsSuite.scala:40, took 0.048536 s 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on localhost:40283 in memory (size: 1913.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 121 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 120 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 119 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:40283 in memory (size: 1914.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 118 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 117 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 116 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:40283 in memory (size: 1915.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 115 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned shuffle 2 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:40283 in memory (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.ContextCleaner: Cleaned accumulator 114 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 11 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 23 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 22) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 23.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 23.0 (TID 45, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 23.0 (TID 46, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 23.0 (TID 47, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 23.0 (TID 45) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 23.0 (TID 46) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 23.0 (TID 47) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 23.0 (TID 47). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 23.0 (TID 45). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 23.0 (TID 46). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 23.0 (TID 45) in 7 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 23.0 (TID 47) in 6 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 23.0 (TID 46) in 7 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 23 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 11 finished: first at ComparisonsSuite.scala:40, took 0.013242 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 25 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 12 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 25 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 24) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 24) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1915.0 B, free 24.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:40283 (size: 1915.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 24.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 24.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 24.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 24.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 24.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 24.0 (TID 49) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 24.0 (TID 51) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 24.0 (TID 48) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 24.0 (TID 50) 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 24.0 (TID 50). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 24.0 (TID 49). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 24.0 (TID 50) in 10 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 24.0 (TID 49) in 14 ms on localhost (2/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 24.0 (TID 48). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 24.0 (TID 51). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 24.0 (TID 48) in 20 ms on localhost (3/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 24.0 (TID 51) in 12 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 24 (groupBy at SingleReadBucket.scala:34) finished in 0.021 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 25) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 52) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 52). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 52) in 6 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 25 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 12 finished: first at ComparisonsSuite.scala:40, took 0.036862 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 13 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 27 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 26) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 27.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 27.0 (TID 53, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 27.0 (TID 54, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 27.0 (TID 55, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 27.0 (TID 53) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 27.0 (TID 55) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 27.0 (TID 54) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 27.0 (TID 53). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 27.0 (TID 55). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 27.0 (TID 53) in 8 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 27.0 (TID 55) in 8 ms on localhost (2/3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 27.0 (TID 54). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 27.0 (TID 54) in 15 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 27 (first at ComparisonsSuite.scala:40) finished in 0.016 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 13 finished: first at ComparisonsSuite.scala:40, took 0.025315 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 29 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 14 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 29 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 28) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 28) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 1913.0 B, free 43.6 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:40283 (size: 1913.0 B, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 28.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 28.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 28.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 28.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 28.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 28.0 (TID 59) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 28.0 (TID 58) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 28.0 (TID 56) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 28.0 (TID 57) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 28.0 (TID 57). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 28.0 (TID 58). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 28.0 (TID 56). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 28.0 (TID 58) in 8 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 28.0 (TID 57) in 9 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 28.0 (TID 56) in 11 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 28.0 (TID 59). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 28.0 (TID 59) in 12 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 28 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 29) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 29.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 29.0 (TID 60, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 29.0 (TID 60) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 29.0 (TID 60). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 29.0 (TID 60) in 5 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 29 (first at ComparisonsSuite.scala:40) finished in 0.005 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 14 finished: first at ComparisonsSuite.scala:40, took 0.033558 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 7 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 15 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 31 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 30) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:40283 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 61, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 62, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 31.0 (TID 63, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 31.0 (TID 63) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 61) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 62) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 61). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 31.0 (TID 63). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 62). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 61) in 6 ms on localhost (1/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 31.0 (TID 63) in 6 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 62) in 7 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 31 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 15 finished: first at ComparisonsSuite.scala:40, took 0.012419 s 19/03/03 21:21:54 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:54 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:54 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:54 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:54 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:54 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:54 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriver' on port 57904. 19/03/03 21:21:54 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:54 INFO Remoting: Starting remoting 19/03/03 21:21:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:57905] 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 57905. 19/03/03 21:21:54 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:54 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:54 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-84fa6bd7-f290-442a-8e76-07783159a054 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore started with capacity 522.4 MB 19/03/03 21:21:54 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:54 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40153. 19/03/03 21:21:54 INFO netty.NettyBlockTransferService: Server created on 40153 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:54 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:40153 with 522.4 MB RAM, BlockManagerId(driver, localhost, 40153) 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:54 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:54 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:54 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:54 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:54 INFO spark.SparkContext: Successfully stopped SparkContext - Mismatched mapped positions histogram generated 19/03/03 21:21:54 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:54 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:54 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriver' on port 58000. 19/03/03 21:21:54 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:54 INFO Remoting: Starting remoting 19/03/03 21:21:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58001] 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58001. 19/03/03 21:21:54 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:54 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:54 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-7ada2c64-40fd-4e45-bb64-4d4634529607 19/03/03 21:21:54 INFO storage.MemoryStore: MemoryStore started with capacity 522.4 MB 19/03/03 21:21:54 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:54 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:54 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42106. 19/03/03 21:21:54 INFO netty.NettyBlockTransferService: Server created on 42106 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:54 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:42106 with 522.4 MB RAM, BlockManagerId(driver, localhost, 42106) 19/03/03 21:21:54 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 1 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 0 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1915.0 B, free 5.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:42106 (size: 1915.0 B, free: 522.4 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 10 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 12 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 10 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 12 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.6 KB, free 9.9 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 12.2 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 5 ms on localhost (1/1) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 1 (first at ComparisonsSuite.scala:40) finished in 0.005 s 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 0 finished: first at ComparisonsSuite.scala:40, took 0.032677 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 1 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 3 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 5) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 6) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 7) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 7). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 5). 1171 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 7) in 5 ms on localhost (1/3) 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 6). 1306 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 6 ms on localhost (2/3) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 7 ms on localhost (3/3) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ResultStage 3 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Job 1 finished: first at ComparisonsSuite.scala:40, took 0.013346 s 19/03/03 21:21:54 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Registering RDD 5 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Got job 2 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 4) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1912.0 B, free 24.5 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:42106 (size: 1912.0 B, free: 522.4 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 8) 19/03/03 21:21:54 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 9) 19/03/03 21:21:54 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 11) 19/03/03 21:21:54 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 10) 19/03/03 21:21:54 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 8 ms on localhost (1/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 9 ms on localhost (2/4) 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 12 ms on localhost (3/4) 19/03/03 21:21:54 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 11) in 12 ms on localhost (4/4) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:54 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:54 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:54 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:54 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:54 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:54 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:54 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:54 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks 19/03/03 21:21:54 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:54 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 12) 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:54 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 12). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 5 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 5 (first at ComparisonsSuite.scala:40) finished in 0.005 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 2 finished: first at ComparisonsSuite.scala:40, took 0.033495 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 3 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 6) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 13) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 14) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 15) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 15). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 13). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 15) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 13) in 7 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 14) in 7 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 7 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 3 finished: first at ComparisonsSuite.scala:40, took 0.013431 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 9 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 4 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 8) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1915.0 B, free 43.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:42106 (size: 1915.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 8.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 8.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 16) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 8.0 (TID 18) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 8.0 (TID 17) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 8.0 (TID 19) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 8.0 (TID 17). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 8.0 (TID 18). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 16). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 16) in 10 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 17) in 8 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 8.0 (TID 18) in 7 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 8.0 (TID 19). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 8.0 (TID 19) in 15 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (groupBy at SingleReadBucket.scala:34) finished in 0.019 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 20, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 20) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 20). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 20) in 7 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 9 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 4 finished: first at ComparisonsSuite.scala:40, took 0.036410 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 5 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 11 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 10) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 21, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 22, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 23, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 22) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 23) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 21) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 23). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 21). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 22). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 23) in 8 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 21) in 8 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 22) in 8 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 11 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 5 finished: first at ComparisonsSuite.scala:40, took 0.014659 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 13 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 6 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 12) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 12) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.5 KB, free 60.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1914.0 B, free 62.7 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:42106 (size: 1914.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 25) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 26) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 24) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 27) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 25). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 26). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 24). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 25) in 8 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 26) in 7 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 24) in 10 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 27). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 27) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 12 (groupBy at SingleReadBucket.scala:34) finished in 0.014 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 13) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.6 KB, free 67.3 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.3 KB, free 69.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 28, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 28) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 28). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 28) in 7 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 13 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 6 finished: first at ComparisonsSuite.scala:40, took 0.031594 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 7 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 14) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.6 KB, free 74.2 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 76.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 29, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 30, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 31, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 30) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 31) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 29) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 29). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 30). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 31). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 29) in 7 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 31) in 7 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 30) in 8 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 15 (first at ComparisonsSuite.scala:40) finished in 0.009 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 7 finished: first at ComparisonsSuite.scala:40, took 0.017358 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 17 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 8 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 3.5 KB, free 80.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1913.0 B, free 81.9 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:42106 (size: 1913.0 B, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 32) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 34) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 35) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 33) 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 32). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 34). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 33). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 9 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 34) in 7 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 8 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 35). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 35) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (groupBy at SingleReadBucket.scala:34) finished in 0.015 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.6 KB, free 86.4 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.3 KB, free 88.7 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 36, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 36) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 36). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 36) in 7 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 17 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 8 finished: first at ComparisonsSuite.scala:40, took 0.032343 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 9 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 19 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 18) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.6 KB, free 93.3 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.3 KB, free 95.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 37, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 19.0 (TID 38, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 19.0 (TID 39, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 37) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 19.0 (TID 39) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 19.0 (TID 38) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 19.0 (TID 39). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 37). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 19.0 (TID 38). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 19.0 (TID 39) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 37) in 7 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 19.0 (TID 38) in 7 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 19 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 9 finished: first at ComparisonsSuite.scala:40, took 0.013012 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 21 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 10 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 21 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 20) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 20) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 3.5 KB, free 99.1 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 1914.0 B, free 101.0 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:42106 (size: 1914.0 B, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 20.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 20.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 20.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 20.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 20.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 20.0 (TID 40) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 20.0 (TID 43) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 20.0 (TID 42) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 20.0 (TID 41) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 20.0 (TID 41). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 20.0 (TID 42). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 20.0 (TID 40). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 20.0 (TID 42) in 7 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 20.0 (TID 40) in 11 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 20.0 (TID 41) in 9 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 20.0 (TID 43). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 20.0 (TID 43) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 20 (groupBy at SingleReadBucket.scala:34) finished in 0.015 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 21) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.6 KB, free 105.5 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.3 KB, free 107.8 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:42106 in memory (size: 1915.0 B, free: 522.3 MB) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 44, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 44) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 147 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 146 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on localhost:42106 in memory (size: 1913.0 B, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 145 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.3 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 144 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 44). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 143 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:42106 in memory (size: 1914.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 142 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 44) in 7 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 21 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 141 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 10 finished: first at ComparisonsSuite.scala:40, took 0.040366 s 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 140 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:42106 in memory (size: 1915.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 133 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 137 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:42106 in memory (size: 1912.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 136 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 1 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 135 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 134 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 139 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 2 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 158 bytes 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:42106 in memory (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 138 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 11 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 23 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 22) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 23.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 23.0 (TID 45, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 23.0 (TID 46, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 23.0 (TID 47, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 23.0 (TID 45) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 23.0 (TID 47) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 23.0 (TID 46) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 23.0 (TID 45). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 23.0 (TID 47). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 23.0 (TID 47) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 23.0 (TID 45) in 7 ms on localhost (2/3) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 23.0 (TID 46). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 23.0 (TID 46) in 8 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 23 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 11 finished: first at ComparisonsSuite.scala:40, took 0.013989 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 25 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 12 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 25 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 24) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 24) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1915.0 B, free 24.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:42106 (size: 1915.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 24.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 24.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 24.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 24.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 24.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 24.0 (TID 48) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 24.0 (TID 51) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 24.0 (TID 50) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 24.0 (TID 49) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 24.0 (TID 49). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 24.0 (TID 48). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 24.0 (TID 50). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 24.0 (TID 48) in 10 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 24.0 (TID 49) in 9 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 24.0 (TID 50) in 8 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 24.0 (TID 51). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 24.0 (TID 51) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 24 (groupBy at SingleReadBucket.scala:34) finished in 0.014 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 25) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 52) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 52). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 52) in 5 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 25 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 12 finished: first at ComparisonsSuite.scala:40, took 0.030474 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 13 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 27 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 26) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 27.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 27.0 (TID 53, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 27.0 (TID 54, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 27.0 (TID 55, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 27.0 (TID 53) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 27.0 (TID 54) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 27.0 (TID 55) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 27.0 (TID 53). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 27.0 (TID 54). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 27.0 (TID 55). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 27.0 (TID 54) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 27.0 (TID 53) in 7 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 27.0 (TID 55) in 6 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 27 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 13 finished: first at ComparisonsSuite.scala:40, took 0.011752 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 29 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 14 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 29 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 28) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 28) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 1913.0 B, free 43.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:42106 (size: 1913.0 B, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 28.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 28.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 28.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 28.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 28.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 28.0 (TID 56) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 28.0 (TID 57) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 28.0 (TID 59) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 28.0 (TID 58) 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 28.0 (TID 56). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 28.0 (TID 57). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 28.0 (TID 58). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 28.0 (TID 56) in 13 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 28.0 (TID 58) in 11 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 28.0 (TID 57) in 13 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 28.0 (TID 59). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 28.0 (TID 59) in 14 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 28 (groupBy at SingleReadBucket.scala:34) finished in 0.019 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 29) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 29.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 29.0 (TID 60, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 29.0 (TID 60) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 29.0 (TID 60). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 29.0 (TID 60) in 6 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 29 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 14 finished: first at ComparisonsSuite.scala:40, took 0.040543 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 7 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 15 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 31 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 30) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:42106 (size: 2.3 KB, free: 522.4 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 61, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 62, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 31.0 (TID 63, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 31.0 (TID 63) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 62) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 61) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 61). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 31.0 (TID 63). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 62). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 61) in 7 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 31.0 (TID 63) in 8 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 62) in 8 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 31 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 15 finished: first at ComparisonsSuite.scala:40, took 0.017295 s 19/03/03 21:21:55 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:55 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:55 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:55 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:55 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:55 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:55 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:55 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:55 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'sparkDriver' on port 51617. 19/03/03 21:21:55 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:55 INFO Remoting: Starting remoting 19/03/03 21:21:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:51618] 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 51618. 19/03/03 21:21:55 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:55 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:55 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-9c5822fb-53e3-4d1a-8762-cb8be963b57a 19/03/03 21:21:55 INFO storage.MemoryStore: MemoryStore started with capacity 508.5 MB 19/03/03 21:21:55 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:55 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41874. 19/03/03 21:21:55 INFO netty.NettyBlockTransferService: Server created on 41874 19/03/03 21:21:55 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:55 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41874 with 508.5 MB RAM, BlockManagerId(driver, localhost, 41874) 19/03/03 21:21:55 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:55 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:55 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:55 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:55 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:55 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:55 INFO spark.SparkContext: Successfully stopped SparkContext - Test map quality scores 19/03/03 21:21:55 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:55 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:55 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:55 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'sparkDriver' on port 53995. 19/03/03 21:21:55 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:55 INFO Remoting: Starting remoting 19/03/03 21:21:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:53996] 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 53996. 19/03/03 21:21:55 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:55 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:55 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-f0c8df88-7fe0-45b5-b0ad-584fd7ec411e 19/03/03 21:21:55 INFO storage.MemoryStore: MemoryStore started with capacity 508.5 MB 19/03/03 21:21:55 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:55 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:55 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46831. 19/03/03 21:21:55 INFO netty.NettyBlockTransferService: Server created on 46831 19/03/03 21:21:55 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:55 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:46831 with 508.5 MB RAM, BlockManagerId(driver, localhost, 46831) 19/03/03 21:21:55 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 1 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 0 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.5 KB, free 3.5 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1915.0 B, free 5.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:46831 (size: 1915.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 12 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 11 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 14 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 15 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (groupBy at SingleReadBucket.scala:34) finished in 0.019 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.6 KB, free 9.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 12.2 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 11 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 1 (first at ComparisonsSuite.scala:40) finished in 0.012 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 0 finished: first at ComparisonsSuite.scala:40, took 0.050692 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 1 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 3 (MapPartitionsRDD[3] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 5) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 6) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 7) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 6). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 7). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 5). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 7) in 10 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 11 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 11 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 3 (first at ComparisonsSuite.scala:40) finished in 0.012 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 1 finished: first at ComparisonsSuite.scala:40, took 0.024002 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 5 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 2 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1912.0 B, free 24.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:46831 (size: 1912.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[5] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 8) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 11) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 10) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 9) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 9 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 10 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 7 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 11) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (groupBy at SingleReadBucket.scala:34) finished in 0.015 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 12) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 12). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 5 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 5 (first at ComparisonsSuite.scala:40) finished in 0.005 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 2 finished: first at ComparisonsSuite.scala:40, took 0.031122 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 3 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 6) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 7 (MapPartitionsRDD[7] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 13) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 15) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 14) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 13). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 15). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 14). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 13) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 15) in 5 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 14) in 7 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 7 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 3 finished: first at ComparisonsSuite.scala:40, took 0.013747 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 9 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 4 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 8) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1915.0 B, free 43.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:46831 (size: 1915.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[9] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 8.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 8.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 16) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 8.0 (TID 19) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 8.0 (TID 18) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 8.0 (TID 17) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 8.0 (TID 17). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 8.0 (TID 18). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 16). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 17) in 7 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 8.0 (TID 18) in 6 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 16) in 10 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 8.0 (TID 19). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 8.0 (TID 19) in 10 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (groupBy at SingleReadBucket.scala:34) finished in 0.014 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 20, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 20) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 20). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 20) in 5 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 9 (first at ComparisonsSuite.scala:40) finished in 0.005 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 4 finished: first at ComparisonsSuite.scala:40, took 0.027623 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 5 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 11 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 10) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 11 (MapPartitionsRDD[11] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 11.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 21, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 22, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 23, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 11.0 (TID 22) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 11.0 (TID 21) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 11.0 (TID 23) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 11.0 (TID 21). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 11.0 (TID 22). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 21) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 11.0 (TID 23). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 22) in 6 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 23) in 7 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 11 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 5 finished: first at ComparisonsSuite.scala:40, took 0.012264 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 13 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 6 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 12) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 12) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.5 KB, free 60.9 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1914.0 B, free 62.7 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:46831 (size: 1914.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 12 (MapPartitionsRDD[13] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 12.0 (TID 24) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 12.0 (TID 27) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 12.0 (TID 25) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 12.0 (TID 26) 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 12.0 (TID 26). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 12.0 (TID 24). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 26) in 7 ms on localhost (1/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 12.0 (TID 25). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 24) in 10 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 25) in 10 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 12.0 (TID 27). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 27) in 11 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 12 (groupBy at SingleReadBucket.scala:34) finished in 0.016 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 13) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.6 KB, free 67.3 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.3 KB, free 69.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 28, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 28) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 28). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 28) in 6 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 13 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 6 finished: first at ComparisonsSuite.scala:40, took 0.036389 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 7 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 14) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.6 KB, free 74.2 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 76.5 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[15] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 29, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 15.0 (TID 30, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 15.0 (TID 31, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 15.0 (TID 31) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 29) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 15.0 (TID 30) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 29). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 29) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 15.0 (TID 31). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 15.0 (TID 31) in 9 ms on localhost (2/3) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 15.0 (TID 30). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 15.0 (TID 30) in 11 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 15 (first at ComparisonsSuite.scala:40) finished in 0.012 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 7 finished: first at ComparisonsSuite.scala:40, took 0.022010 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 17 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 8 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 17 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 16) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 16) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 3.5 KB, free 80.0 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1913.0 B, free 81.9 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:46831 (size: 1913.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 16 (MapPartitionsRDD[17] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 32) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 35) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 33) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 34) 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 34). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 33). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 32). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 34) in 8 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 16 ms on localhost (2/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 11 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 35). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 35) in 15 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 16 (groupBy at SingleReadBucket.scala:34) finished in 0.024 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 17) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.6 KB, free 86.4 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.3 KB, free 88.7 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 17 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 1 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 36, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 36) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 36). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 36) in 6 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 17 (first at ComparisonsSuite.scala:40) finished in 0.006 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 8 finished: first at ComparisonsSuite.scala:40, took 0.039084 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 158 bytes 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 9 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 19 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 18) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.6 KB, free 93.3 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.3 KB, free 95.6 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 19 (MapPartitionsRDD[19] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 3 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 37, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 19.0 (TID 38, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 19.0 (TID 39, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 37) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 19.0 (TID 39) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 19.0 (TID 38) 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 19.0 (TID 39). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 37). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 19.0 (TID 38). 1306 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 19.0 (TID 39) in 6 ms on localhost (1/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 37) in 6 ms on localhost (2/3) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 19.0 (TID 38) in 7 ms on localhost (3/3) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 19 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 9 finished: first at ComparisonsSuite.scala:40, took 0.014313 s 19/03/03 21:21:55 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Registering RDD 21 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Got job 10 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 21 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 20) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 20) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 3.5 KB, free 99.1 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 1914.0 B, free 101.0 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:46831 (size: 1914.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[21] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 20.0 with 4 tasks 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 20.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 20.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 20.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 20.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 1.0 in stage 20.0 (TID 41) 19/03/03 21:21:55 INFO executor.Executor: Running task 2.0 in stage 20.0 (TID 42) 19/03/03 21:21:55 INFO executor.Executor: Running task 3.0 in stage 20.0 (TID 43) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 20.0 (TID 40) 19/03/03 21:21:55 INFO executor.Executor: Finished task 1.0 in stage 20.0 (TID 41). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO executor.Executor: Finished task 2.0 in stage 20.0 (TID 42). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 20.0 (TID 41) in 11 ms on localhost (1/4) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 20.0 (TID 42) in 11 ms on localhost (2/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 20.0 (TID 40). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 20.0 (TID 40) in 15 ms on localhost (3/4) 19/03/03 21:21:55 INFO executor.Executor: Finished task 3.0 in stage 20.0 (TID 43). 1090 bytes result sent to driver 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 20.0 (TID 43) in 14 ms on localhost (4/4) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ShuffleMapStage 20 (groupBy at SingleReadBucket.scala:34) finished in 0.018 s 19/03/03 21:21:55 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:55 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 21) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.6 KB, free 105.5 KB) 19/03/03 21:21:55 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.3 KB, free 107.8 KB) 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 1 tasks 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 44, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:55 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 44) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 160 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:55 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 1 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 159 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 158 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:46831 in memory (size: 1915.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 157 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 171 19/03/03 21:21:55 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 44). 1171 bytes result sent to driver 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 170 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on localhost:46831 in memory (size: 1913.0 B, free: 508.5 MB) 19/03/03 21:21:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 44) in 7 ms on localhost (1/1) 19/03/03 21:21:55 INFO scheduler.DAGScheduler: ResultStage 21 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 169 19/03/03 21:21:55 INFO scheduler.DAGScheduler: Job 10 finished: first at ComparisonsSuite.scala:40, took 0.050773 s 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned shuffle 4 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 168 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:55 INFO spark.ContextCleaner: Cleaned accumulator 167 19/03/03 21:21:55 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:46831 in memory (size: 1914.0 B, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 166 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 165 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 164 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:46831 in memory (size: 1915.0 B, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 163 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 2 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 158 bytes 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 162 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 11 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 23 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 22) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:46831 in memory (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 161 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:46831 in memory (size: 1912.0 B, free: 508.5 MB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.6 KB, free 16.8 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.3 KB, free 19.1 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 23 (MapPartitionsRDD[23] at map at SingleReadBucket.scala:35) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 23.0 with 3 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 23.0 (TID 45, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 23.0 (TID 46, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 23.0 (TID 47, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 23.0 (TID 45) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 23.0 (TID 46) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 23.0 (TID 47) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 23.0 (TID 45). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 23.0 (TID 47). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 23.0 (TID 46). 1306 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 23.0 (TID 45) in 6 ms on localhost (1/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 23.0 (TID 47) in 6 ms on localhost (2/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 23.0 (TID 46) in 7 ms on localhost (3/3) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 23 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 11 finished: first at ComparisonsSuite.scala:40, took 0.012506 s 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 25 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 12 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 25 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 24) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 24) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 3.5 KB, free 22.6 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1915.0 B, free 24.5 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:46831 (size: 1915.0 B, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 24 (MapPartitionsRDD[25] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 24.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 24.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 24.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 24.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 24.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 24.0 (TID 48) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 24.0 (TID 50) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 24.0 (TID 49) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 24.0 (TID 51) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 24.0 (TID 50). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 24.0 (TID 49). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 24.0 (TID 48). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 24.0 (TID 50) in 6 ms on localhost (1/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 24.0 (TID 49) in 7 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 24.0 (TID 48) in 9 ms on localhost (3/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 24.0 (TID 51). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 24.0 (TID 51) in 10 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 24 (groupBy at SingleReadBucket.scala:34) finished in 0.013 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 25) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.6 KB, free 29.0 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.3 KB, free 31.4 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 1 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 52, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 52) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 52). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 52) in 6 ms on localhost (1/1) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 25 (first at ComparisonsSuite.scala:40) finished in 0.007 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 12 finished: first at ComparisonsSuite.scala:40, took 0.028538 s 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:56 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 158 bytes 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 13 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 27 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 26) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.6 KB, free 35.9 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.2 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 27 (MapPartitionsRDD[27] at map at SingleReadBucket.scala:35) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 27.0 with 3 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 27.0 (TID 53, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 27.0 (TID 54, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 27.0 (TID 55, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 27.0 (TID 53) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 27.0 (TID 54) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 27.0 (TID 55) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 27.0 (TID 54). 1306 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 27.0 (TID 55). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 27.0 (TID 53). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 27.0 (TID 55) in 6 ms on localhost (1/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 27.0 (TID 54) in 8 ms on localhost (2/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 27.0 (TID 53) in 10 ms on localhost (3/3) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 27 (first at ComparisonsSuite.scala:40) finished in 0.010 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 13 finished: first at ComparisonsSuite.scala:40, took 0.019244 s 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 29 (groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 14 (first at ComparisonsSuite.scala:40) with 1 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 29 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 28) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 28) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 3.5 KB, free 41.7 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 1913.0 B, free 43.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:46831 (size: 1913.0 B, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 28 (MapPartitionsRDD[29] at groupBy at SingleReadBucket.scala:34) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 28.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 28.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 28.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 28.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 2012 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 28.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 2144 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 28.0 (TID 56) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 28.0 (TID 59) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 28.0 (TID 58) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 28.0 (TID 57) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 28.0 (TID 56). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 28.0 (TID 56) in 11 ms on localhost (1/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 28.0 (TID 59). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 28.0 (TID 57). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 28.0 (TID 58). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 28.0 (TID 57) in 13 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 28.0 (TID 58) in 12 ms on localhost (3/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 28.0 (TID 59) in 11 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 28 (groupBy at SingleReadBucket.scala:34) finished in 0.017 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 29) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 4.6 KB, free 48.2 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 2.3 KB, free 50.5 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 29 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 29.0 with 1 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 29.0 (TID 60, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 29.0 (TID 60) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 29.0 (TID 60). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 29.0 (TID 60) in 8 ms on localhost (1/1) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 29 (first at ComparisonsSuite.scala:40) finished in 0.008 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 14 finished: first at ComparisonsSuite.scala:40, took 0.036952 s 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: first at ComparisonsSuite.scala:40 19/03/03 21:21:56 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 7 is 158 bytes 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 15 (first at ComparisonsSuite.scala:40) with 3 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 31 (first at ComparisonsSuite.scala:40) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 30) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.6 KB, free 55.0 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.3 KB, free 57.4 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:46831 (size: 2.3 KB, free: 508.5 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ResultStage 31 (MapPartitionsRDD[31] at map at SingleReadBucket.scala:35) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 3 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 61, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 62, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 31.0 (TID 63, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 61) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 62) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 31.0 (TID 63) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 61). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 62). 1306 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 31.0 (TID 63). 1171 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 61) in 15 ms on localhost (1/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 31.0 (TID 63) in 14 ms on localhost (2/3) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 62) in 15 ms on localhost (3/3) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 31 (first at ComparisonsSuite.scala:40) finished in 0.016 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 15 finished: first at ComparisonsSuite.scala:40, took 0.020672 s 19/03/03 21:21:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:56 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:56 INFO spark.SparkContext: Successfully stopped SparkContext 19/03/03 21:21:56 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:56 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriver' on port 56867. 19/03/03 21:21:56 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:56 INFO Remoting: Starting remoting 19/03/03 21:21:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:56868] 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 56868. 19/03/03 21:21:56 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:56 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:56 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-3dac8b3e-7203-4c2e-a060-6f89893a48cb 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:56 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:56 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44678. 19/03/03 21:21:56 INFO netty.NettyBlockTransferService: Server created on 44678 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:56 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:44678 with 521.6 MB RAM, BlockManagerId(driver, localhost, 44678) 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:56 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:56 INFO spark.SparkContext: Successfully stopped SparkContext - Test base quality scores 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:56 WARN contamination.VariantSite: Failed to create observation for read null due to java.lang.StringIndexOutOfBoundsException: String index out of range: 0. Dropping... 19/03/03 21:21:56 INFO spark.SparkContext: Running Spark version 1.6.1 VariantSiteSuite: 19/03/03 21:21:56 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) - read overlaps a site and agrees with the reference - read overlaps a site and agrees with the alt - read overlaps a site and is an other-alt - read has no bases and should fail on conversion ReadStatsSuite: - perfect read - noQualRead should be equivalent to the perfect read, and should not NPE - read failed vendor qc - read is a dupe - all of the bases in a read are low quality - the start/end of the read have low quality scores - read's mapping quality is zero - read is mapped with low, but non-zero mapping quality - first-of-pair read overlaps mate - second-of-pair read overlaps mate - unpaired, but otherwise perfect read - merge two read stats instances 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriver' on port 55196. 19/03/03 21:21:56 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:56 INFO Remoting: Starting remoting 19/03/03 21:21:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55197] 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55197. 19/03/03 21:21:56 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:56 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:56 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-ad8d3bb7-b275-4d34-af64-478a8f90832b 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:56 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:56 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35560. 19/03/03 21:21:56 INFO netty.NettyBlockTransferService: Server created on 35560 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:56 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:35560 with 521.6 MB RAM, BlockManagerId(driver, localhost, 35560) 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: reduce at ReadStats.scala:253 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 0 (reduce at ReadStats.scala:253) with 4 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at ReadStats.scala:253) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at ReadStats.scala:252), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.2 KB, free 2.2 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1430.0 B, free 3.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:35560 (size: 1430.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at ReadStats.scala:252) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2290 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2422 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2290 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2422 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 952 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 952 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 952 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 952 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 12 ms on localhost (1/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 16 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 14 ms on localhost (3/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 17 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at ReadStats.scala:253) finished in 0.019 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 0 finished: reduce at ReadStats.scala:253, took 0.027551 s 19/03/03 21:21:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:56 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:56 INFO spark.SparkContext: Successfully stopped SparkContext - run an aggregate across all reads 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:56 WARN rna.RNASeqAlignmentCounts: Found unknown feature type random. Ignoring. 19/03/03 21:21:56 WARN rna.RNASeqAlignmentCounts: Found unknown feature type SO:0000110. Ignoring. 19/03/03 21:21:56 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:56 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) RNASeqAlignmentCountsSuite: - should get an intergenic count if we pass a None - should get an intragenic count if we pass a transcript - should get an exonic count if we pass an exon - should get an intronic count if we pass an intron - should get an rRNA count if we pass an rRNA site - should get nothing if we pass a random feature - should get an intragenic count if we pass an SO transcript - should get an exonic count if we pass an SO exon - should get an intronic count if we pass an SO intron - should get an rRNA count if we pass an SO rRNA site - should get nothing if we pass a random SO feature 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriver' on port 54655. 19/03/03 21:21:56 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:56 INFO Remoting: Starting remoting 19/03/03 21:21:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:54656] 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 54656. 19/03/03 21:21:56 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:56 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:56 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-2faa8bf1-4e5b-48c8-8a35-8242c5e2ab46 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:56 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:56 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37101. 19/03/03 21:21:56 INFO netty.NettyBlockTransferService: Server created on 37101 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:56 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:37101 with 521.6 MB RAM, BlockManagerId(driver, localhost, 37101) 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: reduce at RNASeqAlignmentCounts.scala:72 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 0 (reduce at RNASeqAlignmentCounts.scala:72) with 4 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at RNASeqAlignmentCounts.scala:72) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at flatMap at RNASeqAlignmentCounts.scala:71), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2032.0 B, free 2032.0 B) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1266.0 B, free 3.2 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:37101 (size: 1266.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at flatMap at RNASeqAlignmentCounts.scala:71) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2014 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2054 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2067 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2087 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 937 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 937 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 937 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 937 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 21 ms on localhost (1/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 17 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 21 ms on localhost (3/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 28 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at RNASeqAlignmentCounts.scala:72) finished in 0.028 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 0 finished: reduce at RNASeqAlignmentCounts.scala:72, took 0.037037 s 19/03/03 21:21:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:56 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:56 INFO spark.SparkContext: Successfully stopped SparkContext - run aggregate 19/03/03 21:21:56 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:56 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) GenotypeConcordanceRDDFunctionsSuite: 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriver' on port 56649. 19/03/03 21:21:56 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:56 INFO Remoting: Starting remoting 19/03/03 21:21:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:56650] 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 56650. 19/03/03 21:21:56 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:56 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:56 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-3399c2b2-729d-4369-9155-5a72bf0f0c68 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:56 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:56 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36693. 19/03/03 21:21:56 INFO netty.NettyBlockTransferService: Server created on 36693 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:56 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:36693 with 521.6 MB RAM, BlockManagerId(driver, localhost, 36693) 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:42 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 5 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 3 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 5 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 12 (union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 0 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:42) with 8 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:42) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 3) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 3) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.8 KB, free 2.8 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1705.0 B, free 4.5 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:36693 (size: 1705.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[3] at keyBy at GenotypeConcordanceRDDFunctions.scala:49), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.8 KB, free 7.4 KB) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1709.0 B, free 9.0 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:36693 (size: 1709.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[3] at keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.8 KB, free 11.9 KB) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1676.0 B, free 13.5 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:36693 (size: 1676.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 24 ms on localhost (1/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 29 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 25 ms on localhost (3/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 12 ms on localhost (1/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 12 ms on localhost (2/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 29 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.039 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 1, ShuffleMapStage 2) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 15 ms on localhost (3/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 12 ms on localhost (1/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 10 ms on localhost (2/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 8 ms on localhost (3/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 18 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) finished in 0.034 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 2) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 14 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.036 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ShuffleMapStage 3, ResultStage 4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (UnionRDD[12] at union at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.7 KB, free 18.2 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.4 KB, free 20.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:36693 (size: 2.4 KB, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 3 (UnionRDD[12] at union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 8 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 3.0 (TID 16, localhost, partition 4,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 4.0 in stage 3.0 (TID 16) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 3.0 (TID 17, localhost, partition 5,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 5.0 in stage 3.0 (TID 17) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 13 ms on localhost (1/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 12 ms on localhost (2/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 3.0 (TID 18, localhost, partition 6,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 6.0 in stage 3.0 (TID 18) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 14 ms on localhost (3/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 3.0 (TID 19, localhost, partition 7,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 7.0 in stage 3.0 (TID 19) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 20 ms on localhost (4/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 4.0 in stage 3.0 (TID 16). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 5.0 in stage 3.0 (TID 17). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 6.0 in stage 3.0 (TID 18). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 3.0 (TID 16) in 14 ms on localhost (5/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 3.0 (TID 17) in 13 ms on localhost (6/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 3.0 (TID 18) in 10 ms on localhost (7/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 7.0 in stage 3.0 (TID 19). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 3.0 (TID 19) in 15 ms on localhost (8/8) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (union at GenotypeConcordanceRDDFunctions.scala:65) finished in 0.036 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (ShuffledRDD[13] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 2.6 KB, free 23.1 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1474.0 B, free 24.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:36693 (size: 1474.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 8 missing tasks from ResultStage 4 (ShuffledRDD[13] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 8 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 4.0 (TID 20, localhost, partition 5,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 21, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 22, localhost, partition 1,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 23, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 5.0 in stage 4.0 (TID 20) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 22) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 23) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 21) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 22). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 23). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 24, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 24) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 4.0 (TID 25, localhost, partition 4,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 4.0 in stage 4.0 (TID 25) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 22) in 7 ms on localhost (1/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 23) in 7 ms on localhost (2/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 21). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 4.0 (TID 26, localhost, partition 6,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 6.0 in stage 4.0 (TID 26) 19/03/03 21:21:56 INFO executor.Executor: Finished task 5.0 in stage 4.0 (TID 20). 1217 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 4.0 (TID 27, localhost, partition 7,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 7.0 in stage 4.0 (TID 27) 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 24). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 21) in 11 ms on localhost (3/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 4.0 in stage 4.0 (TID 25). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 4.0 (TID 20) in 11 ms on localhost (4/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 24) in 7 ms on localhost (5/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 4.0 (TID 25) in 6 ms on localhost (6/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 7.0 in stage 4.0 (TID 27). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 6.0 in stage 4.0 (TID 26). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 4.0 (TID 27) in 5 ms on localhost (7/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 4.0 (TID 26) in 7 ms on localhost (8/8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 4 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:42) finished in 0.017 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 0 finished: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:42, took 0.139099 s 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:50 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 19 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 17 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 19 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Registering RDD 26 (union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 1 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:50) with 8 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:50) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[19] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 2.8 KB, free 27.4 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1711.0 B, free 29.1 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:36693 (size: 1711.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[19] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[17] at keyBy at GenotypeConcordanceRDDFunctions.scala:49), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 2.8 KB, free 31.9 KB) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 28, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1707.0 B, free 33.6 KB) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 29, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:36693 (size: 1707.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[17] at keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 5.0 (TID 30, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 5.0 (TID 31, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 5.0 (TID 30) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 28) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 29) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 5.0 (TID 31) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 7 (MapPartitionsRDD[19] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 2.8 KB, free 36.4 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1680.0 B, free 38.1 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:36693 (size: 1680.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 5.0 (TID 30). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 7 (MapPartitionsRDD[19] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 4 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 32) 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 29). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 28). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 6.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 5.0 (TID 30) in 9 ms on localhost (1/4) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 33) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 32). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 5.0 (TID 31). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 6.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 29) in 12 ms on localhost (2/4) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 6.0 (TID 34) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 6.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 6.0 (TID 35) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 28) in 15 ms on localhost (3/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 6.0 (TID 34). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 36, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 36) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:36693 in memory (size: 1709.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 33). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 37, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 5.0 (TID 31) in 30 ms on localhost (4/4) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 37) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.035 s 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 32) in 27 ms on localhost (1/4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 6, ShuffleMapStage 7) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9, ShuffleMapStage 8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 36). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 184 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 38, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 6.0 (TID 34) in 24 ms on localhost (2/4) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 38) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:36693 in memory (size: 1705.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 183 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 3 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 2 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 1 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 37). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 7.0 (TID 39, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 7.0 (TID 39) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 6.0 (TID 33) in 28 ms on localhost (3/4) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:36693 in memory (size: 1474.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 187 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 37) in 7 ms on localhost (1/4) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 38). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 36) in 10 ms on localhost (2/4) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:36693 in memory (size: 2.4 KB, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 186 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:36693 in memory (size: 1676.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.ContextCleaner: Cleaned accumulator 185 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 6.0 (TID 35). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 38) in 7 ms on localhost (3/4) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 6.0 (TID 35) in 31 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 6 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) finished in 0.040 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 7) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9, ShuffleMapStage 8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 7.0 (TID 39). 1090 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 7.0 (TID 39) in 12 ms on localhost (4/4) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 7 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.040 s 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9, ShuffleMapStage 8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 8 (UnionRDD[26] at union at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.7 KB, free 18.2 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.4 KB, free 20.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:36693 (size: 2.4 KB, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 8 (UnionRDD[26] at union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 8 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 8.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 8.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 8.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 40) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 8.0 (TID 42) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 8.0 (TID 41) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 8.0 (TID 43) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 40). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 8.0 (TID 44, localhost, partition 4,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 4.0 in stage 8.0 (TID 44) 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 8.0 (TID 42). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 8.0 (TID 45, localhost, partition 5,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 5.0 in stage 8.0 (TID 45) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 40) in 12 ms on localhost (1/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 8.0 (TID 42) in 12 ms on localhost (2/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 8.0 (TID 43). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 8.0 (TID 46, localhost, partition 6,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 6.0 in stage 8.0 (TID 46) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 8.0 (TID 41). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 8.0 (TID 47, localhost, partition 7,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 8.0 (TID 43) in 14 ms on localhost (3/8) 19/03/03 21:21:56 INFO executor.Executor: Running task 7.0 in stage 8.0 (TID 47) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 8.0 (TID 41) in 16 ms on localhost (4/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 4.0 in stage 8.0 (TID 44). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 5.0 in stage 8.0 (TID 45). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 8.0 (TID 44) in 9 ms on localhost (5/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 8.0 (TID 45) in 9 ms on localhost (6/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 6.0 in stage 8.0 (TID 46). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 8.0 (TID 46) in 8 ms on localhost (7/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 7.0 in stage 8.0 (TID 47). 1310 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 8.0 (TID 47) in 13 ms on localhost (8/8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (union at GenotypeConcordanceRDDFunctions.scala:65) finished in 0.028 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:56 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 9) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (ShuffledRDD[27] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 2.6 KB, free 23.1 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1475.0 B, free 24.6 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:36693 (size: 1475.0 B, free: 521.6 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 8 missing tasks from ResultStage 9 (ShuffledRDD[27] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 8 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 9.0 (TID 48, localhost, partition 5,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 49, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 50, localhost, partition 1,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 9.0 (TID 51, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 49) 19/03/03 21:21:56 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 50) 19/03/03 21:21:56 INFO executor.Executor: Running task 2.0 in stage 9.0 (TID 51) 19/03/03 21:21:56 INFO executor.Executor: Running task 5.0 in stage 9.0 (TID 48) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 1.0 in stage 9.0 (TID 50). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 5.0 in stage 9.0 (TID 48). 1217 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 9.0 (TID 52, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 3.0 in stage 9.0 (TID 52) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 9.0 (TID 53, localhost, partition 4,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 4.0 in stage 9.0 (TID 53) 19/03/03 21:21:56 INFO executor.Executor: Finished task 0.0 in stage 9.0 (TID 49). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 2.0 in stage 9.0 (TID 51). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 9.0 (TID 54, localhost, partition 6,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 6.0 in stage 9.0 (TID 54) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 9.0 (TID 55, localhost, partition 7,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 7.0 in stage 9.0 (TID 55) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 9.0 (TID 48) in 8 ms on localhost (1/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 50) in 7 ms on localhost (2/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 49) in 8 ms on localhost (3/8) 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 8 blocks 19/03/03 21:21:56 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:56 INFO executor.Executor: Finished task 3.0 in stage 9.0 (TID 52). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 4.0 in stage 9.0 (TID 53). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 9.0 (TID 51) in 9 ms on localhost (4/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 9.0 (TID 52) in 5 ms on localhost (5/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 9.0 (TID 53) in 6 ms on localhost (6/8) 19/03/03 21:21:56 INFO executor.Executor: Finished task 7.0 in stage 9.0 (TID 55). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO executor.Executor: Finished task 6.0 in stage 9.0 (TID 54). 1124 bytes result sent to driver 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 9.0 (TID 54) in 10 ms on localhost (7/8) 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 9.0 (TID 55) in 9 ms on localhost (8/8) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: ResultStage 9 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:50) finished in 0.017 s 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Job 1 finished: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:50, took 0.112314 s 19/03/03 21:21:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:56 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:56 INFO spark.SparkContext: Successfully stopped SparkContext - concordance of identical and non-identical genotypes 19/03/03 21:21:56 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:56 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriver' on port 52911. 19/03/03 21:21:56 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:56 INFO Remoting: Starting remoting 19/03/03 21:21:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:52912] 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 52912. 19/03/03 21:21:56 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:56 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:56 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-73dd1304-6070-4e4a-9ee1-e68a43814862 19/03/03 21:21:56 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:56 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:56 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:56 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39042. 19/03/03 21:21:56 INFO netty.NettyBlockTransferService: Server created on 39042 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:56 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:39042 with 522.0 MB RAM, BlockManagerId(driver, localhost, 39042) 19/03/03 21:21:56 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:56 INFO rdd.ADAMContext: Loading /home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/small.vcf as VCF, and converting to Genotypes. Projection is ignored. 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 33.1 KB, free 33.1 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.7 KB, free 36.8 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:39042 (size: 3.7 KB, free: 522.0 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 0 from newAPIHadoopFile at ADAMContext.scala:586 19/03/03 21:21:56 INFO input.FileInputFormat: Total input paths to process : 1 19/03/03 21:21:56 INFO spark.SparkContext: Starting job: count at GenotypeConcordanceRDDFunctionsSuite.scala:60 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Got job 0 (count at GenotypeConcordanceRDDFunctionsSuite.scala:60) with 1 output partitions 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at GenotypeConcordanceRDDFunctionsSuite.scala:60) 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at filter at GenotypeConcordanceRDDFunctionsSuite.scala:60), which has no missing parents 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.1 KB, free 39.9 KB) 19/03/03 21:21:56 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1853.0 B, free 41.7 KB) 19/03/03 21:21:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:39042 (size: 1853.0 B, free: 522.0 MB) 19/03/03 21:21:56 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:56 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at filter at GenotypeConcordanceRDDFunctionsSuite.scala:60) 19/03/03 21:21:56 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 19/03/03 21:21:56 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2260 bytes) 19/03/03 21:21:56 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:56 INFO rdd.NewHadoopRDD: Input split: file:/home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/small.vcf:0+15651 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2057 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 62 ms on localhost (1/1) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 0 (count at GenotypeConcordanceRDDFunctionsSuite.scala:60) finished in 0.063 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 0 finished: count at GenotypeConcordanceRDDFunctionsSuite.scala:60, took 0.073818 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:62 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 7 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 5 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 7 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 14 (union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 1 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:62) with 2 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:62) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.1 KB, free 45.8 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.3 KB, free 48.1 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:39042 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[7] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:49), which has no missing parents 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,PROCESS_LOCAL, 2249 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.1 KB, free 52.2 KB) 19/03/03 21:21:57 INFO rdd.NewHadoopRDD: Input split: file:/home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/small.vcf:0+15651 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.3 KB, free 54.4 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:39042 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[5] at keyBy at GenotypeConcordanceRDDFunctions.scala:49) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[7] at keyBy at GenotypeConcordanceRDDFunctions.scala:51), which has no missing parents 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, partition 0,PROCESS_LOCAL, 2249 bytes) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.1 KB, free 58.5 KB) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2) 19/03/03 21:21:57 INFO rdd.NewHadoopRDD: Input split: file:/home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/small.vcf:0+15651 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 60.8 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:39042 (size: 2.3 KB, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[7] at keyBy at GenotypeConcordanceRDDFunctions.scala:51) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, partition 0,PROCESS_LOCAL, 2249 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 3) 19/03/03 21:21:57 INFO rdd.NewHadoopRDD: Input split: file:/home/jenkins/workspace/quinine/hadoop.version/1.0.4/label/centos/quinine-core/target/scala-2.10.4/test-classes/small.vcf:0+15651 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 2237 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 26 ms on localhost (1/1) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.026 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 2, ShuffleMapStage 3) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5, ShuffleMapStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 2237 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 3). 2237 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 28 ms on localhost (1/1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (keyBy at GenotypeConcordanceRDDFunctions.scala:49) finished in 0.028 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 3) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5, ShuffleMapStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (keyBy at GenotypeConcordanceRDDFunctions.scala:51) finished in 0.024 s 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 24 ms on localhost (1/1) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5, ShuffleMapStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 4 (UnionRDD[14] at union at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.7 KB, free 65.5 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.4 KB, free 67.9 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:39042 (size: 2.4 KB, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 4 (UnionRDD[14] at union at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 2 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2065 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2173 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 4) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 5) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 4). 1304 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 62 ms on localhost (1/2) 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 5). 1304 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 5) in 70 ms on localhost (2/2) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 4 (union at GenotypeConcordanceRDDFunctions.scala:65) finished in 0.071 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 5) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (ShuffledRDD[15] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 2.6 KB, free 70.4 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1475.0 B, free 71.9 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:39042 (size: 1475.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (ShuffledRDD[15] at combineByKey at GenotypeConcordanceRDDFunctions.scala:65) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 2 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 6, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 5.0 (TID 7, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 6) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 5.0 (TID 7) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 2 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 2 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 6). 1217 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 5.0 (TID 7). 1257 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 5.0 (TID 7) in 6 ms on localhost (1/2) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 6) in 7 ms on localhost (2/2) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 5 (collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:62) finished in 0.007 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 1 finished: collectAsMap at GenotypeConcordanceRDDFunctionsSuite.scala:62, took 0.138642 s 19/03/03 21:21:57 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:57 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:57 INFO spark.SparkContext: Successfully stopped SparkContext - concordance of identical VCFs should be 1.0 19/03/03 21:21:57 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:57 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) GenotypesSummarySuite: 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriver' on port 57042. 19/03/03 21:21:57 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:57 INFO Remoting: Starting remoting 19/03/03 21:21:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:57043] 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 57043. 19/03/03 21:21:57 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:57 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:57 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-1b43d75a-6ecd-4662-9e3c-6a9ae261fe1d 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:57 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:57 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41395. 19/03/03 21:21:57 INFO netty.NettyBlockTransferService: Server created on 41395 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41395 with 522.0 MB RAM, BlockManagerId(driver, localhost, 41395) 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: fold at GenotypesSummary.scala:196 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 0 (fold at GenotypesSummary.scala:196) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (fold at GenotypesSummary.scala:196) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at GenotypesSummary.scala:195), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.1 KB, free 2.1 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1295.0 B, free 3.4 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:41395 (size: 1295.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at GenotypesSummary.scala:195) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2220 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2287 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2287 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2287 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1138 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1135 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1138 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1240 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 20 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 16 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 22 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 19 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 0 (fold at GenotypesSummary.scala:196) finished in 0.024 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 0 finished: fold at GenotypesSummary.scala:196, took 0.028332 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: countByValue at GenotypesSummary.scala:202 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 5 (countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 1 (countByValue at GenotypesSummary.scala:202) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at countByValue at GenotypesSummary.scala:202), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.4 KB, free 6.7 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1878.0 B, free 8.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41395 (size: 1878.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2209 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2276 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2276 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2276 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 13 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 9 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 12 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 11 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (countByValue at GenotypesSummary.scala:202) finished in 0.015 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[6] at countByValue at GenotypesSummary.scala:202), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.6 KB, free 11.2 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1583.0 B, free 12.8 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41395 (size: 1583.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[6] at countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1159 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1159 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1300 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1229 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 6 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 8 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 8 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 7 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 2 (countByValue at GenotypesSummary.scala:202) finished in 0.008 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 1 finished: countByValue at GenotypesSummary.scala:202, took 0.031807 s 19/03/03 21:21:57 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:57 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:57 INFO spark.SparkContext: Successfully stopped SparkContext - simple genotypes summary 19/03/03 21:21:57 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:57 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriver' on port 56835. 19/03/03 21:21:57 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:57 INFO Remoting: Starting remoting 19/03/03 21:21:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:56836] 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 56836. 19/03/03 21:21:57 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:57 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:57 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-4baeb004-ada1-42c4-ba51-d8f0747f3de9 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:57 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:57 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37868. 19/03/03 21:21:57 INFO netty.NettyBlockTransferService: Server created on 37868 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:37868 with 522.0 MB RAM, BlockManagerId(driver, localhost, 37868) 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: fold at GenotypesSummary.scala:196 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 0 (fold at GenotypesSummary.scala:196) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (fold at GenotypesSummary.scala:196) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at GenotypesSummary.scala:195), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.1 KB, free 2.1 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1295.0 B, free 3.4 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:37868 (size: 1295.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at GenotypesSummary.scala:195) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2016 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2016 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2016 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2083 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 915 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 915 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 915 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1065 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 12 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 11 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 13 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 12 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 0 (fold at GenotypesSummary.scala:196) finished in 0.016 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 0 finished: fold at GenotypesSummary.scala:196, took 0.021746 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: countByValue at GenotypesSummary.scala:202 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 5 (countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 1 (countByValue at GenotypesSummary.scala:202) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at countByValue at GenotypesSummary.scala:202), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.4 KB, free 6.7 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1878.0 B, free 8.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:37868 (size: 1878.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,PROCESS_LOCAL, 2005 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,PROCESS_LOCAL, 2072 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 9 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 10 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 11 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 13 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 1 (countByValue at GenotypesSummary.scala:202) finished in 0.013 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[6] at countByValue at GenotypesSummary.scala:202), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.6 KB, free 11.2 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1583.0 B, free 12.8 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:37868 (size: 1583.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 2 (ShuffledRDD[6] at countByValue at GenotypesSummary.scala:202) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 6 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 6 ms on localhost (2/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 6 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 8 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 2 (countByValue at GenotypesSummary.scala:202) finished in 0.008 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 1 finished: countByValue at GenotypesSummary.scala:202, took 0.030459 s 19/03/03 21:21:57 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:57 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:57 INFO spark.SparkContext: Successfully stopped SparkContext - empty genotypes summary 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:57 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:57 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) CoverageStatsSuite: - take simple mean of all equally weighted points - take simple mean of inequally weighted points - take simple median of all equally weighted points - take simple median of inequally weighted points - take median when it lies between points - covered at 0 should return all sites - covered at 2 should return fewer sites - covered at 5 should return no sites GeneratorFilterSuite: - CombinedFilter combines two independent filters 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriver' on port 58022. ReadCoverageSuite: - unmapped reads should emit no coverage observations - generate coverage observations for a mapped read 19/03/03 21:21:57 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:57 INFO Remoting: Starting remoting 19/03/03 21:21:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58023] 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58023. 19/03/03 21:21:57 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:57 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:57 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-5fced4a6-1813-4ce2-ab1f-056ce1c2c753 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore started with capacity 522.0 MB 19/03/03 21:21:57 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:57 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44757. 19/03/03 21:21:57 INFO netty.NettyBlockTransferService: Server created on 44757 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:44757 with 522.0 MB RAM, BlockManagerId(driver, localhost, 44757) 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: collect at ReadCoverageSuite.scala:70 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 1 (flatMap at ReadCoverage.scala:59) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 0 (collect at ReadCoverageSuite.scala:70) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at ReadCoverageSuite.scala:70) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at flatMap at ReadCoverage.scala:59), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.7 KB, free 2.7 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1681.0 B, free 4.4 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44757 (size: 1681.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at flatMap at ReadCoverage.scala:59) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2216 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2216 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2216 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2216 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 33 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 32 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 30 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 31 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (flatMap at ReadCoverage.scala:59) finished in 0.036 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[2] at reduceByKey at ReadCoverage.scala:60), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.8 KB, free 7.2 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1708.0 B, free 8.8 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:44757 (size: 1708.0 B, free: 522.0 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (ShuffledRDD[2] at reduceByKey at ReadCoverage.scala:60) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 4, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 5, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 6, localhost, partition 0,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 7, localhost, partition 2,PROCESS_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 4) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 5) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 6) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 7) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 7). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 5). 1271 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 6). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 7) in 6 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 5) in 7 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 6) in 7 ms on localhost (3/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 4). 1292 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 4) in 10 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 1 (collect at ReadCoverageSuite.scala:70) finished in 0.011 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 0 finished: collect at ReadCoverageSuite.scala:70, took 0.060149 s 19/03/03 21:21:57 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:57 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:57 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:57 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:57 INFO spark.SparkContext: Successfully stopped SparkContext - generate coverage observations for several mapped reads, and an unaligned read 19/03/03 21:21:57 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:57 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:57 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) ContaminationEstimatorSuite: 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriver' on port 58434. 19/03/03 21:21:57 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:57 INFO Remoting: Starting remoting 19/03/03 21:21:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:58435] 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58435. 19/03/03 21:21:57 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:57 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:57 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-3893d4dc-f94c-4655-9111-b0fa6f263be4 19/03/03 21:21:57 INFO storage.MemoryStore: MemoryStore started with capacity 521.3 MB 19/03/03 21:21:57 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:57 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:57 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39910. 19/03/03 21:21:57 INFO netty.NettyBlockTransferService: Server created on 39910 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:39910 with 521.3 MB RAM, BlockManagerId(driver, localhost, 39910) 19/03/03 21:21:57 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: collect at BroadcastRegionJoin.scala:100 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 6 (keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 0 (collect at BroadcastRegionJoin.scala:100) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at BroadcastRegionJoin.scala:100) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[6] at keyBy at BroadcastRegionJoin.scala:98), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.1 KB, free 4.1 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.2 KB, free 6.3 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:39910 (size: 2.2 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[6] at keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2128 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 24 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 27 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 22 ms on localhost (3/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 23 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (keyBy at BroadcastRegionJoin.scala:98) finished in 0.032 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[7] at groupByKey at BroadcastRegionJoin.scala:99), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 11.2 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.5 KB, free 13.7 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:39910 (size: 2.5 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (ShuffledRDD[7] at groupByKey at BroadcastRegionJoin.scala:99) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1124 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1228 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 8 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 9 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 10 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 12 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 1 (collect at BroadcastRegionJoin.scala:100) finished in 0.013 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 0 finished: collect at BroadcastRegionJoin.scala:100, took 0.067057 s 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 216.0 B, free 13.9 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 124.0 B, free 14.0 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:39910 (size: 124.0 B, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 2 from broadcast at BroadcastRegionJoin.scala:109 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: count at ContaminationEstimator.scala:85 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 8 (map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Registering RDD 10 (flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 1 (count at ContaminationEstimator.scala:85) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (count at ContaminationEstimator.scala:85) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2, ShuffleMapStage 3) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 2, ShuffleMapStage 3) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[8] at map at BroadcastRegionJoin.scala:113), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 17.5 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.1 KB, free 19.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:39910 (size: 2.1 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[8] at map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[10] at flatMap at BroadcastRegionJoin.scala:118), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.9 KB, free 23.5 KB) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, localhost, partition 0,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.2 KB, free 25.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:39910 (size: 2.2 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, localhost, partition 1,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[10] at flatMap at BroadcastRegionJoin.scala:118) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, localhost, partition 2,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, localhost, partition 3,PROCESS_LOCAL, 2128 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 8) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 2.0 (TID 10) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 2.0 (TID 11) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 2.0 (TID 9) 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 2.0 (TID 10). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 2.0 (TID 9). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 12, localhost, partition 0,PROCESS_LOCAL, 2279 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 12) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 3.0 (TID 13, localhost, partition 1,PROCESS_LOCAL, 2411 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 3.0 (TID 13) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 9 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 3.0 (TID 14, localhost, partition 2,PROCESS_LOCAL, 2279 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 14 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 13 ms on localhost (3/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 2.0 (TID 11). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 3.0 (TID 15, localhost, partition 3,PROCESS_LOCAL, 2411 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 3.0 (TID 15) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 16 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (map at BroadcastRegionJoin.scala:113) finished in 0.020 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 3) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 12). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 3.0 (TID 13). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 12) in 12 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 3.0 (TID 13) in 12 ms on localhost (2/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 3.0 (TID 15). 1090 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 3.0 (TID 14) in 15 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 3.0 (TID 15) in 12 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (flatMap at BroadcastRegionJoin.scala:118) finished in 0.023 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:57 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[16] at flatMap at ContaminationEstimator.scala:78), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.9 KB, free 29.5 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 31.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:39910 (size: 2.1 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 4 (MapPartitionsRDD[16] at flatMap at ContaminationEstimator.scala:78) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 4.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 4.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 4.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 4.0 (TID 19) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 4.0 (TID 17) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 4.0 (TID 18) 19/03/03 21:21:57 INFO spark.CacheManager: Partition rdd_16_0 not found, computing it 19/03/03 21:21:57 INFO spark.CacheManager: Partition rdd_16_1 not found, computing it 19/03/03 21:21:57 INFO spark.CacheManager: Partition rdd_16_2 not found, computing it 19/03/03 21:21:57 INFO spark.CacheManager: Partition rdd_16_3 not found, computing it 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks 19/03/03 21:21:57 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:57 INFO storage.MemoryStore: Block rdd_16_0 stored as values in memory (estimated size 16.0 B, free 31.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added rdd_16_0 in memory on localhost:39910 (size: 16.0 B, free: 521.2 MB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block rdd_16_1 stored as values in memory (estimated size 16.0 B, free 31.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added rdd_16_1 in memory on localhost:39910 (size: 16.0 B, free: 521.2 MB) 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 1703 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 4.0 (TID 17). 1703 bytes result sent to driver 19/03/03 21:21:57 INFO storage.MemoryStore: Block rdd_16_3 stored as values in memory (estimated size 16.0 B, free 31.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added rdd_16_3 in memory on localhost:39910 (size: 16.0 B, free: 521.2 MB) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 16) in 10 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 4.0 (TID 17) in 11 ms on localhost (2/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 4.0 (TID 19). 1703 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 4.0 (TID 19) in 11 ms on localhost (3/4) 19/03/03 21:21:57 INFO storage.MemoryStore: Block rdd_16_2 stored as values in memory (estimated size 616.0 B, free 32.2 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added rdd_16_2 in memory on localhost:39910 (size: 616.0 B, free: 521.2 MB) 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 4.0 (TID 18). 1703 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 4.0 (TID 18) in 15 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 4 (count at ContaminationEstimator.scala:85) finished in 0.016 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 1 finished: count at ContaminationEstimator.scala:85, took 0.065162 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 158 bytes 19/03/03 21:21:57 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 160 bytes 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 2 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 6) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[17] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 4.4 KB, free 36.6 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.3 KB, free 38.9 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 7 (MapPartitionsRDD[17] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 7.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 7.0 (TID 21) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 20) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 22) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 7.0 (TID 23) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 7.0 (TID 23). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 7.0 (TID 21). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 20). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 22). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 7.0 (TID 23) in 5 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 20) in 6 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 21) in 5 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 22) in 6 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 7 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 2 finished: reduce at ContaminationEstimator.scala:115, took 0.015144 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 3 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 9, ShuffleMapStage 8) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[18] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.4 KB, free 43.2 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 45.5 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 10 (MapPartitionsRDD[18] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 10.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 10.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 10.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 10.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 10.0 (TID 24) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 10.0 (TID 27) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 10.0 (TID 26) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 10.0 (TID 25) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 10.0 (TID 27). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 10.0 (TID 25). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 10.0 (TID 24). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 10.0 (TID 26). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 10.0 (TID 27) in 4 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 10.0 (TID 25) in 4 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 24) in 5 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 10.0 (TID 26) in 5 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 10 (reduce at ContaminationEstimator.scala:115) finished in 0.006 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 3 finished: reduce at ContaminationEstimator.scala:115, took 0.011616 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 4 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 13 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 12, ShuffleMapStage 11) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 13 (MapPartitionsRDD[19] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.4 KB, free 49.9 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 52.1 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 13 (MapPartitionsRDD[19] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 13.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 13.0 (TID 28, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 13.0 (TID 29, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 13.0 (TID 30, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 13.0 (TID 31, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 13.0 (TID 28) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 13.0 (TID 29) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 13.0 (TID 30) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 13.0 (TID 31) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 13.0 (TID 30). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 13.0 (TID 28). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 13.0 (TID 29). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 13.0 (TID 31). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 13.0 (TID 30) in 5 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 13.0 (TID 28) in 5 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 13.0 (TID 29) in 6 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 13.0 (TID 31) in 6 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 13 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 4 finished: reduce at ContaminationEstimator.scala:115, took 0.012070 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 5 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 16 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 15, ShuffleMapStage 14) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[20] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 4.4 KB, free 56.5 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.3 KB, free 58.7 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 16 (MapPartitionsRDD[20] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 32, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 16.0 (TID 33, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 16.0 (TID 34, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 16.0 (TID 35, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 32) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 16.0 (TID 33) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 16.0 (TID 35) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 16.0 (TID 34) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 32). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 16.0 (TID 35). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 16.0 (TID 34). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 16.0 (TID 33). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 32) in 6 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 16.0 (TID 34) in 6 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 16.0 (TID 33) in 6 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 16.0 (TID 35) in 6 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 16 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 5 finished: reduce at ContaminationEstimator.scala:115, took 0.012462 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 6 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 19 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 17, ShuffleMapStage 18) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[21] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.4 KB, free 63.1 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.3 KB, free 65.4 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 19 (MapPartitionsRDD[21] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 36, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 19.0 (TID 37, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 19.0 (TID 38, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 19.0 (TID 39, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 36) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 19.0 (TID 39) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 19.0 (TID 38) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 19.0 (TID 37) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 19.0 (TID 39). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 19.0 (TID 38). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 19.0 (TID 37). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 36). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 19.0 (TID 37) in 6 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 36) in 7 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 19.0 (TID 38) in 6 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 19.0 (TID 39) in 6 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 19 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 6 finished: reduce at ContaminationEstimator.scala:115, took 0.018267 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 7 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 22 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 20, ShuffleMapStage 21) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[22] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.4 KB, free 69.7 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 72.0 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 22 (MapPartitionsRDD[22] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 22.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 22.0 (TID 40, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 22.0 (TID 41, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 22.0 (TID 42, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 22.0 (TID 43, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 22.0 (TID 42) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 22.0 (TID 43) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 22.0 (TID 41) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 22.0 (TID 40) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 22.0 (TID 43). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 22.0 (TID 40). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 22.0 (TID 41). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 22.0 (TID 42). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 22.0 (TID 43) in 4 ms on localhost (1/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 22.0 (TID 40) in 5 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 22.0 (TID 41) in 5 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 22.0 (TID 42) in 5 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 22 (reduce at ContaminationEstimator.scala:115) finished in 0.006 s 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 7 finished: reduce at ContaminationEstimator.scala:115, took 0.011752 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 8 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 25 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 24, ShuffleMapStage 23) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[23] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 4.4 KB, free 76.4 KB) 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.3 KB, free 78.6 KB) 19/03/03 21:21:57 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:57 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 25 (MapPartitionsRDD[23] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 4 tasks 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 44, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 25.0 (TID 45, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 25.0 (TID 46, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 25.0 (TID 47, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:57 INFO executor.Executor: Running task 3.0 in stage 25.0 (TID 47) 19/03/03 21:21:57 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 44) 19/03/03 21:21:57 INFO executor.Executor: Running task 1.0 in stage 25.0 (TID 45) 19/03/03 21:21:57 INFO executor.Executor: Running task 2.0 in stage 25.0 (TID 46) 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:57 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:57 INFO executor.Executor: Finished task 3.0 in stage 25.0 (TID 47). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 25.0 (TID 47) in 3 ms on localhost (1/4) 19/03/03 21:21:57 INFO executor.Executor: Finished task 1.0 in stage 25.0 (TID 45). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 44). 2058 bytes result sent to driver 19/03/03 21:21:57 INFO executor.Executor: Finished task 2.0 in stage 25.0 (TID 46). 2066 bytes result sent to driver 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 25.0 (TID 45) in 8 ms on localhost (2/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 25.0 (TID 46) in 7 ms on localhost (3/4) 19/03/03 21:21:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 44) in 8 ms on localhost (4/4) 19/03/03 21:21:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool 19/03/03 21:21:57 INFO scheduler.DAGScheduler: ResultStage 25 (reduce at ContaminationEstimator.scala:115) finished in 0.008 s 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Job 8 finished: reduce at ContaminationEstimator.scala:115, took 0.018642 s 19/03/03 21:21:57 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Got job 9 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Final stage: ResultStage 28 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 27, ShuffleMapStage 26) 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:57 INFO scheduler.DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[24] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:57 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.4 KB, free 83.0 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.3 KB, free 85.2 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 28 (MapPartitionsRDD[24] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 28.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 28.0 (TID 48, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 28.0 (TID 49, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 28.0 (TID 50, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 28.0 (TID 51, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 28.0 (TID 51) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 28.0 (TID 49) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 28.0 (TID 50) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 28.0 (TID 48) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 28.0 (TID 51). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 28.0 (TID 49). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 28.0 (TID 48). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 28.0 (TID 50). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 28.0 (TID 51) in 5 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 28.0 (TID 49) in 6 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 28.0 (TID 48) in 6 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 28.0 (TID 50) in 6 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 28 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 9 finished: reduce at ContaminationEstimator.scala:115, took 0.012270 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 10 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 31 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 30, ShuffleMapStage 29) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[25] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.4 KB, free 89.6 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.3 KB, free 91.9 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 31 (MapPartitionsRDD[25] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 52, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 53, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 31.0 (TID 54, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 31.0 (TID 55, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 52) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 31.0 (TID 54) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 53) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 31.0 (TID 55) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 31.0 (TID 55). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 31.0 (TID 55) in 3 ms on localhost (1/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 53). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 52). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 31.0 (TID 54). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 53) in 7 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 52) in 8 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 31.0 (TID 54) in 7 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 31 (reduce at ContaminationEstimator.scala:115) finished in 0.008 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 10 finished: reduce at ContaminationEstimator.scala:115, took 0.017534 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 11 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 34 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 33, ShuffleMapStage 32) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 34 (MapPartitionsRDD[26] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 4.4 KB, free 96.2 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.3 KB, free 98.5 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 34 (MapPartitionsRDD[26] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 34.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 34.0 (TID 56, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 34.0 (TID 57, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 34.0 (TID 58, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 34.0 (TID 59, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 34.0 (TID 59) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 34.0 (TID 57) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 34.0 (TID 56) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 34.0 (TID 58) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 34.0 (TID 59). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 34.0 (TID 56). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 34.0 (TID 58). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 34.0 (TID 57). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 34.0 (TID 59) in 6 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 34.0 (TID 56) in 7 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 34.0 (TID 58) in 8 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 34.0 (TID 57) in 8 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 34 (reduce at ContaminationEstimator.scala:115) finished in 0.008 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 11 finished: reduce at ContaminationEstimator.scala:115, took 0.017342 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 12 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 37 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 35, ShuffleMapStage 36) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 37 (MapPartitionsRDD[27] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 4.4 KB, free 102.9 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.3 KB, free 105.1 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 37 (MapPartitionsRDD[27] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 37.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 37.0 (TID 60, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 37.0 (TID 61, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 37.0 (TID 62, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 37.0 (TID 63, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 37.0 (TID 63) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 37.0 (TID 60) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 37.0 (TID 62) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 37.0 (TID 61) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 37.0 (TID 63). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 37.0 (TID 61). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 37.0 (TID 60). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 37.0 (TID 62). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 37.0 (TID 63) in 6 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 37.0 (TID 61) in 9 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 37.0 (TID 60) in 10 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 37.0 (TID 62) in 10 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 37 (reduce at ContaminationEstimator.scala:115) finished in 0.010 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 37.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 12 finished: reduce at ContaminationEstimator.scala:115, took 0.019217 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 13 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 40 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 38, ShuffleMapStage 39) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 40 (MapPartitionsRDD[28] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.4 KB, free 109.5 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.3 KB, free 111.7 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 40 (MapPartitionsRDD[28] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 40.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 40.0 (TID 64, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 40.0 (TID 65, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 40.0 (TID 66, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 40.0 (TID 67, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 40.0 (TID 64) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 40.0 (TID 66) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 40.0 (TID 67) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 40.0 (TID 65) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 40.0 (TID 65). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 40.0 (TID 64). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 40.0 (TID 66). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 40.0 (TID 67). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 40.0 (TID 65) in 6 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 40.0 (TID 64) in 6 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 40.0 (TID 66) in 7 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 40.0 (TID 67) in 8 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 40 (reduce at ContaminationEstimator.scala:115) finished in 0.009 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 40.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 13 finished: reduce at ContaminationEstimator.scala:115, took 0.019326 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 14 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 43 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 42, ShuffleMapStage 41) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 43 (MapPartitionsRDD[29] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 4.4 KB, free 116.1 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.3 KB, free 118.4 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 43 (MapPartitionsRDD[29] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 43.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 43.0 (TID 68, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 43.0 (TID 69, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 43.0 (TID 70, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 43.0 (TID 71, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 43.0 (TID 68) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 43.0 (TID 69) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 43.0 (TID 71) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 43.0 (TID 70) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 43.0 (TID 68). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 43.0 (TID 71). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 43.0 (TID 70). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 43.0 (TID 69). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 43.0 (TID 68) in 9 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 43.0 (TID 71) in 8 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 43.0 (TID 70) in 10 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 43.0 (TID 69) in 11 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 43.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 43 (reduce at ContaminationEstimator.scala:115) finished in 0.011 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 14 finished: reduce at ContaminationEstimator.scala:115, took 0.019756 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 15 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 46 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 45, ShuffleMapStage 44) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 46 (MapPartitionsRDD[30] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.4 KB, free 122.7 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.3 KB, free 125.0 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 46 (MapPartitionsRDD[30] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 46.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 46.0 (TID 72, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 46.0 (TID 73, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 46.0 (TID 74, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 46.0 (TID 75, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 46.0 (TID 73) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 46.0 (TID 75) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 46.0 (TID 72) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 46.0 (TID 74) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 46.0 (TID 74). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 46.0 (TID 72). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 46.0 (TID 73). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 46.0 (TID 75). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 46.0 (TID 74) in 7 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 46.0 (TID 72) in 8 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 46.0 (TID 73) in 8 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 46.0 (TID 75) in 8 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 46.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 46 (reduce at ContaminationEstimator.scala:115) finished in 0.009 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 15 finished: reduce at ContaminationEstimator.scala:115, took 0.019273 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 16 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 49 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 48, ShuffleMapStage 47) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 49 (MapPartitionsRDD[31] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.4 KB, free 129.4 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.3 KB, free 131.6 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 49 (MapPartitionsRDD[31] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 49.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 49.0 (TID 76, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 49.0 (TID 77, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 49.0 (TID 78, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 49.0 (TID 79, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 49.0 (TID 76) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 49.0 (TID 79) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 49.0 (TID 78) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 49.0 (TID 77) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 49.0 (TID 76). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 49.0 (TID 78). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 49.0 (TID 77). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 49.0 (TID 79). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 49.0 (TID 76) in 5 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 49.0 (TID 78) in 5 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 49.0 (TID 77) in 5 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 49.0 (TID 79) in 5 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 49 (reduce at ContaminationEstimator.scala:115) finished in 0.006 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 49.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 16 finished: reduce at ContaminationEstimator.scala:115, took 0.012824 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 17 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 52 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 51, ShuffleMapStage 50) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 52 (MapPartitionsRDD[32] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 4.4 KB, free 136.0 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 2.3 KB, free 138.2 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 52 (MapPartitionsRDD[32] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 52.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 52.0 (TID 80, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 52.0 (TID 81, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 52.0 (TID 82, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 52.0 (TID 83, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 52.0 (TID 80) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 52.0 (TID 83) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 52.0 (TID 81) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 52.0 (TID 82) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 52.0 (TID 83). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 52.0 (TID 80). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 52.0 (TID 81). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 52.0 (TID 82). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 52.0 (TID 83) in 4 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 52.0 (TID 80) in 5 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 52.0 (TID 81) in 5 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 52.0 (TID 82) in 6 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 52 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 52.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 17 finished: reduce at ContaminationEstimator.scala:115, took 0.011628 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 18 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 55 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 53, ShuffleMapStage 54) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 55 (MapPartitionsRDD[33] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 4.4 KB, free 142.6 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 2.3 KB, free 144.9 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 55 (MapPartitionsRDD[33] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 55.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 55.0 (TID 84, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 55.0 (TID 85, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 55.0 (TID 86, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 55.0 (TID 87, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 55.0 (TID 87) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 55.0 (TID 84) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 55.0 (TID 86) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 55.0 (TID 85) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 55.0 (TID 87). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 55.0 (TID 87) in 3 ms on localhost (1/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 55.0 (TID 86). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 55.0 (TID 84). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 55.0 (TID 85). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 55.0 (TID 86) in 7 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 55.0 (TID 84) in 8 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 55.0 (TID 85) in 8 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 55.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 55 (reduce at ContaminationEstimator.scala:115) finished in 0.008 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 18 finished: reduce at ContaminationEstimator.scala:115, took 0.013294 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 19 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 58 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 56, ShuffleMapStage 57) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 58 (MapPartitionsRDD[34] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.4 KB, free 149.2 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.3 KB, free 151.5 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 58 (MapPartitionsRDD[34] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 58.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 58.0 (TID 88, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 58.0 (TID 89, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 58.0 (TID 90, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 58.0 (TID 91, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 58.0 (TID 91) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 58.0 (TID 88) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 58.0 (TID 89) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 58.0 (TID 90) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 58.0 (TID 91). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 58.0 (TID 90). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 58.0 (TID 91) in 4 ms on localhost (1/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 58.0 (TID 88). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 58.0 (TID 89). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 58.0 (TID 90) in 5 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 58.0 (TID 88) in 5 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 58.0 (TID 89) in 6 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 58 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 58.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 19 finished: reduce at ContaminationEstimator.scala:115, took 0.012151 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 20 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 61 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 60, ShuffleMapStage 59) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 61 (MapPartitionsRDD[35] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_24 stored as values in memory (estimated size 4.4 KB, free 155.9 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 2.3 KB, free 158.1 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_24_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 24 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 61 (MapPartitionsRDD[35] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 61.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 61.0 (TID 92, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 61.0 (TID 93, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 61.0 (TID 94, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 61.0 (TID 95, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 61.0 (TID 95) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 61.0 (TID 93) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 61.0 (TID 92) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 61.0 (TID 94) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_23_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 61.0 (TID 93). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 214 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 61.0 (TID 95). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 61.0 (TID 94). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 61.0 (TID 92). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 213 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 61.0 (TID 92) in 15 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 61.0 (TID 93) in 15 ms on localhost (2/4) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 212 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 61.0 (TID 95) in 5 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 61.0 (TID 94) in 5 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 61 (reduce at ContaminationEstimator.scala:115) finished in 0.015 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 61.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:39910 in memory (size: 2.1 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 20 finished: reduce at ContaminationEstimator.scala:115, took 0.019985 s 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 211 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:39910 in memory (size: 2.2 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 210 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:39910 in memory (size: 2.1 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 209 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:39910 in memory (size: 2.5 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 208 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on localhost:39910 in memory (size: 2.2 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 207 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned shuffle 0 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_12_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 218 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 21 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 64 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 63, ShuffleMapStage 62) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 64 (MapPartitionsRDD[36] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_11_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 217 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_25 stored as values in memory (estimated size 4.4 KB, free 91.5 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 216 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 215 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 2.3 KB, free 80.5 KB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 229 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_25_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 64 (MapPartitionsRDD[36] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 64.0 with 4 tasks 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_22_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 64.0 (TID 96, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 64.0 (TID 97, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 228 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 64.0 (TID 98, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 64.0 (TID 99, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 64.0 (TID 97) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 64.0 (TID 98) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 64.0 (TID 96) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 64.0 (TID 99) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_21_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 227 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_20_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 226 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 64.0 (TID 97). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 64.0 (TID 98). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_19_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 225 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 64.0 (TID 97) in 5 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 64.0 (TID 98) in 5 ms on localhost (2/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 64.0 (TID 96). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 64.0 (TID 99). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_18_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 224 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 64.0 (TID 96) in 7 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 64.0 (TID 99) in 6 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 64 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 64.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 21 finished: reduce at ContaminationEstimator.scala:115, took 0.013385 s 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_17_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 223 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_16_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 222 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_15_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 221 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_14_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 220 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Removed broadcast_13_piece0 on localhost:39910 in memory (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 22 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 67 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 66, ShuffleMapStage 65) 19/03/03 21:21:58 INFO spark.ContextCleaner: Cleaned accumulator 219 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 67 (MapPartitionsRDD[37] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_26 stored as values in memory (estimated size 4.4 KB, free 18.6 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 2.3 KB, free 20.9 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_26_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 67 (MapPartitionsRDD[37] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 67.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 67.0 (TID 100, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 67.0 (TID 101, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 67.0 (TID 102, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 67.0 (TID 103, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 67.0 (TID 100) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 67.0 (TID 101) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 67.0 (TID 102) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 67.0 (TID 103) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 67.0 (TID 100). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 67.0 (TID 101). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 67.0 (TID 102). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 67.0 (TID 103). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 67.0 (TID 100) in 9 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 67.0 (TID 102) in 9 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 67.0 (TID 101) in 10 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 67.0 (TID 103) in 10 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 67 (reduce at ContaminationEstimator.scala:115) finished in 0.012 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 67.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 22 finished: reduce at ContaminationEstimator.scala:115, took 0.020573 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 23 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 70 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 68, ShuffleMapStage 69) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 70 (MapPartitionsRDD[38] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_27 stored as values in memory (estimated size 4.4 KB, free 25.2 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 2.3 KB, free 27.5 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_27_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 27 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 70 (MapPartitionsRDD[38] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 70.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 70.0 (TID 104, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 70.0 (TID 105, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 70.0 (TID 106, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 70.0 (TID 107, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 70.0 (TID 104) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 70.0 (TID 106) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 70.0 (TID 107) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 70.0 (TID 105) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 70.0 (TID 104). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 70.0 (TID 105). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 70.0 (TID 106). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 70.0 (TID 107). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 70.0 (TID 104) in 8 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 70.0 (TID 106) in 7 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 70.0 (TID 107) in 6 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 70.0 (TID 105) in 7 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 70 (reduce at ContaminationEstimator.scala:115) finished in 0.008 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 70.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 23 finished: reduce at ContaminationEstimator.scala:115, took 0.014891 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 24 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 73 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 71, ShuffleMapStage 72) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 73 (MapPartitionsRDD[39] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_28 stored as values in memory (estimated size 4.4 KB, free 31.8 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 2.3 KB, free 34.1 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_28_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 28 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 73 (MapPartitionsRDD[39] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 73.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 73.0 (TID 108, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 73.0 (TID 109, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 73.0 (TID 110, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 73.0 (TID 111, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 73.0 (TID 111) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 73.0 (TID 108) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 73.0 (TID 109) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 73.0 (TID 110) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 73.0 (TID 111). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 73.0 (TID 110). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 73.0 (TID 108). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 73.0 (TID 109). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 73.0 (TID 111) in 7 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 73.0 (TID 110) in 7 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 73.0 (TID 108) in 10 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 73.0 (TID 109) in 9 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 73.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 73 (reduce at ContaminationEstimator.scala:115) finished in 0.010 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 24 finished: reduce at ContaminationEstimator.scala:115, took 0.017928 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 25 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 76 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 74, ShuffleMapStage 75) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 76 (MapPartitionsRDD[40] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_29 stored as values in memory (estimated size 4.4 KB, free 38.5 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 2.3 KB, free 40.7 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_29_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 76 (MapPartitionsRDD[40] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 76.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 76.0 (TID 112, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 76.0 (TID 113, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 76.0 (TID 114, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 76.0 (TID 115, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 76.0 (TID 112) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 76.0 (TID 115) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 76.0 (TID 114) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 76.0 (TID 113) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 76.0 (TID 112). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 76.0 (TID 113). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 76.0 (TID 115). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 76.0 (TID 113) in 7 ms on localhost (1/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 76.0 (TID 114). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 76.0 (TID 112) in 8 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 76.0 (TID 115) in 8 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 76.0 (TID 114) in 9 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 76 (reduce at ContaminationEstimator.scala:115) finished in 0.010 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 76.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 25 finished: reduce at ContaminationEstimator.scala:115, took 0.019249 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 26 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 79 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 78, ShuffleMapStage 77) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 79 (MapPartitionsRDD[41] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_30 stored as values in memory (estimated size 4.4 KB, free 45.1 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 2.3 KB, free 47.3 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_30_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 30 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 79 (MapPartitionsRDD[41] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 79.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 79.0 (TID 116, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 79.0 (TID 117, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 79.0 (TID 118, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 79.0 (TID 119, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 79.0 (TID 116) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 79.0 (TID 118) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 79.0 (TID 119) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 79.0 (TID 117) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 79.0 (TID 117). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 79.0 (TID 116). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 79.0 (TID 118). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 79.0 (TID 119). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 79.0 (TID 117) in 4 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 79.0 (TID 116) in 5 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 79.0 (TID 118) in 4 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 79.0 (TID 119) in 5 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 79 (reduce at ContaminationEstimator.scala:115) finished in 0.006 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 79.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 26 finished: reduce at ContaminationEstimator.scala:115, took 0.016314 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 27 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 82 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 81, ShuffleMapStage 80) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 82 (MapPartitionsRDD[42] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_31 stored as values in memory (estimated size 4.4 KB, free 51.7 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 2.3 KB, free 54.0 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_31_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 31 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 82 (MapPartitionsRDD[42] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 82.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 82.0 (TID 120, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 82.0 (TID 121, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 82.0 (TID 122, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 82.0 (TID 123, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 82.0 (TID 122) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 82.0 (TID 121) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 82.0 (TID 120) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 82.0 (TID 123) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 82.0 (TID 122). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 82.0 (TID 123). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 82.0 (TID 122) in 4 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 82.0 (TID 123) in 4 ms on localhost (2/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 82.0 (TID 120). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 82.0 (TID 121). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 82.0 (TID 120) in 7 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 82.0 (TID 121) in 7 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 82.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 82 (reduce at ContaminationEstimator.scala:115) finished in 0.007 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 27 finished: reduce at ContaminationEstimator.scala:115, took 0.012480 s 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: reduce at ContaminationEstimator.scala:115 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 28 (reduce at ContaminationEstimator.scala:115) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 85 (reduce at ContaminationEstimator.scala:115) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 84, ShuffleMapStage 83) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 85 (MapPartitionsRDD[43] at map at ContaminationEstimator.scala:114), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_32 stored as values in memory (estimated size 4.4 KB, free 58.3 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 2.3 KB, free 60.6 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_32_piece0 in memory on localhost:39910 (size: 2.3 KB, free: 521.2 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 32 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 85 (MapPartitionsRDD[43] at map at ContaminationEstimator.scala:114) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 85.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 85.0 (TID 124, localhost, partition 0,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 85.0 (TID 125, localhost, partition 1,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 85.0 (TID 126, localhost, partition 2,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 85.0 (TID 127, localhost, partition 3,PROCESS_LOCAL, 1967 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 85.0 (TID 124) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 85.0 (TID 126) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 85.0 (TID 125) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 85.0 (TID 127) 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_2 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_0 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_3 locally 19/03/03 21:21:58 INFO storage.BlockManager: Found block rdd_16_1 locally 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 85.0 (TID 126). 2066 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 85.0 (TID 127). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 85.0 (TID 124). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 85.0 (TID 125). 2058 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 85.0 (TID 126) in 3 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 85.0 (TID 127) in 4 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 85.0 (TID 124) in 5 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 85.0 (TID 125) in 4 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 85.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 85 (reduce at ContaminationEstimator.scala:115) finished in 0.005 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 28 finished: reduce at ContaminationEstimator.scala:115, took 0.012475 s 19/03/03 21:21:58 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/03/03 21:21:58 INFO storage.MemoryStore: MemoryStore cleared 19/03/03 21:21:58 INFO storage.BlockManager: BlockManager stopped 19/03/03 21:21:58 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 19/03/03 21:21:58 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/03/03 21:21:58 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 19/03/03 21:21:58 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 19/03/03 21:21:58 INFO spark.SparkContext: Successfully stopped SparkContext - estimate contamination from a read dataset with 10% contamination reads 19/03/03 21:21:58 INFO spark.SparkContext: Running Spark version 1.6.1 19/03/03 21:21:58 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 19/03/03 21:21:58 INFO spark.SecurityManager: Changing view acls to: jenkins 19/03/03 21:21:58 INFO spark.SecurityManager: Changing modify acls to: jenkins 19/03/03 21:21:58 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); users with modify permissions: Set(jenkins) 19/03/03 21:21:58 INFO util.Utils: Successfully started service 'sparkDriver' on port 55844. 19/03/03 21:21:58 INFO slf4j.Slf4jLogger: Slf4jLogger started 19/03/03 21:21:58 INFO Remoting: Starting remoting 19/03/03 21:21:58 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@localhost:55845] 19/03/03 21:21:58 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 55845. 19/03/03 21:21:58 INFO spark.SparkEnv: Registering MapOutputTracker 19/03/03 21:21:58 INFO spark.SparkEnv: Registering BlockManagerMaster 19/03/03 21:21:58 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-9b473dbd-d6ee-44f2-abb9-8a02f48b6439 19/03/03 21:21:58 INFO storage.MemoryStore: MemoryStore started with capacity 521.6 MB 19/03/03 21:21:58 INFO spark.SparkEnv: Registering OutputCommitCoordinator 19/03/03 21:21:58 INFO executor.Executor: Starting executor ID driver on host localhost 19/03/03 21:21:58 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41032. 19/03/03 21:21:58 INFO netty.NettyBlockTransferService: Server created on 41032 19/03/03 21:21:58 INFO storage.BlockManagerMaster: Trying to register BlockManager 19/03/03 21:21:58 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41032 with 521.6 MB RAM, BlockManagerId(driver, localhost, 41032) 19/03/03 21:21:58 INFO storage.BlockManagerMaster: Registered BlockManager 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: collect at BroadcastRegionJoin.scala:100 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Registering RDD 6 (keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Got job 0 (collect at BroadcastRegionJoin.scala:100) with 4 output partitions 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at BroadcastRegionJoin.scala:100) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[6] at keyBy at BroadcastRegionJoin.scala:98), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.1 KB, free 4.1 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.2 KB, free 6.3 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:41032 (size: 2.2 KB, free: 521.6 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[6] at keyBy at BroadcastRegionJoin.scala:98) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2082 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2128 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1090 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1090 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 14 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 21 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 18 ms on localhost (3/4) 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1090 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 20 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (keyBy at BroadcastRegionJoin.scala:98) finished in 0.028 s 19/03/03 21:21:58 INFO scheduler.DAGScheduler: looking for newly runnable stages 19/03/03 21:21:58 INFO scheduler.DAGScheduler: running: Set() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: failed: Set() 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[7] at groupByKey at BroadcastRegionJoin.scala:99), which has no missing parents 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.9 KB, free 11.2 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.5 KB, free 13.7 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41032 (size: 2.5 KB, free: 521.6 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (ShuffledRDD[7] at groupByKey at BroadcastRegionJoin.scala:99) 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 4 tasks 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes) 19/03/03 21:21:58 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 5) 19/03/03 21:21:58 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 6) 19/03/03 21:21:58 INFO executor.Executor: Running task 3.0 in stage 1.0 (TID 7) 19/03/03 21:21:58 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 4) 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 19/03/03 21:21:58 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 19/03/03 21:21:58 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 5). 1124 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 3.0 in stage 1.0 (TID 7). 1124 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 4). 1124 bytes result sent to driver 19/03/03 21:21:58 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 6). 1228 bytes result sent to driver 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 8 ms on localhost (1/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 9 ms on localhost (2/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 10 ms on localhost (3/4) 19/03/03 21:21:58 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 13 ms on localhost (4/4) 19/03/03 21:21:58 INFO scheduler.DAGScheduler: ResultStage 1 (collect at BroadcastRegionJoin.scala:100) finished in 0.015 s 19/03/03 21:21:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Job 0 finished: collect at BroadcastRegionJoin.scala:100, took 0.057265 s 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 216.0 B, free 13.9 KB) 19/03/03 21:21:58 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 124.0 B, free 14.0 KB) 19/03/03 21:21:58 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41032 (size: 124.0 B, free: 521.6 MB) 19/03/03 21:21:58 INFO spark.SparkContext: Created broadcast 2 from broadcast at BroadcastRegionJoin.scala:109 19/03/03 21:21:58 INFO spark.SparkContext: Starting job: count at ContaminationEstimator.scala:85 19/03/03 21:21:58 INFO scheduler.DAGScheduler: Registering RDD 8 (map at BroadcastRegionJoin.scala:113) 19/03/03 21:21:58 INFO scheduler.DAGSch