FailedConsole Output

Skipping 12,644 KB.. Full Log
xy+/JNU2Yj/Tagmf50wMjD4M7CWJ6SCJEgYhn6zEskT9nMS8dP3gkqLMvHTriiIGKaihyfl5xfk5qXrOEBpkDgMEMDIxMFQUlDCw2+gXFyTm2QEAI9P8iI4AAAA=
[info] - udf/udf-join-empty-relation.sql - Scalar Pandas UDF (13 seconds, 973 milliseconds)
[info] - udf/pgSQL/udf-select_having.sql - Scala UDF (56 seconds, 70 milliseconds)
[info] - udf/pgSQL/udf-select_having.sql - Regular Python UDF (4 minutes, 59 seconds)
[info] - udf/pgSQL/udf-select_having.sql - Scalar Pandas UDF (5 minutes, 32 seconds)
[info] - udf/pgSQL/udf-aggregates_part2.sql - Scala UDF (11 seconds, 63 milliseconds)
[info] - udf/pgSQL/udf-aggregates_part2.sql - Regular Python UDF (18 seconds, 116 milliseconds)
[info] - udf/pgSQL/udf-aggregates_part2.sql - Scalar Pandas UDF (16 seconds, 486 milliseconds)
[info] - udf/pgSQL/udf-aggregates_part1.sql - Scala UDF (22 seconds, 662 milliseconds)
[info] - udf/pgSQL/udf-aggregates_part1.sql - Regular Python UDF (2 minutes)
[info] - udf/pgSQL/udf-aggregates_part1.sql - Scalar Pandas UDF (2 minutes, 6 seconds)
[info] - udf/pgSQL/udf-case.sql - Scala UDF (12 seconds, 955 milliseconds)
23:52:23.848 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#686076, cast(i#686093 as double), cast(j#686094 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:24.338 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#686076, cast(i#686093 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:28.638 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#686355, cast(i#686372 as double), cast(j#686373 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:29.043 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#686355, cast(i#686372 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:35.038 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#686634, cast(i#686651 as double), cast(j#686652 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:35.608 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#686634, cast(i#686651 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
[info] - udf/pgSQL/udf-case.sql - Regular Python UDF (17 seconds, 46 milliseconds)
23:52:39.380 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#686913, cast(i#686930 as double), cast(j#686931 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:39.849 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#686913, cast(i#686930 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:43.895 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#687192, cast(i#687209 as double), cast(j#687210 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:44.384 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#687192, cast(i#687209 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:48.238 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:(cast(udf(cast(coalesce(f#687471, cast(i#687488 as double), cast(j#687489 as double)) as string)) as double) = 2.0) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
23:52:48.677 WARN org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition: The join condition:cast(udf(cast((coalesce(f#687471, cast(i#687488 as double)) = 2.0) as string)) as boolean) of the join plan contains PythonUDF only, it will be moved out and the join plan will be turned to cross join.
[info] - udf/pgSQL/udf-case.sql - Scalar Pandas UDF (13 seconds, 97 milliseconds)
[info] - ansi/interval.sql (3 seconds, 989 milliseconds)
23:52:53.009 WARN org.apache.spark.sql.SQLQueryTestSuite: 
=== Metrics of Analyzer/Optimizer Rules ===
Total number of runs: 15906872
Total time: 382.904632579 seconds

Rule                                                                                               Effective Time / Total Time                     Effective Runs / Total Runs                    

org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries                               37771482852 / 39957884804                       975 / 24864                                    
org.apache.spark.sql.catalyst.optimizer.ColumnPruning                                              6283125483 / 19934330268                        10926 / 116283                                 
org.apache.spark.sql.execution.datasources.FindDataSourceTable                                     14848231318 / 15111082856                       3702 / 91572                                   
org.apache.spark.sql.catalyst.expressions.codegen.package$ExpressionCanonicalizer$CleanExpressions 456875744 / 12091042017                         189527 / 6705836                               
org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators                                        573473987 / 6892206096                          3627 / 116259                                  
org.apache.spark.sql.catalyst.optimizer.CollapseProject                                            1784570363 / 6162945463                         11154 / 91443                                  
org.apache.spark.sql.catalyst.optimizer.CombineUnions                                              19251892 / 5999769685                           45 / 91467                                     
org.apache.spark.sql.catalyst.optimizer.SimplifyCasts                                              495304355 / 5916558161                          2415 / 66579                                   
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery                                    4743262714 / 5715935699                         1107 / 91644                                   
org.apache.spark.sql.catalyst.optimizer.ConstantFolding                                            1561715913 / 5641030049                         8079 / 66579                                   
org.apache.spark.sql.catalyst.optimizer.LikeSimplification                                         0 / 5555471540                                  0 / 66579                                      
org.apache.spark.sql.catalyst.optimizer.RemoveDispensableExpressions                               0 / 5539976329                                  0 / 66579                                      
org.apache.spark.sql.catalyst.optimizer.NullPropagation                                            120881249 / 5400074960                          741 / 66603                                    
org.apache.spark.sql.catalyst.optimizer.OptimizeIn                                                 1738552 / 5325822248                            12 / 66603                                     
org.apache.spark.sql.catalyst.optimizer.PruneFilters                                               137583230 / 5208516477                          831 / 66579                                    
org.apache.spark.sql.catalyst.optimizer.PushDownPredicates                                         1045586011 / 5204245803                         3288 / 66603                                   
org.apache.spark.sql.catalyst.optimizer.BooleanSimplification                                      27355590 / 5168832447                           87 / 66579                                     
org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator                                 0 / 4873809844                                  0 / 66579                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences                                  2518883680 / 4866374193                         13245 / 91863                                  
org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison                                   7776422 / 4731778456                            33 / 66579                                     
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals                                       43590998 / 4691654482                           549 / 66579                                    
org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions                          0 / 4654610694                                  0 / 66579                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions                          2074905950 / 4604036658                         669 / 91635                                    
org.apache.spark.sql.catalyst.optimizer.EliminateResolvedHint                                      0 / 4538615798                                  0 / 24864                                      
org.apache.spark.sql.catalyst.optimizer.PushLeftSemiLeftAntiThroughJoin                            3491452 / 4529357602                            9 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicate                            1135286 / 4466789464                            9 / 66579                                      
org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps                                    1428926 / 4431194510                            9 / 66579                                      
org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin                                   297981240 / 4180792045                          639 / 66603                                    
org.apache.spark.sql.catalyst.optimizer.CollapseRepartition                                        0 / 4150730172                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.EliminateOuterJoin                                         161997263 / 4123579410                          195 / 66603                                    
org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery                            41104848 / 4121391433                           63 / 66579                                     
org.apache.spark.sql.catalyst.optimizer.ReorderJoin                                                21808710 / 4104974067                           24 / 66603                                     
org.apache.spark.sql.catalyst.optimizer.LimitPushDown                                              0 / 4086722768                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.ConstantPropagation                                        4413130 / 3977481513                            12 / 66603                                     
org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion                                 20539425 / 3974304391                           33 / 66603                                     
org.apache.spark.sql.catalyst.optimizer.EliminateSerialization                                     0 / 3903097721                                  0 / 66579                                      
org.apache.spark.sql.catalyst.optimizer.CombineLimits                                              0 / 3896582904                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.CollapseWindow                                             0 / 3879171524                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.TransposeWindow                                            0 / 3816488340                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.CombineFilters                                             0 / 3758780074                                  0 / 66603                                      
org.apache.spark.sql.catalyst.optimizer.EliminateSorts                                             2444525 / 3755385152                            18 / 66579                                     
org.apache.spark.sql.catalyst.optimizer.FoldablePropagation                                        213069446 / 2989417044                          393 / 66603                                    
org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation                                     98939449 / 2926221892                           771 / 50475                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveDeserializer                                1821878981 / 2890190168                         1908 / 91863                                   
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints                                846601702 / 2850788534                          1482 / 24840                                   
org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates                                 701347676 / 2528086030                          606 / 24864                                    
org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases                                    1980378546 / 2456167788                         12903 / 24864                                  
org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime                                         3783313 / 2350152248                            9 / 24864                                      
org.apache.spark.sql.catalyst.optimizer.ReplaceExpressions                                         37854965 / 2260156771                           159 / 24864                                    
org.apache.spark.sql.execution.python.ExtractPythonUDFs                                            863790911 / 2247277222                          1440 / 24840                                   
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions                                   1217220625 / 2201199195                         11790 / 91644                                  
org.apache.spark.sql.catalyst.optimizer.GetCurrentDatabase                                         0 / 2134909089                                  0 / 24864                                      
org.apache.spark.sql.catalyst.optimizer.ReassignLambdaVariableID                                   0 / 2122286777                                  0 / 24840                                      
org.apache.spark.sql.catalyst.analysis.TypeCoercion$ImplicitTypeCasts                              426023739 / 2096570450                          2001 / 91572                                   
org.apache.spark.sql.catalyst.analysis.DecimalPrecision                                            404473140 / 2074189674                          3660 / 91572                                   
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithFilter                                    45564256 / 2073335499                           54 / 25743                                     
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAliases                                     46999468 / 2030125826                           315 / 66579                                    
org.apache.spark.sql.execution.datasources.PruneFileSourcePartitions                               158961797 / 2016885351                          6 / 24840                                      
org.apache.spark.sql.catalyst.optimizer.ReplaceIntersectWithSemiJoin                               28940398 / 1913010636                           87 / 25743                                     
org.apache.spark.sql.catalyst.optimizer.CombineTypedFilters                                        0 / 1905894215                                  0 / 24840                                      
org.apache.spark.sql.catalyst.analysis.ResolveTimeZone                                             1321001532 / 1900451891                         25332 / 91572                                  
org.apache.spark.sql.catalyst.optimizer.EliminateMapObjects                                        0 / 1898406779                                  0 / 24840                                      
org.apache.spark.sql.catalyst.optimizer.RewriteExceptAll                                           94499174 / 1892360073                           204 / 25743                                    
org.apache.spark.sql.catalyst.optimizer.RewriteIntersectAll                                        124483888 / 1840487268                          156 / 25743                                    
org.apache.spark.sql.catalyst.optimizer.ReplaceDistinctWithAggregate                               165766367 / 1811831939                          639 / 25743                                    
org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithAntiJoin                                  85279836 / 1794996789                           180 / 25743                                    
org.apache.spark.sql.catalyst.optimizer.RewriteDistinctAggregates                                  32441069 / 1685999527                           15 / 24864                                     
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractGenerator                                   5532449 / 1674599723                            24 / 91722                                     
org.apache.spark.sql.catalyst.optimizer.RemoveLiteralFromGroupExpressions                          3111784 / 1637303527                            24 / 24981                                     
org.apache.spark.sql.catalyst.optimizer.RemoveRepetitionFromGroupExpressions                       15231099 / 1625144650                           93 / 24981                                     
org.apache.spark.sql.catalyst.analysis.EliminateView                                               0 / 1618282378                                  0 / 24864                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAliases                                     934234538 / 1593226911                          11694 / 91644                                  
org.apache.spark.sql.catalyst.optimizer.OptimizeLimitZero                                          0 / 1581325954                                  0 / 24864                                      
org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery                                   347958021 / 1539099962                          912 / 24840                                    
org.apache.spark.sql.catalyst.optimizer.ReplaceDeduplicateWithAggregate                            0 / 1534149653                                  0 / 24864                                      
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PromoteStrings                                 229653212 / 1532487664                          2244 / 91572                                   
org.apache.spark.sql.catalyst.optimizer.PushPredicateThroughNonJoin                                11037123 / 1518098162                           30 / 24840                                     
org.apache.spark.sql.catalyst.optimizer.DecimalAggregates                                          0 / 1476668099                                  0 / 24840                                      
org.apache.spark.sql.execution.python.ExtractPythonUDFFromAggregate                                160692113 / 1452336169                          624 / 24840                                    
org.apache.spark.sql.catalyst.analysis.TypeCoercion$InConversion                                   68237339 / 1445061295                           399 / 91572                                    
org.apache.spark.sql.catalyst.analysis.UpdateAttributeNullability                                  90472085 / 1442796849                           567 / 44328                                    
org.apache.spark.sql.catalyst.analysis.TypeCoercion$FunctionArgumentConversion                     278349134 / 1437845478                          1011 / 91572                                   
org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingNumbers                                   14714204 / 1328773382                           18 / 24840                                     
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantSorts                                       0 / 1302160574                                  0 / 24840                                      
org.apache.spark.sql.catalyst.optimizer.PullOutPythonUDFInJoinCondition                            6716242 / 1273758295                            12 / 24840                                     
org.apache.spark.sql.catalyst.optimizer.ObjectSerializerPruning                                    0 / 1247503513                                  0 / 24840                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations                                   994730777 / 1216447958                          18204 / 92001                                  
org.apache.spark.sql.catalyst.optimizer.CheckCartesianProducts                                     0 / 1179782415                                  0 / 24840                                      
org.apache.spark.sql.catalyst.analysis.TypeCoercion$BooleanEquality                                7402749 / 1122944074                            204 / 91572                                    
org.apache.spark.sql.catalyst.analysis.TypeCoercion$DateTimeOperations                             10000566 / 1090704617                           84 / 91572                                     
org.apache.spark.sql.catalyst.analysis.TypeCoercion$IfCoercion                                     29306989 / 1084846879                           195 / 91572                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRandomSeed                                  413030 / 1059675550                             3 / 91572                                      
org.apache.spark.sql.catalyst.analysis.TimeWindowing                                               0 / 1039833249                                  0 / 91635                                      
org.apache.spark.sql.catalyst.analysis.ResolveCreateNamedStruct                                    19255609 / 1039289756                           174 / 91863                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions                                    0 / 1036903345                                  0 / 44772                                      
org.apache.spark.sql.catalyst.analysis.TypeCoercion$Division                                       37490867 / 1019271625                           306 / 91572                                    
org.apache.spark.sql.catalyst.analysis.TypeCoercion$ConcatCoercion                                 9445826 / 1001482191                            48 / 91572                                     
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveMissingReferences                           72060223 / 982988538                            183 / 91722                                    
org.apache.spark.sql.catalyst.analysis.TypeCoercion$EltCoercion                                    3630489 / 947816703                             9 / 91572                                      
org.apache.spark.sql.catalyst.analysis.ResolveHigherOrderFunctions                                 15577368 / 917125377                            120 / 91572                                    
org.apache.spark.sql.catalyst.analysis.ResolveInlineTables                                         550247418 / 852655314                           3276 / 91575                                   
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowOrder                                 3377272 / 847951582                             3 / 91635                                      
org.apache.spark.sql.catalyst.analysis.TypeCoercion$CaseWhenCoercion                               33374564 / 820106271                            186 / 91572                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$GlobalAggregates                                   24214396 / 785569564                            1065 / 91635                                   
org.apache.spark.sql.catalyst.analysis.TypeCoercion$MapZipWithCoercion                             12365948 / 780901167                            39 / 91572                                     
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveWindowFrame                                 10726404 / 775055433                            69 / 91635                                     
org.apache.spark.sql.catalyst.analysis.ResolveLambdaVariables                                      47017826 / 770776934                            117 / 91572                                    
org.apache.spark.sql.catalyst.analysis.CleanupAliases                                              345719149 / 764817600                           15102 / 59430                                  
org.apache.spark.sql.catalyst.analysis.TypeCoercion$WindowFrameCoercion                            4118157 / 763607308                             30 / 91572                                     
org.apache.spark.sql.catalyst.analysis.TypeCoercion$StackCoercion                                  0 / 753502747                                   0 / 91572                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractWindowExpressions                           75046543 / 733869299                            129 / 91635                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast                                      136720408 / 724543433                           1266 / 91863                                   
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGroupingAnalytics                           149159107 / 613996594                           348 / 91815                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance                                 0 / 606190609                                   0 / 91863                                      
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveJoinStrategyHints                       0 / 545171377                                   0 / 44787                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$PullOutNondeterministic                            4743405 / 534023771                             3 / 44328                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot                                       172807161 / 477920682                           234 / 91743                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$HandleNullInputsForUDF                             2157963 / 469839000                             6 / 44328                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveTables                                      0 / 428140163                                   0 / 92001                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOrdinalInOrderByAndGroupBy                  5129642 / 427333726                             96 / 91722                                     
org.apache.spark.sql.catalyst.analysis.TypeCoercion$WidenSetOperationTypes                         60343167 / 409954533                            249 / 91572                                    
org.apache.spark.sql.execution.datasources.DataSourceResolution                                    74987851 / 399503819                            621 / 91572                                    
org.apache.spark.sql.catalyst.analysis.CTESubstitution                                             43962381 / 385600641                            204 / 45102                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggAliasInGroupBy                           10025289 / 354127251                            51 / 91722                                     
org.apache.spark.sql.execution.datasources.PreprocessTableInsertion                                207446774 / 344863727                           1980 / 44328                                   
org.apache.spark.sql.catalyst.analysis.ResolveTableValuedFunctions                                 45177799 / 329003988                            207 / 92001                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAlterTable                                  0 / 298660721                                   0 / 92001                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGenerate                                    3085182 / 297532069                             12 / 91722                                     
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNaturalAndUsingJoin                         34737752 / 295330190                            171 / 91635                                    
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubqueryColumnAliases                       13201005 / 288685990                            246 / 91638                                    
org.apache.spark.sql.catalyst.optimizer.CombineConcats                                             3252615 / 274667039                             21 / 66579                                     
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveOutputRelation                              0 / 268795674                                   0 / 91635                                      
org.apache.spark.sql.execution.datasources.ResolveSQLOnFile                                        0 / 268404682                                   0 / 91572                                      
org.apache.spark.sql.catalyst.analysis.ResolveHints$ResolveCoalesceHints                           1877779 / 259734789                             3 / 44787                                      
org.apache.spark.sql.execution.datasources.DataSourceAnalysis                                      106854609 / 249512259                           2226 / 44328                                   
org.apache.spark.sql.execution.datasources.FallBackFileSourceV2                                    0 / 248527130                                   0 / 91572                                      
org.apache.spark.sql.execution.datasources.PreprocessTableCreation                                 865329 / 236226918                              3 / 44331                                      
org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder                                       0 / 235428469                                   0 / 24840                                      
org.apache.spark.sql.catalyst.analysis.UpdateOuterReferences                                       11779671 / 190131074                            27 / 44328                                     
org.apache.spark.sql.catalyst.analysis.SubstituteUnresolvedOrdinals                                11692300 / 171161803                            123 / 45102                                    
org.apache.spark.sql.catalyst.optimizer.EliminateDistinct                                          0 / 170471652                                   0 / 24864                                      
org.apache.spark.sql.catalyst.analysis.Analyzer$WindowsSubstitution                                4399008 / 141205255                             9 / 45102                                      
org.apache.spark.sql.catalyst.analysis.EliminateUnions                                             0 / 130718101                                   0 / 45102                                      
org.apache.spark.sql.catalyst.analysis.ResolveHints$RemoveAllHints                                 0 / 121849964                                   0 / 44787                                      
org.apache.spark.sql.execution.datasources.SchemaPruning                                           0 / 113229586                                   0 / 24840                                      
org.apache.spark.sql.execution.OptimizeMetadataOnlyQuery                                           0 / 110771374                                   0 / 24840                                      
     
23:52:58.159 WARN org.apache.spark.sql.SQLQueryTestSuite: 

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.SQLQueryTestSuite, thread names: Idle Worker Monitor for python3.6 =====

[info] ScalaTest
[info] Run completed in 2 hours, 1 minute, 30 seconds.
[info] Total number of tests run: 5899
[info] Suites: completed 287, aborted 0
[info] Tests: succeeded 5899, failed 0, canceled 0, ignored 24, pending 0
[info] All tests passed.
[info] Passed: Total 6005, Failed 0, Errors 0, Passed 6005, Ignored 24
[success] Total time: 7312 s, completed Jul 21, 2019 11:53:01 PM
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Packaging /home/jenkins/workspace/SparkPullRequestBuilder/core/target/scala-2.12/spark-core_2.12-3.0.0-SNAPSHOT.jar ...
[info] Done packaging.
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala:36: class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead
[warn]   def createPlan(batchId: Long): WriteToDataSourceV2 = {
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala:37: class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead
[warn]     WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:283: class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead
[warn]               WriteToDataSourceV2(write, df.logicalPlan)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:128: value ENABLE_JOB_SUMMARY in class ParquetOutputFormat is deprecated: see corresponding Javadoc for more information.
[warn]       && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:262: class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
[warn]         new org.apache.parquet.hadoop.ParquetInputSplit(
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:273: method readFooter in class ParquetFileReader is deprecated: see corresponding Javadoc for more information.
[warn]         ParquetFileReader.readFooter(sharedConf, filePath, SKIP_ROW_GROUPS).getFileMetaData
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:447: method readFooter in class ParquetFileReader is deprecated: see corresponding Javadoc for more information.
[warn]           ParquetFileReader.readFooter(
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala:91: value ENABLE_JOB_SUMMARY in class ParquetOutputFormat is deprecated: see corresponding Javadoc for more information.
[warn]       && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:121: class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
[warn]           Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T] = {
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:126: class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
[warn]       new org.apache.parquet.hadoop.ParquetInputSplit(
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:135: method readFooter in class ParquetFileReader is deprecated: see corresponding Javadoc for more information.
[warn]       ParquetFileReader.readFooter(conf, filePath, SKIP_ROW_GROUPS).getFileMetaData
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:184: class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
[warn]       split: ParquetInputSplit,
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:213: class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
[warn]       split: ParquetInputSplit,
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala:119: method poll in trait Consumer is deprecated: see corresponding Javadoc for more information.
[warn]     consumer.poll(0)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala:167: method poll in trait Consumer is deprecated: see corresponding Javadoc for more information.
[warn]         consumer.poll(0)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala:215: method poll in trait Consumer is deprecated: see corresponding Javadoc for more information.
[warn]       consumer.poll(0)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala:245: method poll in trait Consumer is deprecated: see corresponding Javadoc for more information.
[warn]       consumer.poll(0)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala:321: method poll in trait Consumer is deprecated: see corresponding Javadoc for more information.
[warn]           consumer.poll(0)
[warn] 
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala:474: method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for more information.
[warn]     val p = consumer.poll(pollTimeoutMs)
[warn] 
[info] KafkaRelationSuiteV1:
[info] - explicit earliest to latest offsets (11 seconds, 301 milliseconds)
[info] - default starting and ending offsets (1 second, 989 milliseconds)
[info] - explicit offsets (2 seconds, 969 milliseconds)
[info] - reuse same dataframe in query (1 second, 651 milliseconds)
[info] - test late binding start offsets (5 seconds, 510 milliseconds)
[info] - bad batch query options (30 milliseconds)
[info] - allow group.id prefix (2 seconds, 341 milliseconds)
[info] - allow group.id override (3 seconds, 575 milliseconds)
[info] - read Kafka transactional messages: read_committed (8 seconds, 544 milliseconds)
[info] - read Kafka transactional messages: read_uncommitted (5 seconds, 948 milliseconds)
[info] - V1 Source is used when set through SQLConf (12 milliseconds)
[info] KafkaRelationSuiteV2:
[info] - explicit earliest to latest offsets (2 seconds, 635 milliseconds)
[info] - default starting and ending offsets (1 second, 704 milliseconds)
[info] - explicit offsets (3 seconds, 641 milliseconds)
[info] - reuse same dataframe in query (1 second, 899 milliseconds)
[info] - test late binding start offsets (5 seconds, 678 milliseconds)
[info] - bad batch query options (199 milliseconds)
[info] - allow group.id prefix (2 seconds, 887 milliseconds)
[info] - allow group.id override (2 seconds, 652 milliseconds)
[info] - read Kafka transactional messages: read_committed (10 seconds, 987 milliseconds)
[info] - read Kafka transactional messages: read_uncommitted (4 seconds, 701 milliseconds)
[info] - V2 Source is used when set through SQLConf (9 milliseconds)
[info] KafkaOffsetRangeCalculatorSuite:
[info] - with no minPartition: N TopicPartitions to N offset ranges (4 milliseconds)
[info] - with no minPartition: empty ranges ignored (1 millisecond)
[info] - with minPartition = 3: N TopicPartitions to N offset ranges (2 milliseconds)
[info] - with minPartition = 4: 1 TopicPartition to N offset ranges (1 millisecond)
[info] - with minPartition = 3: N skewed TopicPartitions to M offset ranges (0 milliseconds)
[info] - with minPartition = 3: range inexact multiple of minPartitions (0 milliseconds)
[info] - with minPartition = 3: empty ranges ignored (1 millisecond)
[info] KafkaSourceProviderSuite:
[info] - micro-batch mode - options should be handled as case-insensitive (721 milliseconds)
[info] - SPARK-28142 - continuous mode - options should be handled as case-insensitive (6 milliseconds)
[info] KafkaSourceOffsetSuite:
[info] - comparison {"t":{"0":1}} <=> {"t":{"0":2}} (1 millisecond)
[info] - comparison {"t":{"1":0,"0":1}} <=> {"t":{"1":1,"0":2}} (1 millisecond)
[info] - comparison {"t":{"0":1},"T":{"0":0}} <=> {"t":{"0":2},"T":{"0":1}} (1 millisecond)
[info] - comparison {"t":{"0":1}} <=> {"t":{"1":1,"0":2}} (1 millisecond)
[info] - comparison {"t":{"0":1}} <=> {"t":{"1":3,"0":2}} (1 millisecond)
[info] - basic serialization - deserialization (3 milliseconds)
[info] - OffsetSeqLog serialization - deserialization (187 milliseconds)
[info] - read Spark 2.1.0 offset format (3 milliseconds)
[info] KafkaDataConsumerSuite:
[info] - SPARK-19886: Report error cause correctly in reportDataLoss (3 milliseconds)
[info] - SPARK-23623: concurrent use of KafkaDataConsumer (1 minute, 1 second)
[info] KafkaContinuousSourceSuite:
[info] - cannot stop Kafka stream (1 second, 694 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: true) (9 seconds, 443 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: true) (8 seconds, 951 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: true) (4 seconds, 933 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: true) (11 seconds, 130 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: true) (9 seconds, 995 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: true) (4 seconds, 39 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: true) (12 seconds, 11 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: true) (10 seconds, 27 milliseconds)
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: true) (4 seconds, 262 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: false) (8 seconds, 568 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: false) (7 seconds, 41 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: false) (4 seconds, 15 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: false) (12 seconds, 92 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: false) (9 seconds, 974 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: false) (3 seconds, 946 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: false) (10 seconds, 37 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: false) (8 seconds, 8 milliseconds)
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: false) (6 seconds, 914 milliseconds)
[info] - bad source options (11 milliseconds)
[info] - unsupported kafka configs (16 milliseconds)
[info] - get offsets from case insensitive parameters (2 milliseconds)
[info] - Kafka column types (2 seconds, 193 milliseconds)
[info] - ensure continuous stream is being used (79 milliseconds)
[info] - read Kafka transactional messages: read_committed (10 seconds, 578 milliseconds)
[info] - read Kafka transactional messages: read_uncommitted (3 seconds, 227 milliseconds)
[info] - SPARK-27494: read kafka record containing null key/values. (2 seconds, 981 milliseconds)
[info] KafkaContinuousSourceTopicDeletionSuite:
[info] - ensure continuous stream is being used (127 milliseconds)
[info] - subscribing topic by pattern with topic deletions (9 seconds, 305 milliseconds)
[info] KafkaContinuousSourceStressForDontFailOnDataLossSuite:
[info] - stress test for failOnDataLoss=false (21 seconds, 244 milliseconds)
[info] KafkaContinuousSinkSuite:
[info] - ensure continuous stream is being used (95 milliseconds)
[info] - streaming - write to kafka with topic field (3 seconds, 491 milliseconds)
[info] - streaming - write w/o topic field, with topic option (1 second, 566 milliseconds)
[info] - streaming - topic field and topic option (1 second, 627 milliseconds)
[info] - null topic attribute (1 second, 9 milliseconds)
[info] - streaming - write data with bad schema (376 milliseconds)
[info] - streaming - write data with valid schema but wrong types (395 milliseconds)
[info] - streaming - write to non-existing topic (1 second, 482 milliseconds)
[info] - streaming - exception on config serializer (255 milliseconds)
[info] - generic - write big data with small producer buffer (1 minute, 13 seconds)
[info] KafkaSinkStreamingSuite:
[info] - streaming - write to kafka with topic field (1 second, 771 milliseconds)
[info] - streaming - write aggregation w/o topic field, with topic option (3 seconds, 354 milliseconds)
[info] - streaming - aggregation with topic field and topic option (2 seconds, 428 milliseconds)
[info] - streaming - sink progress is produced (588 milliseconds)
[info] - streaming - write data with bad schema (263 milliseconds)
[info] - streaming - write data with valid schema but wrong types (354 milliseconds)
[info] - streaming - write to non-existing topic (5 seconds, 248 milliseconds)
[info] - streaming - exception on config serializer (194 milliseconds)
[info] KafkaSinkBatchSuiteV1:
[info] - batch - write to kafka (1 second, 229 milliseconds)
[info] - batch - null topic field value, and no topic option (74 milliseconds)
[info] - SPARK-20496: batch - enforce analyzed plans (1 second, 248 milliseconds)
[info] - batch - unsupported save modes (453 milliseconds)
[info] KafkaSinkBatchSuiteV2:
[info] - batch - write to kafka (1 second, 381 milliseconds)
[info] - batch - null topic field value, and no topic option (155 milliseconds)
[info] - SPARK-20496: batch - enforce analyzed plans (914 milliseconds)
[info] - batch - unsupported save modes (297 milliseconds)
[info] - generic - write big data with small producer buffer (1 minute, 15 seconds)
[info] KafkaMicroBatchV1SourceSuite:
[info] - cannot stop Kafka stream (1 second, 459 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: true) (4 seconds, 544 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: true) (4 seconds, 137 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: true) (2 seconds, 242 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: true) (4 seconds, 134 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: true) (4 seconds, 226 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: true) (2 seconds, 677 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: true) (3 seconds, 533 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: true) (3 seconds, 351 milliseconds)
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: true) (2 seconds, 843 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: false) (4 seconds, 936 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: false) (3 seconds, 286 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: false) (3 seconds, 338 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: false) (5 seconds, 671 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: false) (5 seconds, 616 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: false) (5 seconds, 199 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: false) (5 seconds, 855 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: false) (5 seconds, 116 milliseconds)
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: false) (3 seconds, 589 milliseconds)
[info] - bad source options (12 milliseconds)
[info] - unsupported kafka configs (15 milliseconds)
[info] - get offsets from case insensitive parameters (1 millisecond)
[info] - Kafka column types (1 second, 128 milliseconds)
[info] - (de)serialization of initial offsets (930 milliseconds)
[info] - SPARK-26718 Rate limit set to Long.Max should not overflow integer during end offset calculation (1 second, 175 milliseconds)
[info] - maxOffsetsPerTrigger (5 seconds, 870 milliseconds)
[info] - input row metrics (1 second, 301 milliseconds)
[info] - subscribing topic by pattern with topic deletions (10 seconds, 788 milliseconds)
[info] - subscribe topic by pattern with topic recreation between batches (3 seconds, 967 milliseconds)
[info] - ensure that initial offset are written with an extra byte in the beginning (SPARK-19517) (2 seconds, 384 milliseconds)
[info] - deserialization of initial offset written by Spark 2.1.0 (SPARK-19517) (1 second, 458 milliseconds)
[info] - deserialization of initial offset written by future version (295 milliseconds)
[info] - KafkaSource with watermark (1 second, 657 milliseconds)
Attempting to post to Github...
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Phadoop-2.7 -Phive-thriftserver -Phive -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest hive-thriftserver/test avro/test mllib/test hive/test repl/test examples/test sql/test sql-kafka-0-10/test ; process was terminated by signal 9
 > Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107979/
Test FAILed.
Finished: FAILURE