1. [SPARK-25710][SQL] range should report metrics correctly (details)
Commit 34f229bc2116e443fa7f4e20d5277c4a27d02c47 by wenchen
[SPARK-25710][SQL] range should report metrics correctly
## What changes were proposed in this pull request?
Currently `Range` reports metrics in batch granularity. This is
acceptable, but it's better if we can make it row granularity without
performance penalty.
Before this PR,  the metrics are updated when preparing the batch, which
is before we actually consume data. In this PR, the metrics are updated
after the data are consumed. There are 2 different cases: 1. The data
processing loop has a stop check. The metrics are updated when we need
to stop. 2. no stop check. The metrics are updated after the loop.
## How was this patch tested?
existing tests and a new benchmark
Closes #22698 from cloud-fan/range.
Authored-by: Wenchen Fan <> Signed-off-by: Wenchen
Fan <>
The file was addedsql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/RangeBenchmark.scala
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala (diff)
The file was addedsql/core/benchmarks/RangeBenchmark-results.txt
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala (diff)