SuccessChanges

Summary

  1. [SPARK-23942][PYTHON][SQL][BRANCH-2.3] Makes collect in PySpark as (commit: d4f204c5321cdc3955a48e9717ba06aaebbc2ab4) (details)
Commit d4f204c5321cdc3955a48e9717ba06aaebbc2ab4 by hyukjinkwon
[SPARK-23942][PYTHON][SQL][BRANCH-2.3] Makes collect in PySpark as
action for a query executor listener
## What changes were proposed in this pull request?
This PR proposes to add `collect` to  a query executor as an action.
Seems `collect` / `collect` with Arrow are not recognised via
`QueryExecutionListener` as an action. For example, if we have a custom
listener as below:
```scala package org.apache.spark.sql
import org.apache.spark.internal.Logging import
org.apache.spark.sql.execution.QueryExecution import
org.apache.spark.sql.util.QueryExecutionListener
class TestQueryExecutionListener extends QueryExecutionListener with
Logging {
override def onSuccess(funcName: String, qe: QueryExecution,
durationNs: Long): Unit = {
   logError("Look at me! I'm 'onSuccess'")
}
  override def onFailure(funcName: String, qe: QueryExecution,
exception: Exception): Unit = { }
}
``` and set `spark.sql.queryExecutionListeners` to
`org.apache.spark.sql.TestQueryExecutionListener`
Other operations in PySpark or Scala side seems fine:
```python
>>> sql("SELECT * FROM range(1)").show()
```
``` 18/04/09 17:02:04 ERROR TestQueryExecutionListener: Look at me! I'm
'onSuccess'
+---+
| id|
+---+
|  0|
+---+
```
```scala scala> sql("SELECT * FROM range(1)").collect()
```
``` 18/04/09 16:58:41 ERROR TestQueryExecutionListener: Look at me! I'm
'onSuccess' res1: Array[org.apache.spark.sql.Row] = Array([0])
```
but ..
**Before**
```python
>>> sql("SELECT * FROM range(1)").collect()
```
```
[Row(id=0)]
```
```python
>>> spark.conf.set("spark.sql.execution.arrow.enabled", "true")
>>> sql("SELECT * FROM range(1)").toPandas()
```
```
  id 0   0
```
**After**
```python
>>> sql("SELECT * FROM range(1)").collect()
```
``` 18/04/09 16:57:58 ERROR TestQueryExecutionListener: Look at me! I'm
'onSuccess'
[Row(id=0)]
```
```python
>>> spark.conf.set("spark.sql.execution.arrow.enabled", "true")
>>> sql("SELECT * FROM range(1)").toPandas()
```
``` 18/04/09 17:53:26 ERROR TestQueryExecutionListener: Look at me! I'm
'onSuccess'
  id 0   0
```
## How was this patch tested?
I have manually tested as described above and unit test was added.
Author: hyukjinkwon <gurwls223@apache.org>
Closes #21060 from HyukjinKwon/PR_TOOL_PICK_PR_21007_BRANCH-2.3.
(commit: d4f204c5321cdc3955a48e9717ba06aaebbc2ab4)
The file was modifiedpython/pyspark/sql/tests.py (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala (diff)
The file was addedsql/core/src/test/scala/org/apache/spark/sql/TestQueryExecutionListener.scala