1. [SPARK-24259][SQL] ArrayWriter for Arrow produces wrong output (details)
Commit e16ab6fe05503872a323a31dc6755aaa5d12c68c by wenchen
[SPARK-24259][SQL] ArrayWriter for Arrow produces wrong output
## What changes were proposed in this pull request?
Right now `ArrayWriter` used to output Arrow data for array type,
doesn't do `clear` or `reset` after each batch. It produces wrong
## How was this patch tested?
Added test.
Author: Liang-Chi Hsieh <>
Closes #21312 from viirya/SPARK-24259.
(cherry picked from commit d610d2a3f57ca551f72cb4e5dfed78f27be62eec)
Signed-off-by: Wenchen Fan <>
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala (diff)
The file was modifiedpython/pyspark/sql/ (diff)