SuccessChanges

Summary

  1. [SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for (commit: 14b50d7fee58d56cb8843994b1a423a6b475dcb5) (details)
  2. [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7 (commit: 136588e95f69923a04458abe4862d336e5244c84) (details)
Commit 14b50d7fee58d56cb8843994b1a423a6b475dcb5 by cody
[SPARK-24987][SS] - Fix Kafka consumer leak when no new offsets for
TopicPartition
## What changes were proposed in this pull request?
This small fix adds a `consumer.release()` call to `KafkaSourceRDD` in
the case where we've retrieved offsets from Kafka, but the `fromOffset`
is equal to the `lastOffset`, meaning there is no new data to read for a
particular topic partition. Up until now, we'd just return an empty
iterator without closing the consumer which would cause a FD leak.
If accepted, this pull request should be merged into master as well.
## How was this patch tested?
Haven't ran any specific tests, would love help on how to test methods
running inside `RDD.compute`.
Author: Yuval Itzchakov <yuval.itzchakov@clicktale.com>
Closes #21997 from YuvalItzchakov/master.
(cherry picked from commit b7fdf8eb2011ae76f0161caa9da91e29f52f05e4)
Signed-off-by: cody koeninger <cody@koeninger.org>
(commit: 14b50d7fee58d56cb8843994b1a423a6b475dcb5)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala (diff)
Commit 136588e95f69923a04458abe4862d336e5244c84 by srowen
[SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
## What changes were proposed in this pull request?
Update Hadoop 2.7 to 2.7.7 to pull in bug and security fixes.
## How was this patch tested?
Existing tests.
Author: Sean Owen <srowen@gmail.com>
Closes #21987 from srowen/SPARK-25015.
(cherry picked from commit 5f9633dc97ad5f78dd17cad39945ea32f3441f06)
Signed-off-by: Sean Owen <srowen@gmail.com>
(commit: 136588e95f69923a04458abe4862d336e5244c84)
The file was modifieddocs/building-spark.md (diff)
The file was modifiedassembly/README (diff)
The file was modifiedpom.xml (diff)
The file was modifieddev/deps/spark-deps-hadoop-2.7 (diff)