SuccessChanges

Summary

  1. [SPARK-25116][TESTS] Fix the Kafka cluster leak and clean up cached (details)
Commit da2dc69291cda8c8e7bb6b4a15001f768a97f65e by zsxwing
[SPARK-25116][TESTS] Fix the Kafka cluster leak and clean up cached
producers
## What changes were proposed in this pull request?
KafkaContinuousSinkSuite leaks a Kafka cluster because both
KafkaSourceTest and KafkaContinuousSinkSuite create a Kafka cluster but
`afterAll` only shuts down one cluster. This leaks a Kafka cluster and
causes that some Kafka thread crash and kill JVM when SBT is trying to
clean up tests.
This PR fixes the leak and also adds a shut down hook to detect Kafka
cluster leak.
In additions, it also fixes `AdminClient` leak and cleans up cached
producers (When a record is writtn using a producer, the producer will
keep refreshing the topic and I don't find an API to clear it except
closing the producer) to eliminate the following annoying logs:
``` 8/13 15:34:42.568 kafka-admin-client-thread | adminclient-4 WARN
NetworkClient: [AdminClient clientId=adminclient-4] Connection to node 0
could not be established. Broker may not be available. 18/08/13
15:34:42.570 kafka-admin-client-thread | adminclient-6 WARN
NetworkClient: [AdminClient clientId=adminclient-6] Connection to node 0
could not be established. Broker may not be available. 18/08/13
15:34:42.606 kafka-admin-client-thread | adminclient-8 WARN
NetworkClient: [AdminClient clientId=adminclient-8] Connection to node
-1 could not be established. Broker may not be available. 18/08/13
15:34:42.729 kafka-producer-network-thread | producer-797 WARN
NetworkClient: [Producer clientId=producer-797] Connection to node -1
could not be established. Broker may not be available. 18/08/13
15:34:42.906 kafka-producer-network-thread | producer-1598 WARN
NetworkClient: [Producer clientId=producer-1598] Connection to node 0
could not be established. Broker may not be available.
```
I also reverted
https://github.com/apache/spark/pull/22097/commits/b5eb54244ed573c8046f5abf7bf087f5f08dba58
introduced by #22097 since it doesn't help.
## How was this patch tested?
Jenkins
Closes #22106 from zsxwing/SPARK-25116.
Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Shixiong
Zhu <zsxwing@gmail.com>
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSinkSuite.scala (diff)
The file was addedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTest.scala
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaProducer.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaRelationSuite.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousReader.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala (diff)
The file was modifiedexternal/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/CachedKafkaProducerSuite.scala (diff)
The file was modifiedexternal/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSinkSuite.scala (diff)