AbortedChanges

Summary

  1. [SPARK-30480][PYTHON][TESTS] Increases the memory limit being tested in (details)
  2. [SPARK-30493][PYTHON][ML] Remove OneVsRestModel setClassifier, (details)
  3. [SPARK-30377][ML] Make Regressors extend abstract class Regressor (details)
  4. [SPARK-30351][ML][PYSPARK] BisectingKMeans support instance weighting (details)
  5. [SPARK-30188][SQL] Resolve the failed unit tests when enable AQE (details)
Commit 0823aec4630e70323e66bea243871aaab761d9ca by gurwls223
[SPARK-30480][PYTHON][TESTS] Increases the memory limit being tested in
'WorkerMemoryTest.test_memory_limit'
### What changes were proposed in this pull request?
This PR proposes to increase the memory in
`WorkerMemoryTest.test_memory_limit` in order to make the test pass with
PyPy.
The test is currently failed only in PyPy as below in some PRs
unexpectedly:
``` Current mem limits: 18446744073709551615 of max 18446744073709551615
Setting mem limits to 1048576 of max 1048576
RPython traceback:
File "pypy_module_pypyjit_interp_jit.c", line 289, in portal_5
File "pypy_interpreter_pyopcode.c", line 3468, in
handle_bytecode__AccessDirect_None
File "pypy_interpreter_pyopcode.c", line 5558, in
dispatch_bytecode__AccessDirect_None out of memory: couldn't allocate
the next arena ERROR
```
It seems related to how PyPy allocates the memory and GC works
PyPy-specifically. There seems nothing wrong in this configuration
implementation itself in PySpark side.
I roughly tested in higher PyPy versions on Ubuntu (PyPy v7.3.0) and
this test seems passing fine so I suspect this might be an issue in old
PyPy behaviours.
The change only increases the limit so it would not affect actual memory
allocations. It just needs to test if the limit is properly set in
worker sides. For clarification, the memory is unlimited in the machine
if not set.
### Why are the changes needed?
To make the tests pass and unblock other PRs.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Manually and Jenkins should test it out.
Closes #27186 from HyukjinKwon/SPARK-30480.
Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by:
HyukjinKwon <gurwls223@apache.org>
The file was modifiedpython/pyspark/tests/test_worker.py (diff)
The file was modifiedpython/run-tests.py (diff)
Commit 6502c66025718bf45e0e2ee12398b7b92da41a0c by ruifengz
[SPARK-30493][PYTHON][ML] Remove OneVsRestModel setClassifier,
setLabelCol and setWeightCol methods
### What changes were proposed in this pull request?
Removal of `OneVsRestModel.setClassifier`, `OneVsRestModel.setLabelCol`
and `OneVsRestModel.setWeightCol`  methods.
### Why are the changes needed?
Aforementioned methods shouldn't by included by
[SPARK-29093](https://issues.apache.org/jira/browse/SPARK-29093), as
they're not present in Scala `OneVsRestModel` and have no practical
application.
### Does this PR introduce any user-facing change?
Not beyond scope of SPARK-29093].
### How was this patch tested?
Existing tests.
CC huaxingao zhengruifeng
Closes #27181 from zero323/SPARK-30493.
Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by:
zhengruifeng <ruifengz@foxmail.com>
The file was modifiedpython/pyspark/ml/classification.py (diff)
Commit d6e28f29228aa3ff3ba40b00bd79e1844fcc0d71 by srowen
[SPARK-30377][ML] Make Regressors extend abstract class Regressor
### What changes were proposed in this pull request? Make Regressors
extend abstract class Regressor:
```AFTSurvivalRegression extends Estimator => extends Regressor```
```DecisionTreeRegressor extends Predictor => extends Regressor```
```FMRegressor extends Predictor => extends Regressor```
```GBTRegressor extends Predictor => extends Regressor```
```RandomForestRegressor extends Predictor => extends Regressor```
We will not make ```IsotonicRegression``` extend ```Regressor``` because
it is tricky to handle both DoubleType and VectorType.
### Why are the changes needed? Make class hierarchy consistent for all
Regressors
### Does this PR introduce any user-facing change? No
### How was this patch tested? existing tests
Closes #27168 from huaxingao/spark-30377.
Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Sean Owen
<srowen@gmail.com>
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala (diff)
The file was modifiedpython/pyspark/ml/regression.py (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala (diff)
The file was modifiedproject/MimaExcludes.scala (diff)
Commit f77dcfc55af968cdfb29ac798e21229fe7f6c063 by srowen
[SPARK-30351][ML][PYSPARK] BisectingKMeans support instance weighting
### What changes were proposed in this pull request? add weight support
in BisectingKMeans
### Why are the changes needed? BisectingKMeans should support instance
weighting
### Does this PR introduce any user-facing change? Yes.
BisectingKMeans.setWeight
### How was this patch tested? Unit test
Closes #27035 from huaxingao/spark_30351.
Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Sean Owen
<srowen@gmail.com>
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/clustering/DistanceMeasure.scala (diff)
The file was modifiedpython/pyspark/ml/clustering.py (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala (diff)
The file was modifiedmllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala (diff)
Commit b389b8c5f0650a7e63098f18437fcaa29998732a by wenchen
[SPARK-30188][SQL] Resolve the failed unit tests when enable AQE
### What changes were proposed in this pull request? Fix all the failed
tests when enable AQE.
### Why are the changes needed? Run more tests with AQE to catch bugs,
and make it easier to enable AQE by default in the future.
### Does this PR introduce any user-facing change? No
### How was this patch tested? Existing unit tests
Closes #26813 from JkSelf/enableAQEDefault.
Authored-by: jiake <ke.a.jia@intel.com> Signed-off-by: Wenchen Fan
<wenchen@databricks.com>
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ObjectHashAggregateSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/DeprecatedWholeStageCodegenSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/debug/DebuggingSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala (diff)
The file was modifiedpython/pyspark/sql/window.py (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExecSuite.scala (diff)
The file was modifiedsql/core/src/test/resources/sql-tests/inputs/explain.sql (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/ConfigBehaviorSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala (diff)
The file was modifiedR/pkg/tests/fulltests/test_mllib_recommendation.R (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowFunctionsSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/BroadcastExchangeSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala (diff)
The file was modifiedmllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala (diff)
The file was modifiedpython/pyspark/sql/udf.py (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/LogicalPlanTagInSparkPlanSuite.scala (diff)
The file was modifiedpython/pyspark/sql/column.py (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala (diff)
The file was modifiedpython/pyspark/sql/dataframe.py (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala (diff)
The file was modifiedsql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLMetricsSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSparkSubmitSuite.scala (diff)
The file was modifiedsql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/DemoteBroadcastHashJoin.scala (diff)
The file was modifiedpython/pyspark/sql/tests/test_pandas_udf_grouped_agg.py (diff)
The file was modifiedsql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala (diff)