GitHub pull request #5143 of commit 06c445fa4deed19491bdcf2feb9b436e5c2ca732 automatically merged. [EnvInject] - Loading node environment variables. Building remotely on ubuntu-testing (ubuntu-testing-gpu) in workspace /home/jenkins/workspace/Ray-Perf-Integration-PRB [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository https://github.com/ray-project/ray.git > git init /home/jenkins/workspace/Ray-Perf-Integration-PRB # timeout=10 Fetching upstream changes from https://github.com/ray-project/ray.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10 Fetching upstream changes from https://github.com/ray-project/ray.git > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/pull/*:refs/remotes/origin/pr/* > git rev-parse origin/pr/5143/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains ac4bce8a9b9699e58b93e988e641254036c9bf4e # timeout=10 Checking out Revision ac4bce8a9b9699e58b93e988e641254036c9bf4e (origin/pr/5143/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f ac4bce8a9b9699e58b93e988e641254036c9bf4e > git rev-list f59308a9117f00c45fa2c7dd10c9349abc00187b # timeout=10 First time build. Skipping changelog. [Ray-Perf-Integration-PRB] $ /bin/bash /tmp/hudson6728044425974215826.sh + [[ 2 -gt 0 ]] + key=--output-sha + case $key in + OUTPUT_SHA=YES + shift + [[ 1 -gt 0 ]] + key=--no-cache + case $key in + NO_CACHE=--no-cache + shift + [[ 0 -gt 0 ]] + '[' YES ']' ++ docker build --no-cache -q -t ray-project/base-deps docker/base-deps + IMAGE_SHA=sha256:eac19b63fad6f872782df257f48739a33c2207ae0496e22b178346c6c1d5bf64 + git rev-parse HEAD ++ git rev-parse HEAD + git archive -o ./docker/deploy/ray.tar ac4bce8a9b9699e58b93e988e641254036c9bf4e + '[' YES ']' ++ docker build --no-cache -q -t ray-project/deploy docker/deploy + IMAGE_SHA=sha256:5c0583e73cf676547c07e24bbdcdc69117f504f981b699264cb366b0bb849c2a + rm ./docker/deploy/ray.tar ./docker/deploy/git-rev + '[' '!' ']' + '[' YES ']' ++ docker build --no-cache -q -t ray-project/examples docker/examples + IMAGE_SHA=sha256:896df471550e50595f68fcee102cd72e514c083f31a02b45086773bf00d597b1 + '[' YES ']' + echo sha256:896df471550e50595f68fcee102cd72e514c083f31a02b45086773bf00d597b1 + sed s/sha256:// Using Docker image 896df471550e50595f68fcee102cd72e514c083f31a02b45086773bf00d597b1 WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. +++ dirname ./ci/jenkins_tests/perf_integration_tests/run_perf_integration.sh ++ cd ./ci/jenkins_tests/perf_integration_tests ++ pwd + ROOT_DIR=/ray/ci/jenkins_tests/perf_integration_tests + pushd /ray/ci/jenkins_tests/perf_integration_tests + python -m pip install pytest-benchmark /ray/ci/jenkins_tests/perf_integration_tests /ray Collecting pytest-benchmark Downloading https://files.pythonhosted.org/packages/ab/0f/8488bebc18924bbf5ed9eff3d3cecd0287446c88962f10e99fa727a3d2d7/pytest_benchmark-3.2.2-py2.py3-none-any.whl (48kB) Collecting py-cpuinfo (from pytest-benchmark) Downloading https://files.pythonhosted.org/packages/42/60/63f28a5401da733043abe7053e7d9591491b4784c4f87c339bf51215aa0a/py-cpuinfo-5.0.0.tar.gz (82kB) Requirement already satisfied: pytest>=3.8 in /opt/conda/lib/python3.6/site-packages (from pytest-benchmark) (5.0.1) Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.18) Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (18.1.0) Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (1.5.3) Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.1.7) Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (4.1.0) Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (1.3.0) Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.12.0) Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (17.1) Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=3.8->pytest-benchmark) (0.5.2) Requirement already satisfied: six<2.0.0,>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from more-itertools>=4.0.0->pytest>=3.8->pytest-benchmark) (1.11.0) Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest>=3.8->pytest-benchmark) (2.2.0) Building wheels for collected packages: py-cpuinfo Running setup.py bdist_wheel for py-cpuinfo: started Running setup.py bdist_wheel for py-cpuinfo: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/01/7e/a9/b982d0fea22b7e4ae5619de949570cde5ad55420cec16e86a5 Successfully built py-cpuinfo distributed 1.21.8 requires dask>=0.17.0, which is not installed. distributed 1.21.8 requires msgpack, which is not installed. blaze 0.11.3 requires dask[dataframe], which is not installed. jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible. Installing collected packages: py-cpuinfo, pytest-benchmark Successfully installed py-cpuinfo-5.0.0 pytest-benchmark-3.2.2 You are using pip version 10.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. + pip install -U https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl Collecting ray==0.8.0.dev2 from https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl Downloading https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl (48.4MB) Requirement not upgraded as not directly required: pyyaml in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.12) Requirement not upgraded as not directly required: colorama in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (0.3.9) Requirement not upgraded as not directly required: click in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (6.7) Requirement not upgraded as not directly required: filelock in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.0.4) Requirement not upgraded as not directly required: funcsigs in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.0.2) Requirement not upgraded as not directly required: redis in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.2.1) Requirement not upgraded as not directly required: pytest in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (5.0.1) Requirement not upgraded as not directly required: protobuf>=3.8.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.9.0) Requirement not upgraded as not directly required: six>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.11.0) Requirement not upgraded as not directly required: numpy>=1.14 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.16.4) Requirement not upgraded as not directly required: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.5.3) Requirement not upgraded as not directly required: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (4.1.0) Requirement not upgraded as not directly required: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.3.0) Requirement not upgraded as not directly required: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.12.0) Requirement not upgraded as not directly required: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (18.1.0) Requirement not upgraded as not directly required: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.1.7) Requirement not upgraded as not directly required: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (17.1) Requirement not upgraded as not directly required: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.18) Requirement not upgraded as not directly required: setuptools in /opt/conda/lib/python3.6/site-packages (from protobuf>=3.8.0->ray==0.8.0.dev2) (41.0.1) Requirement not upgraded as not directly required: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->ray==0.8.0.dev2) (2.2.0) Requirement not upgraded as not directly required: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->ray==0.8.0.dev2) (0.5.2) distributed 1.21.8 requires dask>=0.17.0, which is not installed. distributed 1.21.8 requires msgpack, which is not installed. blaze 0.11.3 requires dask[dataframe], which is not installed. jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible. Installing collected packages: ray Found existing installation: ray 0.8.0.dev2 Uninstalling ray-0.8.0.dev2: Successfully uninstalled ray-0.8.0.dev2 Successfully installed ray-0.8.0.dev2 You are using pip version 10.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. + python -m pytest --benchmark-autosave --benchmark-min-rounds=10 '--benchmark-columns=min, max, mean' /ray/ci/jenkins_tests/perf_integration_tests/../../../python/ray/tests/perf_integration_tests/test_perf_integration.py ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-5.0.1, py-1.5.3, pluggy-0.12.0 benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=10 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /ray/python plugins: arraydiff-0.2, openfiles-0.3.0, doctestplus-0.1.3, benchmark-3.2.2, remotedata-0.3.1, timeout-1.3.3 collected 12 items ../../../python/ray/tests/perf_integration_tests/test_perf_integration.py . [ 8%] ........... [100%] Saved benchmark data in: /ray/ci/jenkins_tests/perf_integration_tests/.benchmarks/Linux-CPython-3.6-64bit/0001_unversioned_20190713_225612.json Computing stats ... Computing stats ... group 1/1 Computing stats ... group 1/1: min Computing stats ... group 1/1: min (1/12) Computing stats ... group 1/1: min (2/12) Computing stats ... group 1/1: min (3/12) Computing stats ... group 1/1: min (4/12) Computing stats ... group 1/1: min (5/12) Computing stats ... group 1/1: min (6/12) Computing stats ... group 1/1: min (7/12) Computing stats ... group 1/1: min (8/12) Computing stats ... group 1/1: min (9/12) Computing stats ... group 1/1: min (10/12) Computing stats ... group 1/1: min (11/12) Computing stats ... group 1/1: min (12/12) Computing stats ... group 1/1: min (1/12) Computing stats ... group 1/1: min (2/12) Computing stats ... group 1/1: min (3/12) Computing stats ... group 1/1: min (4/12) Computing stats ... group 1/1: min (5/12) Computing stats ... group 1/1: min (6/12) Computing stats ... group 1/1: min (7/12) Computing stats ... group 1/1: min (8/12) Computing stats ... group 1/1: min (9/12) Computing stats ... group 1/1: min (10/12) Computing stats ... group 1/1: min (11/12) Computing stats ... group 1/1: min (12/12) Computing stats ... group 1/1: max Computing stats ... group 1/1: max (1/12) Computing stats ... group 1/1: max (2/12) Computing stats ... group 1/1: max (3/12) Computing stats ... group 1/1: max (4/12) Computing stats ... group 1/1: max (5/12) Computing stats ... group 1/1: max (6/12) Computing stats ... group 1/1: max (7/12) Computing stats ... group 1/1: max (8/12) Computing stats ... group 1/1: max (9/12) Computing stats ... group 1/1: max (10/12) Computing stats ... group 1/1: max (11/12) Computing stats ... group 1/1: max (12/12) Computing stats ... group 1/1: max (1/12) Computing stats ... group 1/1: max (2/12) Computing stats ... group 1/1: max (3/12) Computing stats ... group 1/1: max (4/12) Computing stats ... group 1/1: max (5/12) Computing stats ... group 1/1: max (6/12) Computing stats ... group 1/1: max (7/12) Computing stats ... group 1/1: max (8/12) Computing stats ... group 1/1: max (9/12) Computing stats ... group 1/1: max (10/12) Computing stats ... group 1/1: max (11/12) Computing stats ... group 1/1: max (12/12) Computing stats ... group 1/1: mean Computing stats ... group 1/1: mean (1/12) Computing stats ... group 1/1: mean (2/12) Computing stats ... group 1/1: mean (3/12) Computing stats ... group 1/1: mean (4/12) Computing stats ... group 1/1: mean (5/12) Computing stats ... group 1/1: mean (6/12) Computing stats ... group 1/1: mean (7/12) Computing stats ... group 1/1: mean (8/12) Computing stats ... group 1/1: mean (9/12) Computing stats ... group 1/1: mean (10/12) Computing stats ... group 1/1: mean (11/12) Computing stats ... group 1/1: mean (12/12) Computing stats ... group 1/1: mean (1/12) Computing stats ... group 1/1: mean (2/12) Computing stats ... group 1/1: mean (3/12) Computing stats ... group 1/1: mean (4/12) Computing stats ... group 1/1: mean (5/12) Computing stats ... group 1/1: mean (6/12) Computing stats ... group 1/1: mean (7/12) Computing stats ... group 1/1: mean (8/12) Computing stats ... group 1/1: mean (9/12) Computing stats ... group 1/1: mean (10/12) Computing stats ... group 1/1: mean (11/12) Computing stats ... group 1/1: mean (12/12) Computing stats ... group 1/1: median Computing stats ... group 1/1: median (1/12) Computing stats ... group 1/1: median (2/12) Computing stats ... group 1/1: median (3/12) Computing stats ... group 1/1: median (4/12) Computing stats ... group 1/1: median (5/12) Computing stats ... group 1/1: median (6/12) Computing stats ... group 1/1: median (7/12) Computing stats ... group 1/1: median (8/12) Computing stats ... group 1/1: median (9/12) Computing stats ... group 1/1: median (10/12) Computing stats ... group 1/1: median (11/12) Computing stats ... group 1/1: median (12/12) Computing stats ... group 1/1: median (1/12) Computing stats ... group 1/1: median (2/12) Computing stats ... group 1/1: median (3/12) Computing stats ... group 1/1: median (4/12) Computing stats ... group 1/1: median (5/12) Computing stats ... group 1/1: median (6/12) Computing stats ... group 1/1: median (7/12) Computing stats ... group 1/1: median (8/12) Computing stats ... group 1/1: median (9/12) Computing stats ... group 1/1: median (10/12) Computing stats ... group 1/1: median (11/12) Computing stats ... group 1/1: median (12/12) Computing stats ... group 1/1: iqr Computing stats ... group 1/1: iqr (1/12) Computing stats ... group 1/1: iqr (2/12) Computing stats ... group 1/1: iqr (3/12) Computing stats ... group 1/1: iqr (4/12) Computing stats ... group 1/1: iqr (5/12) Computing stats ... group 1/1: iqr (6/12) Computing stats ... group 1/1: iqr (7/12) Computing stats ... group 1/1: iqr (8/12) Computing stats ... group 1/1: iqr (9/12) Computing stats ... group 1/1: iqr (10/12) Computing stats ... group 1/1: iqr (11/12) Computing stats ... group 1/1: iqr (12/12) Computing stats ... group 1/1: iqr (1/12) Computing stats ... group 1/1: iqr (2/12) Computing stats ... group 1/1: iqr (3/12) Computing stats ... group 1/1: iqr (4/12) Computing stats ... group 1/1: iqr (5/12) Computing stats ... group 1/1: iqr (6/12) Computing stats ... group 1/1: iqr (7/12) Computing stats ... group 1/1: iqr (8/12) Computing stats ... group 1/1: iqr (9/12) Computing stats ... group 1/1: iqr (10/12) Computing stats ... group 1/1: iqr (11/12) Computing stats ... group 1/1: iqr (12/12) Computing stats ... group 1/1: stddev Computing stats ... group 1/1: stddev (1/12) Computing stats ... group 1/1: stddev (2/12) Computing stats ... group 1/1: stddev (3/12) Computing stats ... group 1/1: stddev (4/12) Computing stats ... group 1/1: stddev (5/12) Computing stats ... group 1/1: stddev (6/12) Computing stats ... group 1/1: stddev (7/12) Computing stats ... group 1/1: stddev (8/12) Computing stats ... group 1/1: stddev (9/12) Computing stats ... group 1/1: stddev (10/12) Computing stats ... group 1/1: stddev (11/12) Computing stats ... group 1/1: stddev (12/12) Computing stats ... group 1/1: stddev (1/12) Computing stats ... group 1/1: stddev (2/12) Computing stats ... group 1/1: stddev (3/12) Computing stats ... group 1/1: stddev (4/12) Computing stats ... group 1/1: stddev (5/12) Computing stats ... group 1/1: stddev (6/12) Computing stats ... group 1/1: stddev (7/12) Computing stats ... group 1/1: stddev (8/12) Computing stats ... group 1/1: stddev (9/12) Computing stats ... group 1/1: stddev (10/12) Computing stats ... group 1/1: stddev (11/12) Computing stats ... group 1/1: stddev (12/12) Computing stats ... group 1/1: ops Computing stats ... group 1/1: ops (1/12) Computing stats ... group 1/1: ops (2/12) Computing stats ... group 1/1: ops (3/12) Computing stats ... group 1/1: ops (4/12) Computing stats ... group 1/1: ops (5/12) Computing stats ... group 1/1: ops (6/12) Computing stats ... group 1/1: ops (7/12) Computing stats ... group 1/1: ops (8/12) Computing stats ... group 1/1: ops (9/12) Computing stats ... group 1/1: ops (10/12) Computing stats ... group 1/1: ops (11/12) Computing stats ... group 1/1: ops (12/12) Computing stats ... group 1/1: ops (1/12) Computing stats ... group 1/1: ops (2/12) Computing stats ... group 1/1: ops (3/12) Computing stats ... group 1/1: ops (4/12) Computing stats ... group 1/1: ops (5/12) Computing stats ... group 1/1: ops (6/12) Computing stats ... group 1/1: ops (7/12) Computing stats ... group 1/1: ops (8/12) Computing stats ... group 1/1: ops (9/12) Computing stats ... group 1/1: ops (10/12) Computing stats ... group 1/1: ops (11/12) Computing stats ... group 1/1: ops (12/12) Computing stats ... group 1/1: ops: outliers Computing stats ... group 1/1: ops: outliers (1/12) Computing stats ... group 1/1: ops: outliers (2/12) Computing stats ... group 1/1: ops: outliers (3/12) Computing stats ... group 1/1: ops: outliers (4/12) Computing stats ... group 1/1: ops: outliers (5/12) Computing stats ... group 1/1: ops: outliers (6/12) Computing stats ... group 1/1: ops: outliers (7/12) Computing stats ... group 1/1: ops: outliers (8/12) Computing stats ... group 1/1: ops: outliers (9/12) Computing stats ... group 1/1: ops: outliers (10/12) Computing stats ... group 1/1: ops: outliers (11/12) Computing stats ... group 1/1: ops: outliers (12/12) Computing stats ... group 1/1: ops: rounds Computing stats ... group 1/1: ops: rounds (1/12) Computing stats ... group 1/1: ops: rounds (2/12) Computing stats ... group 1/1: ops: rounds (3/12) Computing stats ... group 1/1: ops: rounds (4/12) Computing stats ... group 1/1: ops: rounds (5/12) Computing stats ... group 1/1: ops: rounds (6/12) Computing stats ... group 1/1: ops: rounds (7/12) Computing stats ... group 1/1: ops: rounds (8/12) Computing stats ... group 1/1: ops: rounds (9/12) Computing stats ... group 1/1: ops: rounds (10/12) Computing stats ... group 1/1: ops: rounds (11/12) Computing stats ... group 1/1: ops: rounds (12/12) Computing stats ... group 1/1: ops: iterations Computing stats ... group 1/1: ops: iterations (1/12) Computing stats ... group 1/1: ops: iterations (2/12) Computing stats ... group 1/1: ops: iterations (3/12) Computing stats ... group 1/1: ops: iterations (4/12) Computing stats ... group 1/1: ops: iterations (5/12) Computing stats ... group 1/1: ops: iterations (6/12) Computing stats ... group 1/1: ops: iterations (7/12) Computing stats ... group 1/1: ops: iterations (8/12) Computing stats ... group 1/1: ops: iterations (9/12) Computing stats ... group 1/1: ops: iterations (10/12) Computing stats ... group 1/1: ops: iterations (11/12) Computing stats ... group 1/1: ops: iterations (12/12) --------------------------------------------- benchmark: 12 tests ---------------------------------------------- Name (time in ms) Min Max Mean ---------------------------------------------------------------------------------------------------------------- test_transfer_performance[1000-500] 78.0780 (1.0) 98.9794 (1.0) 88.3041 (1.0) test_transfer_performance[1000-5000] 103.9944 (1.33) 113.6821 (1.15) 108.2724 (1.23) test_task_forward[1000_tasks] 320.1597 (4.10) 377.1385 (3.81) 351.0253 (3.98) test_transfer_performance[10000-500] 705.7309 (9.04) 803.4605 (8.12) 760.8389 (8.62) test_transfer_performance[10000-5000] 722.6228 (9.26) 795.0024 (8.03) 765.3734 (8.67) test_task_forward[10000_tasks] 2,942.9156 (37.69) 3,326.9708 (33.61) 3,073.8600 (34.81) test_task_submission[100_tasks] 22,535.5087 (288.63) 22,826.6716 (230.62) 22,624.1203 (256.21) test_task_submission[1000_tasks] 23,234.5893 (297.58) 24,211.8219 (244.61) 23,567.0319 (266.88) test_task_submission[10000_tasks] 23,285.6245 (298.24) 23,874.6958 (241.21) 23,551.8852 (266.71) test_task_submission[10_tasks] 29,259.1614 (374.74) 30,245.1633 (305.57) 29,756.6330 (336.98) test_task_submission[1_tasks] 112,663.9791 (>1000.0) 135,063.5094 (>1000.0) 129,188.0302 (>1000.0) test_task_submission[100000_tasks] 149,491.9673 (>1000.0) 201,440.6912 (>1000.0) 161,452.0020 (>1000.0) ---------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean ========================= 12 passed in 4761.13 seconds ========================= + pushd /ray/ci/jenkins_tests/perf_integration_tests/../../../python + python -m pip install -e . /ray/python /ray/ci/jenkins_tests/perf_integration_tests /ray Obtaining file:///ray/python Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.16.4) Requirement already satisfied: filelock in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.0.4) Requirement already satisfied: funcsigs in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.0.2) Requirement already satisfied: click in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (6.7) Requirement already satisfied: colorama in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (0.3.9) Requirement already satisfied: pytest in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (5.0.1) Requirement already satisfied: pyyaml in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.12) Requirement already satisfied: redis in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.2.1) Requirement already satisfied: six>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.11.0) Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.9.0) Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.18) Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.1.7) Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (18.1.0) Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.5.3) Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (17.1) Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (4.1.0) Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.12.0) Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.3.0) Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from protobuf>=3.8.0->ray==0.8.0.dev2) (41.0.1) Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->ray==0.8.0.dev2) (0.5.2) Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->ray==0.8.0.dev2) (2.2.0) distributed 1.21.8 requires dask>=0.17.0, which is not installed. distributed 1.21.8 requires msgpack, which is not installed. blaze 0.11.3 requires dask[dataframe], which is not installed. jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible. Installing collected packages: ray Found existing installation: ray 0.8.0.dev2 Uninstalling ray-0.8.0.dev2: Successfully uninstalled ray-0.8.0.dev2 Running setup.py develop for ray Successfully installed ray-0.8.0.dev2 You are using pip version 10.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. /ray/ci/jenkins_tests/perf_integration_tests /ray + popd + python -m pytest --benchmark-compare --benchmark-min-rounds=10 --benchmark-compare-fail=min:5% '--benchmark-columns=min, max, mean' /ray/ci/jenkins_tests/perf_integration_tests/../../../python/ray/tests/perf_integration_tests/test_perf_integration.py Comparing against benchmarks from: Linux-CPython-3.6-64bit/0001_unversioned_20190713_225612.json ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-5.0.1, py-1.5.3, pluggy-0.12.0 benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=10 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /ray/python plugins: arraydiff-0.2, openfiles-0.3.0, doctestplus-0.1.3, benchmark-3.2.2, remotedata-0.3.1, timeout-1.3.3 collected 12 items ../../../python/ray/tests/perf_integration_tests/test_perf_integration.py . [ 8%] ........... [100%] Computing stats ... Computing stats ... group 1/1 Computing stats ... group 1/1: min Computing stats ... group 1/1: min (1/24) Computing stats ... group 1/1: min (2/24) Computing stats ... group 1/1: min (3/24) Computing stats ... group 1/1: min (4/24) Computing stats ... group 1/1: min (5/24) Computing stats ... group 1/1: min (6/24) Computing stats ... group 1/1: min (7/24) Computing stats ... group 1/1: min (8/24) Computing stats ... group 1/1: min (9/24) Computing stats ... group 1/1: min (10/24) Computing stats ... group 1/1: min (11/24) Computing stats ... group 1/1: min (12/24) Computing stats ... group 1/1: min (13/24) Computing stats ... group 1/1: min (14/24) Computing stats ... group 1/1: min (15/24) Computing stats ... group 1/1: min (16/24) Computing stats ... group 1/1: min (17/24) Computing stats ... group 1/1: min (18/24) Computing stats ... group 1/1: min (19/24) Computing stats ... group 1/1: min (20/24) Computing stats ... group 1/1: min (21/24) Computing stats ... group 1/1: min (22/24) Computing stats ... group 1/1: min (23/24) Computing stats ... group 1/1: min (24/24) Computing stats ... group 1/1: min (1/24) Computing stats ... group 1/1: min (2/24) Computing stats ... group 1/1: min (3/24) Computing stats ... group 1/1: min (4/24) Computing stats ... group 1/1: min (5/24) Computing stats ... group 1/1: min (6/24) Computing stats ... group 1/1: min (7/24) Computing stats ... group 1/1: min (8/24) Computing stats ... group 1/1: min (9/24) Computing stats ... group 1/1: min (10/24) Computing stats ... group 1/1: min (11/24) Computing stats ... group 1/1: min (12/24) Computing stats ... group 1/1: min (13/24) Computing stats ... group 1/1: min (14/24) Computing stats ... group 1/1: min (15/24) Computing stats ... group 1/1: min (16/24) Computing stats ... group 1/1: min (17/24) Computing stats ... group 1/1: min (18/24) Computing stats ... group 1/1: min (19/24) Computing stats ... group 1/1: min (20/24) Computing stats ... group 1/1: min (21/24) Computing stats ... group 1/1: min (22/24) Computing stats ... group 1/1: min (23/24) Computing stats ... group 1/1: min (24/24) Computing stats ... group 1/1: max Computing stats ... group 1/1: max (1/24) Computing stats ... group 1/1: max (2/24) Computing stats ... group 1/1: max (3/24) Computing stats ... group 1/1: max (4/24) Computing stats ... group 1/1: max (5/24) Computing stats ... group 1/1: max (6/24) Computing stats ... group 1/1: max (7/24) Computing stats ... group 1/1: max (8/24) Computing stats ... group 1/1: max (9/24) Computing stats ... group 1/1: max (10/24) Computing stats ... group 1/1: max (11/24) Computing stats ... group 1/1: max (12/24) Computing stats ... group 1/1: max (13/24) Computing stats ... group 1/1: max (14/24) Computing stats ... group 1/1: max (15/24) Computing stats ... group 1/1: max (16/24) Computing stats ... group 1/1: max (17/24) Computing stats ... group 1/1: max (18/24) Computing stats ... group 1/1: max (19/24) Computing stats ... group 1/1: max (20/24) Computing stats ... group 1/1: max (21/24) Computing stats ... group 1/1: max (22/24) Computing stats ... group 1/1: max (23/24) Computing stats ... group 1/1: max (24/24) Computing stats ... group 1/1: max (1/24) Computing stats ... group 1/1: max (2/24) Computing stats ... group 1/1: max (3/24) Computing stats ... group 1/1: max (4/24) Computing stats ... group 1/1: max (5/24) Computing stats ... group 1/1: max (6/24) Computing stats ... group 1/1: max (7/24) Computing stats ... group 1/1: max (8/24) Computing stats ... group 1/1: max (9/24) Computing stats ... group 1/1: max (10/24) Computing stats ... group 1/1: max (11/24) Computing stats ... group 1/1: max (12/24) Computing stats ... group 1/1: max (13/24) Computing stats ... group 1/1: max (14/24) Computing stats ... group 1/1: max (15/24) Computing stats ... group 1/1: max (16/24) Computing stats ... group 1/1: max (17/24) Computing stats ... group 1/1: max (18/24) Computing stats ... group 1/1: max (19/24) Computing stats ... group 1/1: max (20/24) Computing stats ... group 1/1: max (21/24) Computing stats ... group 1/1: max (22/24) Computing stats ... group 1/1: max (23/24) Computing stats ... group 1/1: max (24/24) Computing stats ... group 1/1: mean Computing stats ... group 1/1: mean (1/24) Computing stats ... group 1/1: mean (2/24) Computing stats ... group 1/1: mean (3/24) Computing stats ... group 1/1: mean (4/24) Computing stats ... group 1/1: mean (5/24) Computing stats ... group 1/1: mean (6/24) Computing stats ... group 1/1: mean (7/24) Computing stats ... group 1/1: mean (8/24) Computing stats ... group 1/1: mean (9/24) Computing stats ... group 1/1: mean (10/24) Computing stats ... group 1/1: mean (11/24) Computing stats ... group 1/1: mean (12/24) Computing stats ... group 1/1: mean (13/24) Computing stats ... group 1/1: mean (14/24) Computing stats ... group 1/1: mean (15/24) Computing stats ... group 1/1: mean (16/24) Computing stats ... group 1/1: mean (17/24) Computing stats ... group 1/1: mean (18/24) Computing stats ... group 1/1: mean (19/24) Computing stats ... group 1/1: mean (20/24) Computing stats ... group 1/1: mean (21/24) Computing stats ... group 1/1: mean (22/24) Computing stats ... group 1/1: mean (23/24) Computing stats ... group 1/1: mean (24/24) Computing stats ... group 1/1: mean (1/24) Computing stats ... group 1/1: mean (2/24) Computing stats ... group 1/1: mean (3/24) Computing stats ... group 1/1: mean (4/24) Computing stats ... group 1/1: mean (5/24) Computing stats ... group 1/1: mean (6/24) Computing stats ... group 1/1: mean (7/24) Computing stats ... group 1/1: mean (8/24) Computing stats ... group 1/1: mean (9/24) Computing stats ... group 1/1: mean (10/24) Computing stats ... group 1/1: mean (11/24) Computing stats ... group 1/1: mean (12/24) Computing stats ... group 1/1: mean (13/24) Computing stats ... group 1/1: mean (14/24) Computing stats ... group 1/1: mean (15/24) Computing stats ... group 1/1: mean (16/24) Computing stats ... group 1/1: mean (17/24) Computing stats ... group 1/1: mean (18/24) Computing stats ... group 1/1: mean (19/24) Computing stats ... group 1/1: mean (20/24) Computing stats ... group 1/1: mean (21/24) Computing stats ... group 1/1: mean (22/24) Computing stats ... group 1/1: mean (23/24) Computing stats ... group 1/1: mean (24/24) Computing stats ... group 1/1: median Computing stats ... group 1/1: median (1/24) Computing stats ... group 1/1: median (2/24) Computing stats ... group 1/1: median (3/24) Computing stats ... group 1/1: median (4/24) Computing stats ... group 1/1: median (5/24) Computing stats ... group 1/1: median (6/24) Computing stats ... group 1/1: median (7/24) Computing stats ... group 1/1: median (8/24) Computing stats ... group 1/1: median (9/24) Computing stats ... group 1/1: median (10/24) Computing stats ... group 1/1: median (11/24) Computing stats ... group 1/1: median (12/24) Computing stats ... group 1/1: median (13/24) Computing stats ... group 1/1: median (14/24) Computing stats ... group 1/1: median (15/24) Computing stats ... group 1/1: median (16/24) Computing stats ... group 1/1: median (17/24) Computing stats ... group 1/1: median (18/24) Computing stats ... group 1/1: median (19/24) Computing stats ... group 1/1: median (20/24) Computing stats ... group 1/1: median (21/24) Computing stats ... group 1/1: median (22/24) Computing stats ... group 1/1: median (23/24) Computing stats ... group 1/1: median (24/24) Computing stats ... group 1/1: median (1/24) Computing stats ... group 1/1: median (2/24) Computing stats ... group 1/1: median (3/24) Computing stats ... group 1/1: median (4/24) Computing stats ... group 1/1: median (5/24) Computing stats ... group 1/1: median (6/24) Computing stats ... group 1/1: median (7/24) Computing stats ... group 1/1: median (8/24) Computing stats ... group 1/1: median (9/24) Computing stats ... group 1/1: median (10/24) Computing stats ... group 1/1: median (11/24) Computing stats ... group 1/1: median (12/24) Computing stats ... group 1/1: median (13/24) Computing stats ... group 1/1: median (14/24) Computing stats ... group 1/1: median (15/24) Computing stats ... group 1/1: median (16/24) Computing stats ... group 1/1: median (17/24) Computing stats ... group 1/1: median (18/24) Computing stats ... group 1/1: median (19/24) Computing stats ... group 1/1: median (20/24) Computing stats ... group 1/1: median (21/24) Computing stats ... group 1/1: median (22/24) Computing stats ... group 1/1: median (23/24) Computing stats ... group 1/1: median (24/24) Computing stats ... group 1/1: iqr Computing stats ... group 1/1: iqr (1/24) Computing stats ... group 1/1: iqr (2/24) Computing stats ... group 1/1: iqr (3/24) Computing stats ... group 1/1: iqr (4/24) Computing stats ... group 1/1: iqr (5/24) Computing stats ... group 1/1: iqr (6/24) Computing stats ... group 1/1: iqr (7/24) Computing stats ... group 1/1: iqr (8/24) Computing stats ... group 1/1: iqr (9/24) Computing stats ... group 1/1: iqr (10/24) Computing stats ... group 1/1: iqr (11/24) Computing stats ... group 1/1: iqr (12/24) Computing stats ... group 1/1: iqr (13/24) Computing stats ... group 1/1: iqr (14/24) Computing stats ... group 1/1: iqr (15/24) Computing stats ... group 1/1: iqr (16/24) Computing stats ... group 1/1: iqr (17/24) Computing stats ... group 1/1: iqr (18/24) Computing stats ... group 1/1: iqr (19/24) Computing stats ... group 1/1: iqr (20/24) Computing stats ... group 1/1: iqr (21/24) Computing stats ... group 1/1: iqr (22/24) Computing stats ... group 1/1: iqr (23/24) Computing stats ... group 1/1: iqr (24/24) Computing stats ... group 1/1: iqr (1/24) Computing stats ... group 1/1: iqr (2/24) Computing stats ... group 1/1: iqr (3/24) Computing stats ... group 1/1: iqr (4/24) Computing stats ... group 1/1: iqr (5/24) Computing stats ... group 1/1: iqr (6/24) Computing stats ... group 1/1: iqr (7/24) Computing stats ... group 1/1: iqr (8/24) Computing stats ... group 1/1: iqr (9/24) Computing stats ... group 1/1: iqr (10/24) Computing stats ... group 1/1: iqr (11/24) Computing stats ... group 1/1: iqr (12/24) Computing stats ... group 1/1: iqr (13/24) Computing stats ... group 1/1: iqr (14/24) Computing stats ... group 1/1: iqr (15/24) Computing stats ... group 1/1: iqr (16/24) Computing stats ... group 1/1: iqr (17/24) Computing stats ... group 1/1: iqr (18/24) Computing stats ... group 1/1: iqr (19/24) Computing stats ... group 1/1: iqr (20/24) Computing stats ... group 1/1: iqr (21/24) Computing stats ... group 1/1: iqr (22/24) Computing stats ... group 1/1: iqr (23/24) Computing stats ... group 1/1: iqr (24/24) Computing stats ... group 1/1: stddev Computing stats ... group 1/1: stddev (1/24) Computing stats ... group 1/1: stddev (2/24) Computing stats ... group 1/1: stddev (3/24) Computing stats ... group 1/1: stddev (4/24) Computing stats ... group 1/1: stddev (5/24) Computing stats ... group 1/1: stddev (6/24) Computing stats ... group 1/1: stddev (7/24) Computing stats ... group 1/1: stddev (8/24) Computing stats ... group 1/1: stddev (9/24) Computing stats ... group 1/1: stddev (10/24) Computing stats ... group 1/1: stddev (11/24) Computing stats ... group 1/1: stddev (12/24) Computing stats ... group 1/1: stddev (13/24) Computing stats ... group 1/1: stddev (14/24) Computing stats ... group 1/1: stddev (15/24) Computing stats ... group 1/1: stddev (16/24) Computing stats ... group 1/1: stddev (17/24) Computing stats ... group 1/1: stddev (18/24) Computing stats ... group 1/1: stddev (19/24) Computing stats ... group 1/1: stddev (20/24) Computing stats ... group 1/1: stddev (21/24) Computing stats ... group 1/1: stddev (22/24) Computing stats ... group 1/1: stddev (23/24) Computing stats ... group 1/1: stddev (24/24) Computing stats ... group 1/1: stddev (1/24) Computing stats ... group 1/1: stddev (2/24) Computing stats ... group 1/1: stddev (3/24) Computing stats ... group 1/1: stddev (4/24) Computing stats ... group 1/1: stddev (5/24) Computing stats ... group 1/1: stddev (6/24) Computing stats ... group 1/1: stddev (7/24) Computing stats ... group 1/1: stddev (8/24) Computing stats ... group 1/1: stddev (9/24) Computing stats ... group 1/1: stddev (10/24) Computing stats ... group 1/1: stddev (11/24) Computing stats ... group 1/1: stddev (12/24) Computing stats ... group 1/1: stddev (13/24) Computing stats ... group 1/1: stddev (14/24) Computing stats ... group 1/1: stddev (15/24) Computing stats ... group 1/1: stddev (16/24) Computing stats ... group 1/1: stddev (17/24) Computing stats ... group 1/1: stddev (18/24) Computing stats ... group 1/1: stddev (19/24) Computing stats ... group 1/1: stddev (20/24) Computing stats ... group 1/1: stddev (21/24) Computing stats ... group 1/1: stddev (22/24) Computing stats ... group 1/1: stddev (23/24) Computing stats ... group 1/1: stddev (24/24) Computing stats ... group 1/1: ops Computing stats ... group 1/1: ops (1/24) Computing stats ... group 1/1: ops (2/24) Computing stats ... group 1/1: ops (3/24) Computing stats ... group 1/1: ops (4/24) Computing stats ... group 1/1: ops (5/24) Computing stats ... group 1/1: ops (6/24) Computing stats ... group 1/1: ops (7/24) Computing stats ... group 1/1: ops (8/24) Computing stats ... group 1/1: ops (9/24) Computing stats ... group 1/1: ops (10/24) Computing stats ... group 1/1: ops (11/24) Computing stats ... group 1/1: ops (12/24) Computing stats ... group 1/1: ops (13/24) Computing stats ... group 1/1: ops (14/24) Computing stats ... group 1/1: ops (15/24) Computing stats ... group 1/1: ops (16/24) Computing stats ... group 1/1: ops (17/24) Computing stats ... group 1/1: ops (18/24) Computing stats ... group 1/1: ops (19/24) Computing stats ... group 1/1: ops (20/24) Computing stats ... group 1/1: ops (21/24) Computing stats ... group 1/1: ops (22/24) Computing stats ... group 1/1: ops (23/24) Computing stats ... group 1/1: ops (24/24) Computing stats ... group 1/1: ops (1/24) Computing stats ... group 1/1: ops (2/24) Computing stats ... group 1/1: ops (3/24) Computing stats ... group 1/1: ops (4/24) Computing stats ... group 1/1: ops (5/24) Computing stats ... group 1/1: ops (6/24) Computing stats ... group 1/1: ops (7/24) Computing stats ... group 1/1: ops (8/24) Computing stats ... group 1/1: ops (9/24) Computing stats ... group 1/1: ops (10/24) Computing stats ... group 1/1: ops (11/24) Computing stats ... group 1/1: ops (12/24) Computing stats ... group 1/1: ops (13/24) Computing stats ... group 1/1: ops (14/24) Computing stats ... group 1/1: ops (15/24) Computing stats ... group 1/1: ops (16/24) Computing stats ... group 1/1: ops (17/24) Computing stats ... group 1/1: ops (18/24) Computing stats ... group 1/1: ops (19/24) Computing stats ... group 1/1: ops (20/24) Computing stats ... group 1/1: ops (21/24) Computing stats ... group 1/1: ops (22/24) Computing stats ... group 1/1: ops (23/24) Computing stats ... group 1/1: ops (24/24) Computing stats ... group 1/1: ops: outliers Computing stats ... group 1/1: ops: outliers (1/24) Computing stats ... group 1/1: ops: outliers (2/24) Computing stats ... group 1/1: ops: outliers (3/24) Computing stats ... group 1/1: ops: outliers (4/24) Computing stats ... group 1/1: ops: outliers (5/24) Computing stats ... group 1/1: ops: outliers (6/24) Computing stats ... group 1/1: ops: outliers (7/24) Computing stats ... group 1/1: ops: outliers (8/24) Computing stats ... group 1/1: ops: outliers (9/24) Computing stats ... group 1/1: ops: outliers (10/24) Computing stats ... group 1/1: ops: outliers (11/24) Computing stats ... group 1/1: ops: outliers (12/24) Computing stats ... group 1/1: ops: outliers (13/24) Computing stats ... group 1/1: ops: outliers (14/24) Computing stats ... group 1/1: ops: outliers (15/24) Computing stats ... group 1/1: ops: outliers (16/24) Computing stats ... group 1/1: ops: outliers (17/24) Computing stats ... group 1/1: ops: outliers (18/24) Computing stats ... group 1/1: ops: outliers (19/24) Computing stats ... group 1/1: ops: outliers (20/24) Computing stats ... group 1/1: ops: outliers (21/24) Computing stats ... group 1/1: ops: outliers (22/24) Computing stats ... group 1/1: ops: outliers (23/24) Computing stats ... group 1/1: ops: outliers (24/24) Computing stats ... group 1/1: ops: rounds Computing stats ... group 1/1: ops: rounds (1/24) Computing stats ... group 1/1: ops: rounds (2/24) Computing stats ... group 1/1: ops: rounds (3/24) Computing stats ... group 1/1: ops: rounds (4/24) Computing stats ... group 1/1: ops: rounds (5/24) Computing stats ... group 1/1: ops: rounds (6/24) Computing stats ... group 1/1: ops: rounds (7/24) Computing stats ... group 1/1: ops: rounds (8/24) Computing stats ... group 1/1: ops: rounds (9/24) Computing stats ... group 1/1: ops: rounds (10/24) Computing stats ... group 1/1: ops: rounds (11/24) Computing stats ... group 1/1: ops: rounds (12/24) Computing stats ... group 1/1: ops: rounds (13/24) Computing stats ... group 1/1: ops: rounds (14/24) Computing stats ... group 1/1: ops: rounds (15/24) Computing stats ... group 1/1: ops: rounds (16/24) Computing stats ... group 1/1: ops: rounds (17/24) Computing stats ... group 1/1: ops: rounds (18/24) Computing stats ... group 1/1: ops: rounds (19/24) Computing stats ... group 1/1: ops: rounds (20/24) Computing stats ... group 1/1: ops: rounds (21/24) Computing stats ... group 1/1: ops: rounds (22/24) Computing stats ... group 1/1: ops: rounds (23/24) Computing stats ... group 1/1: ops: rounds (24/24) Computing stats ... group 1/1: ops: iterations Computing stats ... group 1/1: ops: iterations (1/24) Computing stats ... group 1/1: ops: iterations (2/24) Computing stats ... group 1/1: ops: iterations (3/24) Computing stats ... group 1/1: ops: iterations (4/24) Computing stats ... group 1/1: ops: iterations (5/24) Computing stats ... group 1/1: ops: iterations (6/24) Computing stats ... group 1/1: ops: iterations (7/24) Computing stats ... group 1/1: ops: iterations (8/24) Computing stats ... group 1/1: ops: iterations (9/24) Computing stats ... group 1/1: ops: iterations (10/24) Computing stats ... group 1/1: ops: iterations (11/24) Computing stats ... group 1/1: ops: iterations (12/24) Computing stats ... group 1/1: ops: iterations (13/24) Computing stats ... group 1/1: ops: iterations (14/24) Computing stats ... group 1/1: ops: iterations (15/24) Computing stats ... group 1/1: ops: iterations (16/24) Computing stats ... group 1/1: ops: iterations (17/24) Computing stats ... group 1/1: ops: iterations (18/24) Computing stats ... group 1/1: ops: iterations (19/24) Computing stats ... group 1/1: ops: iterations (20/24) Computing stats ... group 1/1: ops: iterations (21/24) Computing stats ... group 1/1: ops: iterations (22/24) Computing stats ... group 1/1: ops: iterations (23/24) Computing stats ... group 1/1: ops: iterations (24/24) ----------------------------------------------------- benchmark: 24 tests ----------------------------------------------------- Name (time in ms) Min Max Mean ------------------------------------------------------------------------------------------------------------------------------- test_transfer_performance[1000-500] (0001_unversi) 78.0780 (1.0) 98.9794 (1.0) 88.3041 (1.0) test_transfer_performance[1000-5000] (NOW) 85.9641 (1.10) 104.3217 (1.05) 90.1999 (1.02) test_transfer_performance[1000-500] (NOW) 92.5152 (1.18) 111.6823 (1.13) 102.0793 (1.16) test_transfer_performance[1000-5000] (0001_unversi) 103.9944 (1.33) 113.6821 (1.15) 108.2724 (1.23) test_task_forward[1000_tasks] (NOW) 317.3376 (4.06) 373.0135 (3.77) 346.7846 (3.93) test_task_forward[1000_tasks] (0001_unversi) 320.1597 (4.10) 377.1385 (3.81) 351.0253 (3.98) test_transfer_performance[10000-500] (NOW) 694.1766 (8.89) 841.4052 (8.50) 727.0985 (8.23) test_transfer_performance[10000-500] (0001_unversi) 705.7309 (9.04) 803.4605 (8.12) 760.8389 (8.62) test_transfer_performance[10000-5000] (0001_unversi) 722.6228 (9.26) 795.0024 (8.03) 765.3734 (8.67) test_transfer_performance[10000-5000] (NOW) 752.1134 (9.63) 881.5791 (8.91) 803.0683 (9.09) test_task_forward[10000_tasks] (0001_unversi) 2,942.9156 (37.69) 3,326.9708 (33.61) 3,073.8600 (34.81) test_task_forward[10000_tasks] (NOW) 3,011.2267 (38.57) 3,236.2038 (32.70) 3,121.9882 (35.35) test_task_submission[100_tasks] (0001_unversi) 22,535.5087 (288.63) 22,826.6716 (230.62) 22,624.1203 (256.21) test_task_submission[1000_tasks] (NOW) 22,850.2988 (292.66) 23,291.3857 (235.32) 23,062.9762 (261.18) test_task_submission[100_tasks] (NOW) 22,871.3187 (292.93) 23,836.5403 (240.82) 23,100.9997 (261.61) test_task_submission[10000_tasks] (NOW) 22,893.0700 (293.21) 24,265.0747 (245.15) 23,770.1878 (269.19) test_task_submission[1000_tasks] (0001_unversi) 23,234.5893 (297.58) 24,211.8219 (244.61) 23,567.0319 (266.88) test_task_submission[10000_tasks] (0001_unversi) 23,285.6245 (298.24) 23,874.6958 (241.21) 23,551.8852 (266.71) test_task_submission[10_tasks] (0001_unversi) 29,259.1614 (374.74) 30,245.1633 (305.57) 29,756.6330 (336.98) test_task_submission[10_tasks] (NOW) 29,619.5004 (379.36) 30,295.1524 (306.08) 29,917.7489 (338.80) test_task_submission[1_tasks] (NOW) 106,572.6352 (>1000.0) 132,014.3307 (>1000.0) 119,434.8921 (>1000.0) test_task_submission[1_tasks] (0001_unversi) 112,663.9791 (>1000.0) 135,063.5094 (>1000.0) 129,188.0302 (>1000.0) test_task_submission[100000_tasks] (NOW) 148,312.0674 (>1000.0) 222,341.2600 (>1000.0) 158,140.3462 (>1000.0) test_task_submission[100000_tasks] (0001_unversi) 149,491.9673 (>1000.0) 201,440.6912 (>1000.0) 161,452.0020 (>1000.0) ------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean -------------------------------------------------------------------------------- Performance has regressed: test_transfer_performance[1000-500] (0001_unversi) - Field 'min' has failed PercentageRegressionCheck: 18.490819441 > 5.000000000 -------------------------------------------------------------------------------- ERROR: Performance has regressed. Build step 'Execute shell' marked build as failure An attempt to send an e-mail to empty list of recipients, ignored. Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1662/ Test FAILed. Finished: FAILURE