SuccessConsole Output

GitHub pull request #5118 of commit fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7.
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-testing (ubuntu-testing-gpu) in workspace /home/jenkins/workspace/Ray-Perf-Integration-PRB
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Cloning the remote Git repository
Cloning repository https://github.com/ray-project/ray.git
 > git init /home/jenkins/workspace/Ray-Perf-Integration-PRB # timeout=10
Fetching upstream changes from https://github.com/ray-project/ray.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10
Fetching upstream changes from https://github.com/ray-project/ray.git
 > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7^{commit} # timeout=10
 > git branch -a -v --no-abbrev --contains fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7 # timeout=10
Checking out Revision fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7 (origin/pr/5118/head)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7
 > git rev-list fb7f8a771d518b32f9372b646ac4c5ec5e58d3c2 # timeout=10
[Ray-Perf-Integration-PRB] $ /bin/bash /tmp/hudson595048478236515104.sh
+ [[ 2 -gt 0 ]]
+ key=--output-sha
+ case $key in
+ OUTPUT_SHA=YES
+ shift
+ [[ 1 -gt 0 ]]
+ key=--no-cache
+ case $key in
+ NO_CACHE=--no-cache
+ shift
+ [[ 0 -gt 0 ]]
+ '[' YES ']'
++ docker build --no-cache -q -t ray-project/base-deps docker/base-deps
+ IMAGE_SHA=sha256:d92b2d4aa601d4269674c9be1c7760e841abf9155e17118742cfd056a69e9a4a
+ git rev-parse HEAD
++ git rev-parse HEAD
+ git archive -o ./docker/deploy/ray.tar fb11e4e4eb1fdf247dc348fcf68fa36890ce4bf7
+ '[' YES ']'
++ docker build --no-cache -q -t ray-project/deploy docker/deploy
+ IMAGE_SHA=sha256:7dd024cbcafcf72c3441de709cfd8d66470b4390f696e0ee3675e25b235f956a
+ rm ./docker/deploy/ray.tar ./docker/deploy/git-rev
+ '[' '!' ']'
+ '[' YES ']'
++ docker build --no-cache -q -t ray-project/examples docker/examples
+ IMAGE_SHA=sha256:2e7ad948706a9c785237ac2b3d6580337eff4de9af34c77cc6b3efca67b995c3
+ '[' YES ']'
+ echo sha256:2e7ad948706a9c785237ac2b3d6580337eff4de9af34c77cc6b3efca67b995c3
+ sed s/sha256://
Using Docker image 2e7ad948706a9c785237ac2b3d6580337eff4de9af34c77cc6b3efca67b995c3
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
+++ dirname ./ci/jenkins_tests/perf_integration_tests/run_perf_integration.sh
++ cd ./ci/jenkins_tests/perf_integration_tests
++ pwd
+ ROOT_DIR=/ray/ci/jenkins_tests/perf_integration_tests
+ pushd /ray/ci/jenkins_tests/perf_integration_tests
+ python -m pip install pytest-benchmark
/ray/ci/jenkins_tests/perf_integration_tests /ray
Collecting pytest-benchmark
  Downloading https://files.pythonhosted.org/packages/ab/0f/8488bebc18924bbf5ed9eff3d3cecd0287446c88962f10e99fa727a3d2d7/pytest_benchmark-3.2.2-py2.py3-none-any.whl (48kB)
Requirement already satisfied: pytest>=3.8 in /opt/conda/lib/python3.6/site-packages (from pytest-benchmark) (5.0.1)
Collecting py-cpuinfo (from pytest-benchmark)
  Downloading https://files.pythonhosted.org/packages/42/60/63f28a5401da733043abe7053e7d9591491b4784c4f87c339bf51215aa0a/py-cpuinfo-5.0.0.tar.gz (82kB)
Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.18)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (17.1)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (4.1.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.12.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (18.1.0)
Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (1.3.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (0.1.7)
Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest>=3.8->pytest-benchmark) (1.5.3)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest>=3.8->pytest-benchmark) (0.5.2)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest>=3.8->pytest-benchmark) (2.2.0)
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from packaging->pytest>=3.8->pytest-benchmark) (1.11.0)
Building wheels for collected packages: py-cpuinfo
  Running setup.py bdist_wheel for py-cpuinfo: started
  Running setup.py bdist_wheel for py-cpuinfo: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/01/7e/a9/b982d0fea22b7e4ae5619de949570cde5ad55420cec16e86a5
Successfully built py-cpuinfo
distributed 1.21.8 requires dask>=0.17.0, which is not installed.
distributed 1.21.8 requires msgpack, which is not installed.
blaze 0.11.3 requires dask[dataframe], which is not installed.
jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible.
Installing collected packages: py-cpuinfo, pytest-benchmark
Successfully installed py-cpuinfo-5.0.0 pytest-benchmark-3.2.2
You are using pip version 10.0.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ pip install -U https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl
Collecting ray==0.8.0.dev2 from https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl
  Downloading https://ray-wheels.s3-us-west-2.amazonaws.com/latest/ray-0.8.0.dev2-cp36-cp36m-manylinux1_x86_64.whl (48.4MB)
Requirement not upgraded as not directly required: funcsigs in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.0.2)
Requirement not upgraded as not directly required: pyyaml in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.12)
Requirement not upgraded as not directly required: filelock in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.0.4)
Requirement not upgraded as not directly required: redis in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.2.1)
Requirement not upgraded as not directly required: six>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.11.0)
Requirement not upgraded as not directly required: numpy>=1.14 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.16.4)
Requirement not upgraded as not directly required: pytest in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (5.0.1)
Requirement not upgraded as not directly required: click in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (6.7)
Requirement not upgraded as not directly required: colorama in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (0.3.9)
Requirement not upgraded as not directly required: protobuf>=3.8.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.9.0)
Requirement not upgraded as not directly required: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.5.3)
Requirement not upgraded as not directly required: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.1.7)
Requirement not upgraded as not directly required: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (18.1.0)
Requirement not upgraded as not directly required: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (17.1)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.12.0)
Requirement not upgraded as not directly required: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (4.1.0)
Requirement not upgraded as not directly required: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.18)
Requirement not upgraded as not directly required: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.3.0)
Requirement not upgraded as not directly required: setuptools in /opt/conda/lib/python3.6/site-packages (from protobuf>=3.8.0->ray==0.8.0.dev2) (41.0.1)
Requirement not upgraded as not directly required: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->ray==0.8.0.dev2) (2.2.0)
Requirement not upgraded as not directly required: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->ray==0.8.0.dev2) (0.5.2)
distributed 1.21.8 requires dask>=0.17.0, which is not installed.
distributed 1.21.8 requires msgpack, which is not installed.
blaze 0.11.3 requires dask[dataframe], which is not installed.
jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible.
Installing collected packages: ray
  Found existing installation: ray 0.8.0.dev2
    Uninstalling ray-0.8.0.dev2:
      Successfully uninstalled ray-0.8.0.dev2
Successfully installed ray-0.8.0.dev2
You are using pip version 10.0.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ python -m pytest --benchmark-autosave --benchmark-min-rounds=10 '--benchmark-columns=min, max, mean' /ray/ci/jenkins_tests/perf_integration_tests/../../../python/ray/tests/perf_integration_tests/test_perf_integration.py
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.5.3, pluggy-0.12.0
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=10 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /ray/python
plugins: arraydiff-0.2, openfiles-0.3.0, doctestplus-0.1.3, benchmark-3.2.2, remotedata-0.3.2, timeout-1.3.3
collected 12 items

../../../python/ray/tests/perf_integration_tests/test_perf_integration.py . [  8%]
...........                                                              [100%]
Saved benchmark data in: /ray/ci/jenkins_tests/perf_integration_tests/.benchmarks/Linux-CPython-3.6-64bit/0001_unversioned_20190723_211215.json




Computing stats ...
Computing stats ... group 1/1
Computing stats ... group 1/1: min
Computing stats ... group 1/1: min (1/12)
Computing stats ... group 1/1: min (2/12)
Computing stats ... group 1/1: min (3/12)
Computing stats ... group 1/1: min (4/12)
Computing stats ... group 1/1: min (5/12)
Computing stats ... group 1/1: min (6/12)
Computing stats ... group 1/1: min (7/12)
Computing stats ... group 1/1: min (8/12)
Computing stats ... group 1/1: min (9/12)
Computing stats ... group 1/1: min (10/12)
Computing stats ... group 1/1: min (11/12)
Computing stats ... group 1/1: min (12/12)
Computing stats ... group 1/1: min (1/12)
Computing stats ... group 1/1: min (2/12)
Computing stats ... group 1/1: min (3/12)
Computing stats ... group 1/1: min (4/12)
Computing stats ... group 1/1: min (5/12)
Computing stats ... group 1/1: min (6/12)
Computing stats ... group 1/1: min (7/12)
Computing stats ... group 1/1: min (8/12)
Computing stats ... group 1/1: min (9/12)
Computing stats ... group 1/1: min (10/12)
Computing stats ... group 1/1: min (11/12)
Computing stats ... group 1/1: min (12/12)
Computing stats ... group 1/1: max
Computing stats ... group 1/1: max (1/12)
Computing stats ... group 1/1: max (2/12)
Computing stats ... group 1/1: max (3/12)
Computing stats ... group 1/1: max (4/12)
Computing stats ... group 1/1: max (5/12)
Computing stats ... group 1/1: max (6/12)
Computing stats ... group 1/1: max (7/12)
Computing stats ... group 1/1: max (8/12)
Computing stats ... group 1/1: max (9/12)
Computing stats ... group 1/1: max (10/12)
Computing stats ... group 1/1: max (11/12)
Computing stats ... group 1/1: max (12/12)
Computing stats ... group 1/1: max (1/12)
Computing stats ... group 1/1: max (2/12)
Computing stats ... group 1/1: max (3/12)
Computing stats ... group 1/1: max (4/12)
Computing stats ... group 1/1: max (5/12)
Computing stats ... group 1/1: max (6/12)
Computing stats ... group 1/1: max (7/12)
Computing stats ... group 1/1: max (8/12)
Computing stats ... group 1/1: max (9/12)
Computing stats ... group 1/1: max (10/12)
Computing stats ... group 1/1: max (11/12)
Computing stats ... group 1/1: max (12/12)
Computing stats ... group 1/1: mean
Computing stats ... group 1/1: mean (1/12)
Computing stats ... group 1/1: mean (2/12)
Computing stats ... group 1/1: mean (3/12)
Computing stats ... group 1/1: mean (4/12)
Computing stats ... group 1/1: mean (5/12)
Computing stats ... group 1/1: mean (6/12)
Computing stats ... group 1/1: mean (7/12)
Computing stats ... group 1/1: mean (8/12)
Computing stats ... group 1/1: mean (9/12)
Computing stats ... group 1/1: mean (10/12)
Computing stats ... group 1/1: mean (11/12)
Computing stats ... group 1/1: mean (12/12)
Computing stats ... group 1/1: mean (1/12)
Computing stats ... group 1/1: mean (2/12)
Computing stats ... group 1/1: mean (3/12)
Computing stats ... group 1/1: mean (4/12)
Computing stats ... group 1/1: mean (5/12)
Computing stats ... group 1/1: mean (6/12)
Computing stats ... group 1/1: mean (7/12)
Computing stats ... group 1/1: mean (8/12)
Computing stats ... group 1/1: mean (9/12)
Computing stats ... group 1/1: mean (10/12)
Computing stats ... group 1/1: mean (11/12)
Computing stats ... group 1/1: mean (12/12)
Computing stats ... group 1/1: median
Computing stats ... group 1/1: median (1/12)
Computing stats ... group 1/1: median (2/12)
Computing stats ... group 1/1: median (3/12)
Computing stats ... group 1/1: median (4/12)
Computing stats ... group 1/1: median (5/12)
Computing stats ... group 1/1: median (6/12)
Computing stats ... group 1/1: median (7/12)
Computing stats ... group 1/1: median (8/12)
Computing stats ... group 1/1: median (9/12)
Computing stats ... group 1/1: median (10/12)
Computing stats ... group 1/1: median (11/12)
Computing stats ... group 1/1: median (12/12)
Computing stats ... group 1/1: median (1/12)
Computing stats ... group 1/1: median (2/12)
Computing stats ... group 1/1: median (3/12)
Computing stats ... group 1/1: median (4/12)
Computing stats ... group 1/1: median (5/12)
Computing stats ... group 1/1: median (6/12)
Computing stats ... group 1/1: median (7/12)
Computing stats ... group 1/1: median (8/12)
Computing stats ... group 1/1: median (9/12)
Computing stats ... group 1/1: median (10/12)
Computing stats ... group 1/1: median (11/12)
Computing stats ... group 1/1: median (12/12)
Computing stats ... group 1/1: iqr
Computing stats ... group 1/1: iqr (1/12)
Computing stats ... group 1/1: iqr (2/12)
Computing stats ... group 1/1: iqr (3/12)
Computing stats ... group 1/1: iqr (4/12)
Computing stats ... group 1/1: iqr (5/12)
Computing stats ... group 1/1: iqr (6/12)
Computing stats ... group 1/1: iqr (7/12)
Computing stats ... group 1/1: iqr (8/12)
Computing stats ... group 1/1: iqr (9/12)
Computing stats ... group 1/1: iqr (10/12)
Computing stats ... group 1/1: iqr (11/12)
Computing stats ... group 1/1: iqr (12/12)
Computing stats ... group 1/1: iqr (1/12)
Computing stats ... group 1/1: iqr (2/12)
Computing stats ... group 1/1: iqr (3/12)
Computing stats ... group 1/1: iqr (4/12)
Computing stats ... group 1/1: iqr (5/12)
Computing stats ... group 1/1: iqr (6/12)
Computing stats ... group 1/1: iqr (7/12)
Computing stats ... group 1/1: iqr (8/12)
Computing stats ... group 1/1: iqr (9/12)
Computing stats ... group 1/1: iqr (10/12)
Computing stats ... group 1/1: iqr (11/12)
Computing stats ... group 1/1: iqr (12/12)
Computing stats ... group 1/1: stddev
Computing stats ... group 1/1: stddev (1/12)
Computing stats ... group 1/1: stddev (2/12)
Computing stats ... group 1/1: stddev (3/12)
Computing stats ... group 1/1: stddev (4/12)
Computing stats ... group 1/1: stddev (5/12)
Computing stats ... group 1/1: stddev (6/12)
Computing stats ... group 1/1: stddev (7/12)
Computing stats ... group 1/1: stddev (8/12)
Computing stats ... group 1/1: stddev (9/12)
Computing stats ... group 1/1: stddev (10/12)
Computing stats ... group 1/1: stddev (11/12)
Computing stats ... group 1/1: stddev (12/12)
Computing stats ... group 1/1: stddev (1/12)
Computing stats ... group 1/1: stddev (2/12)
Computing stats ... group 1/1: stddev (3/12)
Computing stats ... group 1/1: stddev (4/12)
Computing stats ... group 1/1: stddev (5/12)
Computing stats ... group 1/1: stddev (6/12)
Computing stats ... group 1/1: stddev (7/12)
Computing stats ... group 1/1: stddev (8/12)
Computing stats ... group 1/1: stddev (9/12)
Computing stats ... group 1/1: stddev (10/12)
Computing stats ... group 1/1: stddev (11/12)
Computing stats ... group 1/1: stddev (12/12)
Computing stats ... group 1/1: ops
Computing stats ... group 1/1: ops (1/12)
Computing stats ... group 1/1: ops (2/12)
Computing stats ... group 1/1: ops (3/12)
Computing stats ... group 1/1: ops (4/12)
Computing stats ... group 1/1: ops (5/12)
Computing stats ... group 1/1: ops (6/12)
Computing stats ... group 1/1: ops (7/12)
Computing stats ... group 1/1: ops (8/12)
Computing stats ... group 1/1: ops (9/12)
Computing stats ... group 1/1: ops (10/12)
Computing stats ... group 1/1: ops (11/12)
Computing stats ... group 1/1: ops (12/12)
Computing stats ... group 1/1: ops (1/12)
Computing stats ... group 1/1: ops (2/12)
Computing stats ... group 1/1: ops (3/12)
Computing stats ... group 1/1: ops (4/12)
Computing stats ... group 1/1: ops (5/12)
Computing stats ... group 1/1: ops (6/12)
Computing stats ... group 1/1: ops (7/12)
Computing stats ... group 1/1: ops (8/12)
Computing stats ... group 1/1: ops (9/12)
Computing stats ... group 1/1: ops (10/12)
Computing stats ... group 1/1: ops (11/12)
Computing stats ... group 1/1: ops (12/12)
Computing stats ... group 1/1: ops: outliers
Computing stats ... group 1/1: ops: outliers (1/12)
Computing stats ... group 1/1: ops: outliers (2/12)
Computing stats ... group 1/1: ops: outliers (3/12)
Computing stats ... group 1/1: ops: outliers (4/12)
Computing stats ... group 1/1: ops: outliers (5/12)
Computing stats ... group 1/1: ops: outliers (6/12)
Computing stats ... group 1/1: ops: outliers (7/12)
Computing stats ... group 1/1: ops: outliers (8/12)
Computing stats ... group 1/1: ops: outliers (9/12)
Computing stats ... group 1/1: ops: outliers (10/12)
Computing stats ... group 1/1: ops: outliers (11/12)
Computing stats ... group 1/1: ops: outliers (12/12)
Computing stats ... group 1/1: ops: rounds
Computing stats ... group 1/1: ops: rounds (1/12)
Computing stats ... group 1/1: ops: rounds (2/12)
Computing stats ... group 1/1: ops: rounds (3/12)
Computing stats ... group 1/1: ops: rounds (4/12)
Computing stats ... group 1/1: ops: rounds (5/12)
Computing stats ... group 1/1: ops: rounds (6/12)
Computing stats ... group 1/1: ops: rounds (7/12)
Computing stats ... group 1/1: ops: rounds (8/12)
Computing stats ... group 1/1: ops: rounds (9/12)
Computing stats ... group 1/1: ops: rounds (10/12)
Computing stats ... group 1/1: ops: rounds (11/12)
Computing stats ... group 1/1: ops: rounds (12/12)
Computing stats ... group 1/1: ops: iterations
Computing stats ... group 1/1: ops: iterations (1/12)
Computing stats ... group 1/1: ops: iterations (2/12)
Computing stats ... group 1/1: ops: iterations (3/12)
Computing stats ... group 1/1: ops: iterations (4/12)
Computing stats ... group 1/1: ops: iterations (5/12)
Computing stats ... group 1/1: ops: iterations (6/12)
Computing stats ... group 1/1: ops: iterations (7/12)
Computing stats ... group 1/1: ops: iterations (8/12)
Computing stats ... group 1/1: ops: iterations (9/12)
Computing stats ... group 1/1: ops: iterations (10/12)
Computing stats ... group 1/1: ops: iterations (11/12)
Computing stats ... group 1/1: ops: iterations (12/12)
--------------------------------------------- benchmark: 12 tests ----------------------------------------------
Name (time in ms)                                  Min                     Max                    Mean          
----------------------------------------------------------------------------------------------------------------
test_transfer_performance[1000-500]            95.3148 (1.0)          104.4025 (1.0)           99.0258 (1.0)    
test_transfer_performance[1000-5000]          101.5494 (1.07)         128.2741 (1.23)         115.5153 (1.17)   
test_task_forward[1000_tasks]                 304.0597 (3.19)         392.9130 (3.76)         351.5185 (3.55)   
test_transfer_performance[10000-500]          785.5159 (8.24)         885.8352 (8.48)         818.6623 (8.27)   
test_transfer_performance[10000-5000]         800.7206 (8.40)         854.0806 (8.18)         826.4243 (8.35)   
test_task_forward[10000_tasks]              3,003.9293 (31.52)      3,456.9244 (33.11)      3,114.0767 (31.45)  
test_task_submission[100_tasks]            22,842.0423 (239.65)    23,303.9245 (223.21)    23,152.0697 (233.80) 
test_task_submission[10000_tasks]          23,007.7064 (241.39)    24,492.1888 (234.59)    23,609.2221 (238.41) 
test_task_submission[1000_tasks]           23,300.4295 (244.46)    24,048.6048 (230.35)    23,656.6157 (238.89) 
test_task_submission[10_tasks]             29,726.8886 (311.88)    30,618.6358 (293.27)    30,297.6021 (305.96) 
test_task_submission[1_tasks]              99,667.8763 (>1000.0)  129,427.2115 (>1000.0)  111,772.5530 (>1000.0)
test_task_submission[100000_tasks]        146,997.5556 (>1000.0)  151,228.6210 (>1000.0)  149,288.7966 (>1000.0)
----------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
========================= 12 passed in 4477.67 seconds =========================
+ pushd /ray/ci/jenkins_tests/perf_integration_tests/../../../python
+ python -m pip install -e .
/ray/python /ray/ci/jenkins_tests/perf_integration_tests /ray
Obtaining file:///ray/python
Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.16.4)
Requirement already satisfied: filelock in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.0.4)
Requirement already satisfied: funcsigs in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.0.2)
Requirement already satisfied: click in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (6.7)
Requirement already satisfied: colorama in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (0.3.9)
Requirement already satisfied: pytest in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (5.0.1)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.12)
Requirement already satisfied: redis in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.2.1)
Requirement already satisfied: six>=1.0.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (1.11.0)
Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.6/site-packages (from ray==0.8.0.dev2) (3.9.0)
Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.18)
Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.3.0)
Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (1.5.3)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (18.1.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.1.7)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (17.1)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (4.1.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /opt/conda/lib/python3.6/site-packages (from pytest->ray==0.8.0.dev2) (0.12.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from protobuf>=3.8.0->ray==0.8.0.dev2) (41.0.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->ray==0.8.0.dev2) (0.5.2)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->pytest->ray==0.8.0.dev2) (2.2.0)
distributed 1.21.8 requires dask>=0.17.0, which is not installed.
distributed 1.21.8 requires msgpack, which is not installed.
blaze 0.11.3 requires dask[dataframe], which is not installed.
jupyterlab-server 1.0.0 has requirement jsonschema>=3.0.1, but you'll have jsonschema 2.6.0 which is incompatible.
Installing collected packages: ray
  Found existing installation: ray 0.8.0.dev2
    Uninstalling ray-0.8.0.dev2:
      Successfully uninstalled ray-0.8.0.dev2
  Running setup.py develop for ray
Successfully installed ray-0.8.0.dev2
You are using pip version 10.0.1, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ray/ci/jenkins_tests/perf_integration_tests /ray
+ popd
+ python -m pytest --benchmark-compare --benchmark-min-rounds=10 --benchmark-compare-fail=min:5% '--benchmark-columns=min, max, mean' /ray/ci/jenkins_tests/perf_integration_tests/../../../python/ray/tests/perf_integration_tests/test_perf_integration.py
Comparing against benchmarks from: Linux-CPython-3.6-64bit/0001_unversioned_20190723_211215.json
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.5.3, pluggy-0.12.0
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=10 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /ray/python
plugins: arraydiff-0.2, openfiles-0.3.0, doctestplus-0.1.3, benchmark-3.2.2, remotedata-0.3.2, timeout-1.3.3
collected 12 items

../../../python/ray/tests/perf_integration_tests/test_perf_integration.py . [  8%]
...........                                                              [100%]



Computing stats ...
Computing stats ... group 1/1
Computing stats ... group 1/1: min
Computing stats ... group 1/1: min (1/24)
Computing stats ... group 1/1: min (2/24)
Computing stats ... group 1/1: min (3/24)
Computing stats ... group 1/1: min (4/24)
Computing stats ... group 1/1: min (5/24)
Computing stats ... group 1/1: min (6/24)
Computing stats ... group 1/1: min (7/24)
Computing stats ... group 1/1: min (8/24)
Computing stats ... group 1/1: min (9/24)
Computing stats ... group 1/1: min (10/24)
Computing stats ... group 1/1: min (11/24)
Computing stats ... group 1/1: min (12/24)
Computing stats ... group 1/1: min (13/24)
Computing stats ... group 1/1: min (14/24)
Computing stats ... group 1/1: min (15/24)
Computing stats ... group 1/1: min (16/24)
Computing stats ... group 1/1: min (17/24)
Computing stats ... group 1/1: min (18/24)
Computing stats ... group 1/1: min (19/24)
Computing stats ... group 1/1: min (20/24)
Computing stats ... group 1/1: min (21/24)
Computing stats ... group 1/1: min (22/24)
Computing stats ... group 1/1: min (23/24)
Computing stats ... group 1/1: min (24/24)
Computing stats ... group 1/1: min (1/24)
Computing stats ... group 1/1: min (2/24)
Computing stats ... group 1/1: min (3/24)
Computing stats ... group 1/1: min (4/24)
Computing stats ... group 1/1: min (5/24)
Computing stats ... group 1/1: min (6/24)
Computing stats ... group 1/1: min (7/24)
Computing stats ... group 1/1: min (8/24)
Computing stats ... group 1/1: min (9/24)
Computing stats ... group 1/1: min (10/24)
Computing stats ... group 1/1: min (11/24)
Computing stats ... group 1/1: min (12/24)
Computing stats ... group 1/1: min (13/24)
Computing stats ... group 1/1: min (14/24)
Computing stats ... group 1/1: min (15/24)
Computing stats ... group 1/1: min (16/24)
Computing stats ... group 1/1: min (17/24)
Computing stats ... group 1/1: min (18/24)
Computing stats ... group 1/1: min (19/24)
Computing stats ... group 1/1: min (20/24)
Computing stats ... group 1/1: min (21/24)
Computing stats ... group 1/1: min (22/24)
Computing stats ... group 1/1: min (23/24)
Computing stats ... group 1/1: min (24/24)
Computing stats ... group 1/1: max
Computing stats ... group 1/1: max (1/24)
Computing stats ... group 1/1: max (2/24)
Computing stats ... group 1/1: max (3/24)
Computing stats ... group 1/1: max (4/24)
Computing stats ... group 1/1: max (5/24)
Computing stats ... group 1/1: max (6/24)
Computing stats ... group 1/1: max (7/24)
Computing stats ... group 1/1: max (8/24)
Computing stats ... group 1/1: max (9/24)
Computing stats ... group 1/1: max (10/24)
Computing stats ... group 1/1: max (11/24)
Computing stats ... group 1/1: max (12/24)
Computing stats ... group 1/1: max (13/24)
Computing stats ... group 1/1: max (14/24)
Computing stats ... group 1/1: max (15/24)
Computing stats ... group 1/1: max (16/24)
Computing stats ... group 1/1: max (17/24)
Computing stats ... group 1/1: max (18/24)
Computing stats ... group 1/1: max (19/24)
Computing stats ... group 1/1: max (20/24)
Computing stats ... group 1/1: max (21/24)
Computing stats ... group 1/1: max (22/24)
Computing stats ... group 1/1: max (23/24)
Computing stats ... group 1/1: max (24/24)
Computing stats ... group 1/1: max (1/24)
Computing stats ... group 1/1: max (2/24)
Computing stats ... group 1/1: max (3/24)
Computing stats ... group 1/1: max (4/24)
Computing stats ... group 1/1: max (5/24)
Computing stats ... group 1/1: max (6/24)
Computing stats ... group 1/1: max (7/24)
Computing stats ... group 1/1: max (8/24)
Computing stats ... group 1/1: max (9/24)
Computing stats ... group 1/1: max (10/24)
Computing stats ... group 1/1: max (11/24)
Computing stats ... group 1/1: max (12/24)
Computing stats ... group 1/1: max (13/24)
Computing stats ... group 1/1: max (14/24)
Computing stats ... group 1/1: max (15/24)
Computing stats ... group 1/1: max (16/24)
Computing stats ... group 1/1: max (17/24)
Computing stats ... group 1/1: max (18/24)
Computing stats ... group 1/1: max (19/24)
Computing stats ... group 1/1: max (20/24)
Computing stats ... group 1/1: max (21/24)
Computing stats ... group 1/1: max (22/24)
Computing stats ... group 1/1: max (23/24)
Computing stats ... group 1/1: max (24/24)
Computing stats ... group 1/1: mean
Computing stats ... group 1/1: mean (1/24)
Computing stats ... group 1/1: mean (2/24)
Computing stats ... group 1/1: mean (3/24)
Computing stats ... group 1/1: mean (4/24)
Computing stats ... group 1/1: mean (5/24)
Computing stats ... group 1/1: mean (6/24)
Computing stats ... group 1/1: mean (7/24)
Computing stats ... group 1/1: mean (8/24)
Computing stats ... group 1/1: mean (9/24)
Computing stats ... group 1/1: mean (10/24)
Computing stats ... group 1/1: mean (11/24)
Computing stats ... group 1/1: mean (12/24)
Computing stats ... group 1/1: mean (13/24)
Computing stats ... group 1/1: mean (14/24)
Computing stats ... group 1/1: mean (15/24)
Computing stats ... group 1/1: mean (16/24)
Computing stats ... group 1/1: mean (17/24)
Computing stats ... group 1/1: mean (18/24)
Computing stats ... group 1/1: mean (19/24)
Computing stats ... group 1/1: mean (20/24)
Computing stats ... group 1/1: mean (21/24)
Computing stats ... group 1/1: mean (22/24)
Computing stats ... group 1/1: mean (23/24)
Computing stats ... group 1/1: mean (24/24)
Computing stats ... group 1/1: mean (1/24)
Computing stats ... group 1/1: mean (2/24)
Computing stats ... group 1/1: mean (3/24)
Computing stats ... group 1/1: mean (4/24)
Computing stats ... group 1/1: mean (5/24)
Computing stats ... group 1/1: mean (6/24)
Computing stats ... group 1/1: mean (7/24)
Computing stats ... group 1/1: mean (8/24)
Computing stats ... group 1/1: mean (9/24)
Computing stats ... group 1/1: mean (10/24)
Computing stats ... group 1/1: mean (11/24)
Computing stats ... group 1/1: mean (12/24)
Computing stats ... group 1/1: mean (13/24)
Computing stats ... group 1/1: mean (14/24)
Computing stats ... group 1/1: mean (15/24)
Computing stats ... group 1/1: mean (16/24)
Computing stats ... group 1/1: mean (17/24)
Computing stats ... group 1/1: mean (18/24)
Computing stats ... group 1/1: mean (19/24)
Computing stats ... group 1/1: mean (20/24)
Computing stats ... group 1/1: mean (21/24)
Computing stats ... group 1/1: mean (22/24)
Computing stats ... group 1/1: mean (23/24)
Computing stats ... group 1/1: mean (24/24)
Computing stats ... group 1/1: median
Computing stats ... group 1/1: median (1/24)
Computing stats ... group 1/1: median (2/24)
Computing stats ... group 1/1: median (3/24)
Computing stats ... group 1/1: median (4/24)
Computing stats ... group 1/1: median (5/24)
Computing stats ... group 1/1: median (6/24)
Computing stats ... group 1/1: median (7/24)
Computing stats ... group 1/1: median (8/24)
Computing stats ... group 1/1: median (9/24)
Computing stats ... group 1/1: median (10/24)
Computing stats ... group 1/1: median (11/24)
Computing stats ... group 1/1: median (12/24)
Computing stats ... group 1/1: median (13/24)
Computing stats ... group 1/1: median (14/24)
Computing stats ... group 1/1: median (15/24)
Computing stats ... group 1/1: median (16/24)
Computing stats ... group 1/1: median (17/24)
Computing stats ... group 1/1: median (18/24)
Computing stats ... group 1/1: median (19/24)
Computing stats ... group 1/1: median (20/24)
Computing stats ... group 1/1: median (21/24)
Computing stats ... group 1/1: median (22/24)
Computing stats ... group 1/1: median (23/24)
Computing stats ... group 1/1: median (24/24)
Computing stats ... group 1/1: median (1/24)
Computing stats ... group 1/1: median (2/24)
Computing stats ... group 1/1: median (3/24)
Computing stats ... group 1/1: median (4/24)
Computing stats ... group 1/1: median (5/24)
Computing stats ... group 1/1: median (6/24)
Computing stats ... group 1/1: median (7/24)
Computing stats ... group 1/1: median (8/24)
Computing stats ... group 1/1: median (9/24)
Computing stats ... group 1/1: median (10/24)
Computing stats ... group 1/1: median (11/24)
Computing stats ... group 1/1: median (12/24)
Computing stats ... group 1/1: median (13/24)
Computing stats ... group 1/1: median (14/24)
Computing stats ... group 1/1: median (15/24)
Computing stats ... group 1/1: median (16/24)
Computing stats ... group 1/1: median (17/24)
Computing stats ... group 1/1: median (18/24)
Computing stats ... group 1/1: median (19/24)
Computing stats ... group 1/1: median (20/24)
Computing stats ... group 1/1: median (21/24)
Computing stats ... group 1/1: median (22/24)
Computing stats ... group 1/1: median (23/24)
Computing stats ... group 1/1: median (24/24)
Computing stats ... group 1/1: iqr
Computing stats ... group 1/1: iqr (1/24)
Computing stats ... group 1/1: iqr (2/24)
Computing stats ... group 1/1: iqr (3/24)
Computing stats ... group 1/1: iqr (4/24)
Computing stats ... group 1/1: iqr (5/24)
Computing stats ... group 1/1: iqr (6/24)
Computing stats ... group 1/1: iqr (7/24)
Computing stats ... group 1/1: iqr (8/24)
Computing stats ... group 1/1: iqr (9/24)
Computing stats ... group 1/1: iqr (10/24)
Computing stats ... group 1/1: iqr (11/24)
Computing stats ... group 1/1: iqr (12/24)
Computing stats ... group 1/1: iqr (13/24)
Computing stats ... group 1/1: iqr (14/24)
Computing stats ... group 1/1: iqr (15/24)
Computing stats ... group 1/1: iqr (16/24)
Computing stats ... group 1/1: iqr (17/24)
Computing stats ... group 1/1: iqr (18/24)
Computing stats ... group 1/1: iqr (19/24)
Computing stats ... group 1/1: iqr (20/24)
Computing stats ... group 1/1: iqr (21/24)
Computing stats ... group 1/1: iqr (22/24)
Computing stats ... group 1/1: iqr (23/24)
Computing stats ... group 1/1: iqr (24/24)
Computing stats ... group 1/1: iqr (1/24)
Computing stats ... group 1/1: iqr (2/24)
Computing stats ... group 1/1: iqr (3/24)
Computing stats ... group 1/1: iqr (4/24)
Computing stats ... group 1/1: iqr (5/24)
Computing stats ... group 1/1: iqr (6/24)
Computing stats ... group 1/1: iqr (7/24)
Computing stats ... group 1/1: iqr (8/24)
Computing stats ... group 1/1: iqr (9/24)
Computing stats ... group 1/1: iqr (10/24)
Computing stats ... group 1/1: iqr (11/24)
Computing stats ... group 1/1: iqr (12/24)
Computing stats ... group 1/1: iqr (13/24)
Computing stats ... group 1/1: iqr (14/24)
Computing stats ... group 1/1: iqr (15/24)
Computing stats ... group 1/1: iqr (16/24)
Computing stats ... group 1/1: iqr (17/24)
Computing stats ... group 1/1: iqr (18/24)
Computing stats ... group 1/1: iqr (19/24)
Computing stats ... group 1/1: iqr (20/24)
Computing stats ... group 1/1: iqr (21/24)
Computing stats ... group 1/1: iqr (22/24)
Computing stats ... group 1/1: iqr (23/24)
Computing stats ... group 1/1: iqr (24/24)
Computing stats ... group 1/1: stddev
Computing stats ... group 1/1: stddev (1/24)
Computing stats ... group 1/1: stddev (2/24)
Computing stats ... group 1/1: stddev (3/24)
Computing stats ... group 1/1: stddev (4/24)
Computing stats ... group 1/1: stddev (5/24)
Computing stats ... group 1/1: stddev (6/24)
Computing stats ... group 1/1: stddev (7/24)
Computing stats ... group 1/1: stddev (8/24)
Computing stats ... group 1/1: stddev (9/24)
Computing stats ... group 1/1: stddev (10/24)
Computing stats ... group 1/1: stddev (11/24)
Computing stats ... group 1/1: stddev (12/24)
Computing stats ... group 1/1: stddev (13/24)
Computing stats ... group 1/1: stddev (14/24)
Computing stats ... group 1/1: stddev (15/24)
Computing stats ... group 1/1: stddev (16/24)
Computing stats ... group 1/1: stddev (17/24)
Computing stats ... group 1/1: stddev (18/24)
Computing stats ... group 1/1: stddev (19/24)
Computing stats ... group 1/1: stddev (20/24)
Computing stats ... group 1/1: stddev (21/24)
Computing stats ... group 1/1: stddev (22/24)
Computing stats ... group 1/1: stddev (23/24)
Computing stats ... group 1/1: stddev (24/24)
Computing stats ... group 1/1: stddev (1/24)
Computing stats ... group 1/1: stddev (2/24)
Computing stats ... group 1/1: stddev (3/24)
Computing stats ... group 1/1: stddev (4/24)
Computing stats ... group 1/1: stddev (5/24)
Computing stats ... group 1/1: stddev (6/24)
Computing stats ... group 1/1: stddev (7/24)
Computing stats ... group 1/1: stddev (8/24)
Computing stats ... group 1/1: stddev (9/24)
Computing stats ... group 1/1: stddev (10/24)
Computing stats ... group 1/1: stddev (11/24)
Computing stats ... group 1/1: stddev (12/24)
Computing stats ... group 1/1: stddev (13/24)
Computing stats ... group 1/1: stddev (14/24)
Computing stats ... group 1/1: stddev (15/24)
Computing stats ... group 1/1: stddev (16/24)
Computing stats ... group 1/1: stddev (17/24)
Computing stats ... group 1/1: stddev (18/24)
Computing stats ... group 1/1: stddev (19/24)
Computing stats ... group 1/1: stddev (20/24)
Computing stats ... group 1/1: stddev (21/24)
Computing stats ... group 1/1: stddev (22/24)
Computing stats ... group 1/1: stddev (23/24)
Computing stats ... group 1/1: stddev (24/24)
Computing stats ... group 1/1: ops
Computing stats ... group 1/1: ops (1/24)
Computing stats ... group 1/1: ops (2/24)
Computing stats ... group 1/1: ops (3/24)
Computing stats ... group 1/1: ops (4/24)
Computing stats ... group 1/1: ops (5/24)
Computing stats ... group 1/1: ops (6/24)
Computing stats ... group 1/1: ops (7/24)
Computing stats ... group 1/1: ops (8/24)
Computing stats ... group 1/1: ops (9/24)
Computing stats ... group 1/1: ops (10/24)
Computing stats ... group 1/1: ops (11/24)
Computing stats ... group 1/1: ops (12/24)
Computing stats ... group 1/1: ops (13/24)
Computing stats ... group 1/1: ops (14/24)
Computing stats ... group 1/1: ops (15/24)
Computing stats ... group 1/1: ops (16/24)
Computing stats ... group 1/1: ops (17/24)
Computing stats ... group 1/1: ops (18/24)
Computing stats ... group 1/1: ops (19/24)
Computing stats ... group 1/1: ops (20/24)
Computing stats ... group 1/1: ops (21/24)
Computing stats ... group 1/1: ops (22/24)
Computing stats ... group 1/1: ops (23/24)
Computing stats ... group 1/1: ops (24/24)
Computing stats ... group 1/1: ops (1/24)
Computing stats ... group 1/1: ops (2/24)
Computing stats ... group 1/1: ops (3/24)
Computing stats ... group 1/1: ops (4/24)
Computing stats ... group 1/1: ops (5/24)
Computing stats ... group 1/1: ops (6/24)
Computing stats ... group 1/1: ops (7/24)
Computing stats ... group 1/1: ops (8/24)
Computing stats ... group 1/1: ops (9/24)
Computing stats ... group 1/1: ops (10/24)
Computing stats ... group 1/1: ops (11/24)
Computing stats ... group 1/1: ops (12/24)
Computing stats ... group 1/1: ops (13/24)
Computing stats ... group 1/1: ops (14/24)
Computing stats ... group 1/1: ops (15/24)
Computing stats ... group 1/1: ops (16/24)
Computing stats ... group 1/1: ops (17/24)
Computing stats ... group 1/1: ops (18/24)
Computing stats ... group 1/1: ops (19/24)
Computing stats ... group 1/1: ops (20/24)
Computing stats ... group 1/1: ops (21/24)
Computing stats ... group 1/1: ops (22/24)
Computing stats ... group 1/1: ops (23/24)
Computing stats ... group 1/1: ops (24/24)
Computing stats ... group 1/1: ops: outliers
Computing stats ... group 1/1: ops: outliers (1/24)
Computing stats ... group 1/1: ops: outliers (2/24)
Computing stats ... group 1/1: ops: outliers (3/24)
Computing stats ... group 1/1: ops: outliers (4/24)
Computing stats ... group 1/1: ops: outliers (5/24)
Computing stats ... group 1/1: ops: outliers (6/24)
Computing stats ... group 1/1: ops: outliers (7/24)
Computing stats ... group 1/1: ops: outliers (8/24)
Computing stats ... group 1/1: ops: outliers (9/24)
Computing stats ... group 1/1: ops: outliers (10/24)
Computing stats ... group 1/1: ops: outliers (11/24)
Computing stats ... group 1/1: ops: outliers (12/24)
Computing stats ... group 1/1: ops: outliers (13/24)
Computing stats ... group 1/1: ops: outliers (14/24)
Computing stats ... group 1/1: ops: outliers (15/24)
Computing stats ... group 1/1: ops: outliers (16/24)
Computing stats ... group 1/1: ops: outliers (17/24)
Computing stats ... group 1/1: ops: outliers (18/24)
Computing stats ... group 1/1: ops: outliers (19/24)
Computing stats ... group 1/1: ops: outliers (20/24)
Computing stats ... group 1/1: ops: outliers (21/24)
Computing stats ... group 1/1: ops: outliers (22/24)
Computing stats ... group 1/1: ops: outliers (23/24)
Computing stats ... group 1/1: ops: outliers (24/24)
Computing stats ... group 1/1: ops: rounds
Computing stats ... group 1/1: ops: rounds (1/24)
Computing stats ... group 1/1: ops: rounds (2/24)
Computing stats ... group 1/1: ops: rounds (3/24)
Computing stats ... group 1/1: ops: rounds (4/24)
Computing stats ... group 1/1: ops: rounds (5/24)
Computing stats ... group 1/1: ops: rounds (6/24)
Computing stats ... group 1/1: ops: rounds (7/24)
Computing stats ... group 1/1: ops: rounds (8/24)
Computing stats ... group 1/1: ops: rounds (9/24)
Computing stats ... group 1/1: ops: rounds (10/24)
Computing stats ... group 1/1: ops: rounds (11/24)
Computing stats ... group 1/1: ops: rounds (12/24)
Computing stats ... group 1/1: ops: rounds (13/24)
Computing stats ... group 1/1: ops: rounds (14/24)
Computing stats ... group 1/1: ops: rounds (15/24)
Computing stats ... group 1/1: ops: rounds (16/24)
Computing stats ... group 1/1: ops: rounds (17/24)
Computing stats ... group 1/1: ops: rounds (18/24)
Computing stats ... group 1/1: ops: rounds (19/24)
Computing stats ... group 1/1: ops: rounds (20/24)
Computing stats ... group 1/1: ops: rounds (21/24)
Computing stats ... group 1/1: ops: rounds (22/24)
Computing stats ... group 1/1: ops: rounds (23/24)
Computing stats ... group 1/1: ops: rounds (24/24)
Computing stats ... group 1/1: ops: iterations
Computing stats ... group 1/1: ops: iterations (1/24)
Computing stats ... group 1/1: ops: iterations (2/24)
Computing stats ... group 1/1: ops: iterations (3/24)
Computing stats ... group 1/1: ops: iterations (4/24)
Computing stats ... group 1/1: ops: iterations (5/24)
Computing stats ... group 1/1: ops: iterations (6/24)
Computing stats ... group 1/1: ops: iterations (7/24)
Computing stats ... group 1/1: ops: iterations (8/24)
Computing stats ... group 1/1: ops: iterations (9/24)
Computing stats ... group 1/1: ops: iterations (10/24)
Computing stats ... group 1/1: ops: iterations (11/24)
Computing stats ... group 1/1: ops: iterations (12/24)
Computing stats ... group 1/1: ops: iterations (13/24)
Computing stats ... group 1/1: ops: iterations (14/24)
Computing stats ... group 1/1: ops: iterations (15/24)
Computing stats ... group 1/1: ops: iterations (16/24)
Computing stats ... group 1/1: ops: iterations (17/24)
Computing stats ... group 1/1: ops: iterations (18/24)
Computing stats ... group 1/1: ops: iterations (19/24)
Computing stats ... group 1/1: ops: iterations (20/24)
Computing stats ... group 1/1: ops: iterations (21/24)
Computing stats ... group 1/1: ops: iterations (22/24)
Computing stats ... group 1/1: ops: iterations (23/24)
Computing stats ... group 1/1: ops: iterations (24/24)
----------------------------------------------------- benchmark: 24 tests -----------------------------------------------------
Name (time in ms)                                                 Min                     Max                    Mean          
-------------------------------------------------------------------------------------------------------------------------------
test_transfer_performance[1000-500] (NOW)                     92.6270 (1.0)          104.4690 (1.02)          98.6786 (1.0)    
test_transfer_performance[1000-500] (0001_unversi)            95.3148 (1.03)         104.4025 (1.02)          99.0258 (1.00)   
test_transfer_performance[1000-5000] (NOW)                    98.3282 (1.06)         102.1641 (1.0)          100.6395 (1.02)   
test_transfer_performance[1000-5000] (0001_unversi)          101.5494 (1.10)         128.2741 (1.26)         115.5153 (1.17)   
test_task_forward[1000_tasks] (0001_unversi)                 304.0597 (3.28)         392.9130 (3.85)         351.5185 (3.56)   
test_task_forward[1000_tasks] (NOW)                          311.3788 (3.36)         350.5910 (3.43)         324.5184 (3.29)   
test_transfer_performance[10000-500] (NOW)                   763.0859 (8.24)         839.6550 (8.22)         789.4298 (8.00)   
test_transfer_performance[10000-500] (0001_unversi)          785.5159 (8.48)         885.8352 (8.67)         818.6623 (8.30)   
test_transfer_performance[10000-5000] (0001_unversi)         800.7206 (8.64)         854.0806 (8.36)         826.4243 (8.37)   
test_transfer_performance[10000-5000] (NOW)                  808.6880 (8.73)         850.2822 (8.32)         826.1539 (8.37)   
test_task_forward[10000_tasks] (NOW)                       2,851.6052 (30.79)      3,160.9479 (30.94)      3,022.9445 (30.63)  
test_task_forward[10000_tasks] (0001_unversi)              3,003.9293 (32.43)      3,456.9244 (33.84)      3,114.0767 (31.56)  
test_task_submission[100_tasks] (NOW)                     22,787.3039 (246.01)    23,226.1434 (227.34)    23,052.5453 (233.61) 
test_task_submission[100_tasks] (0001_unversi)            22,842.0423 (246.60)    23,303.9245 (228.10)    23,152.0697 (234.62) 
test_task_submission[10000_tasks] (0001_unversi)          23,007.7064 (248.39)    24,492.1888 (239.73)    23,609.2221 (239.25) 
test_task_submission[10000_tasks] (NOW)                   23,030.3136 (248.63)    23,734.8620 (232.32)    23,347.2059 (236.60) 
test_task_submission[1000_tasks] (NOW)                    23,080.1708 (249.17)    23,684.7761 (231.83)    23,376.6780 (236.90) 
test_task_submission[1000_tasks] (0001_unversi)           23,300.4295 (251.55)    24,048.6048 (235.39)    23,656.6157 (239.73) 
test_task_submission[10_tasks] (0001_unversi)             29,726.8886 (320.93)    30,618.6358 (299.70)    30,297.6021 (307.03) 
test_task_submission[10_tasks] (NOW)                      30,813.4869 (332.66)    31,813.7951 (311.40)    31,395.1985 (318.16) 
test_task_submission[1_tasks] (NOW)                       94,761.2239 (>1000.0)  102,546.0986 (>1000.0)   98,784.5631 (>1000.0)
test_task_submission[1_tasks] (0001_unversi)              99,667.8763 (>1000.0)  129,427.2115 (>1000.0)  111,772.5530 (>1000.0)
test_task_submission[100000_tasks] (0001_unversi)        146,997.5556 (>1000.0)  151,228.6210 (>1000.0)  149,288.7966 (>1000.0)
test_task_submission[100000_tasks] (NOW)                 148,399.8366 (>1000.0)  177,404.9730 (>1000.0)  152,522.6876 (>1000.0)
-------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
========================= 12 passed in 4352.50 seconds =========================
Not sending mail to unregistered user kingchin1218@126.com
Not sending mail to unregistered user chenh1024@gmail.com
Not sending mail to unregistered user vipulharsh93@gmail.com
Not sending mail to unregistered user robertnishihara@gmail.com
Not sending mail to unregistered user noreply@github.com
Sending e-mails to: pcmoritz@gmail.com ekhliang@gmail.com rliaw@berkeley.edu mluo@c70.Millennium.Berkeley.EDU swang@cs.berkeley.edu
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1766/
Test PASSed.
Finished: SUCCESS