FailedConsole Output

[EnvInject] - Mask passwords passed as build parameters.
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on amp-jenkins-staging-worker-01 (ubuntu ubuntu-gpu staging staging-01) in workspace /home/jenkins/workspace/Ray-stress-testing
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Cloning the remote Git repository
Cloning repository https://github.com/ray-project/ray.git
 > git init /home/jenkins/workspace/Ray-stress-testing # timeout=10
Fetching upstream changes from https://github.com/ray-project/ray.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/ray-project/ray.git # timeout=10
Fetching upstream changes from https://github.com/ray-project/ray.git
 > git fetch --tags --progress https://github.com/ray-project/ray.git +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
 > git tag -l # timeout=10
 > git rev-parse refs/tags/ray-0.1.1^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.2.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.1.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.2.2^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.3.1^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.4.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.1.2^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.2.1^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.3.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.5.1^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.6.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.5.0^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.5.3^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.6.2^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.5.2^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.6.1^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.6.4^{commit} # timeout=10
 > git rev-parse refs/tags/ray-0.6.3^{commit} # timeout=10
Checking out Revision 1a1027b3ab24de3159551a4126077d2dcce262b6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1a1027b3ab24de3159551a4126077d2dcce262b6
 > git rev-list 8a6403c26e1bbc66994ffb27ecd925918c8203e1 # timeout=10
[Ray-stress-testing] $ /bin/sh -xe /tmp/hudson1255002250622683204.sh
+ ./ci/stress_tests/run_jenkins_stress_test.sh
+ MEMORY_SIZE=20G
+ SHM_SIZE=20G
+ docker build -q --no-cache -t ray-project/base-deps docker/base-deps
sha256:1d94b137b658eb3231f77fdbfe5c6cb0dc2bbe9cdf7f32168b08d54f84ba4458
+ git rev-parse HEAD
++ git rev-parse HEAD
+ git archive -o ./docker/stress_test/ray.tar 1a1027b3ab24de3159551a4126077d2dcce262b6
++ docker build --no-cache -q -t ray-project/stress_test docker/stress_test
+ DOCKER_SHA=sha256:d34c80fbff024d8d246f8b3cb5a53d44040923a0978bea44656e5d09ff87003f
+ echo 'Using Docker image' sha256:d34c80fbff024d8d246f8b3cb5a53d44040923a0978bea44656e5d09ff87003f
Using Docker image sha256:d34c80fbff024d8d246f8b3cb5a53d44040923a0978bea44656e5d09ff87003f
+ docker run --rm --shm-size=20G --memory=20G -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e RAY_AWS_SSH_KEY sha256:d34c80fbff024d8d246f8b3cb5a53d44040923a0978bea44656e5d09ff87003f bash /ray/ci/stress_tests/run_stress_tests.sh
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Logging to /ray/ci/stress_tests/results-2019-03-14_23-28-23.log
+ touch /ray/ci/stress_tests/results-2019-03-14_23-28-23.log
+ pushd /ray/ci/stress_tests
/ray/ci/stress_tests /ray
+ run_test test_many_tasks_and_transfers
+ local test_name=test_many_tasks_and_transfers
+ local CLUSTER=stress_testing_config.yaml
Try running test_many_tasks_and_transfers.
+ echo 'Try running test_many_tasks_and_transfers.'
+ ray up -y stress_testing_config.yaml --cluster-name test_many_tasks_and_transfers
2019-03-14 23:28:25,062	INFO config.py:107 -- _configure_iam_role: Role not specified for head node, using arn:aws:iam::450145409201:instance-profile/ray-autoscaler-v1
2019-03-14 23:28:25,498	INFO config.py:151 -- _configure_key_pair: KeyName not specified for nodes, using ray-autoscaler_us-west-2
2019-03-14 23:28:25,840	INFO config.py:192 -- _configure_subnet: SubnetIds not specified for head node, using [('subnet-503c0e26', 'us-west-2a')]
2019-03-14 23:28:25,841	INFO config.py:198 -- _configure_subnet: SubnetId not specified for workers, using [('subnet-503c0e26', 'us-west-2a')]
2019-03-14 23:28:26,455	INFO config.py:244 -- _configure_security_group: SecurityGroupIds not specified for head node, using ray-autoscaler-test_many_tasks_and_transfers
2019-03-14 23:28:26,455	INFO config.py:251 -- _configure_security_group: SecurityGroupIds not specified for workers, using ray-autoscaler-test_many_tasks_and_transfers
2019-03-14 23:28:26,624	INFO commands.py:189 -- get_or_create_head_node: Launching new head node...
2019-03-14 23:28:28,139	INFO commands.py:202 -- get_or_create_head_node: Updating files on head node...
2019-03-14 23:28:28,142	INFO updater.py:128 -- NodeUpdater: i-0e5f65155b4a19f19: Updating to afd2dc084f86408c3dc2c8be0375b891905bc9e4
2019-03-14 23:28:28,142	INFO updater.py:90 -- NodeUpdater: Waiting for IP of i-0e5f65155b4a19f19...
2019-03-14 23:28:28,398	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=waiting-for-ssh on ['i-0e5f65155b4a19f19'] [LogTimer=255ms]
2019-03-14 23:28:38,244	INFO updater.py:90 -- NodeUpdater: Waiting for IP of i-0e5f65155b4a19f19...
2019-03-14 23:28:38,449	INFO log_timer.py:21 -- NodeUpdater: i-0e5f65155b4a19f19: Got IP [LogTimer=10306ms]
2019-03-14 23:28:38,465	INFO updater.py:155 -- NodeUpdater: i-0e5f65155b4a19f19: Waiting for SSH...
2019-03-14 23:28:38,466	INFO updater.py:268 -- NodeUpdater: Running uptime on 34.220.11.37...
2019-03-14 23:28:48,487	INFO updater.py:268 -- NodeUpdater: Running uptime on 34.220.11.37...
2019-03-14 23:28:58,507	INFO updater.py:268 -- NodeUpdater: Running uptime on 34.220.11.37...
2019-03-14 23:29:08,535	INFO updater.py:268 -- NodeUpdater: Running uptime on 34.220.11.37...
2019-03-14 23:29:18,555	INFO updater.py:268 -- NodeUpdater: Running uptime on 34.220.11.37...
2019-03-14 23:29:21,562	INFO log_timer.py:21 -- NodeUpdater: i-0e5f65155b4a19f19: Got SSH [LogTimer=43097ms]
2019-03-14 23:29:21,562	INFO updater.py:198 -- NodeUpdater: i-0e5f65155b4a19f19: Syncing /tmp/ray-bootstrap-f_fcRO to ~/ray_bootstrap_config.yaml...
2019-03-14 23:29:21,567	INFO updater.py:268 -- NodeUpdater: Running mkdir -p ~ on 34.220.11.37...
2019-03-14 23:29:21,693	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=syncing-files on ['i-0e5f65155b4a19f19'] [LogTimer=130ms]
2019-03-14 23:29:22,316	INFO log_timer.py:21 -- NodeUpdater i-0e5f65155b4a19f19: Synced /tmp/ray-bootstrap-f_fcRO to ~/ray_bootstrap_config.yaml [LogTimer=750ms]
2019-03-14 23:29:22,317	INFO updater.py:198 -- NodeUpdater: i-0e5f65155b4a19f19: Syncing /root/.ssh/ray-autoscaler_us-west-2.pem to ~/ray_bootstrap_key.pem...
2019-03-14 23:29:22,317	INFO updater.py:268 -- NodeUpdater: Running mkdir -p ~ on 34.220.11.37...
2019-03-14 23:29:22,651	INFO log_timer.py:21 -- NodeUpdater i-0e5f65155b4a19f19: Synced /root/.ssh/ray-autoscaler_us-west-2.pem to ~/ray_bootstrap_key.pem [LogTimer=334ms]
2019-03-14 23:29:22,652	INFO log_timer.py:21 -- NodeUpdater: i-0e5f65155b4a19f19: Initialization commands completed [LogTimer=0ms]
2019-03-14 23:29:22,652	INFO updater.py:268 -- NodeUpdater: Running sudo apt-get update on 34.220.11.37...
2019-03-14 23:29:26,814	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=setting-up on ['i-0e5f65155b4a19f19'] [LogTimer=120ms]
2019-03-14 23:29:33,477	INFO updater.py:268 -- NodeUpdater: Running sudo apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip flex bison python on 34.220.11.37...
2019-03-14 23:30:15,288	INFO updater.py:268 -- NodeUpdater: Running wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh || true on 34.220.11.37...
2019-03-14 23:30:20,568	INFO updater.py:268 -- NodeUpdater: Running bash Anaconda3-5.0.1-Linux-x86_64.sh -b -p $HOME/anaconda3 || true on 34.220.11.37...
2019-03-14 23:31:39,895	INFO updater.py:268 -- NodeUpdater: Running echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc on 34.220.11.37...
2019-03-14 23:31:40,006	INFO updater.py:268 -- NodeUpdater: Running pip install boto3==1.4.8 cython==0.29.0 on 34.220.11.37...
2019-03-14 23:31:44,168	INFO updater.py:268 -- NodeUpdater: Running pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.7.0.dev1-cp36-cp36m-manylinux1_x86_64.whl on 34.220.11.37...
2019-03-14 23:31:55,657	INFO updater.py:268 -- NodeUpdater: Running echo "sudo halt" | at now + 60 minutes on 34.220.11.37...
2019-03-14 23:31:55,820	INFO updater.py:268 -- NodeUpdater: Running ray stop on 34.220.11.37...
2019-03-14 23:31:56,375	INFO log_timer.py:21 -- NodeUpdater: i-0e5f65155b4a19f19: Setup commands completed [LogTimer=153723ms]
2019-03-14 23:31:56,375	INFO log_timer.py:21 -- NodeUpdater: i-0e5f65155b4a19f19: Applied config afd2dc084f86408c3dc2c8be0375b891905bc9e4 [LogTimer=208233ms]
2019-03-14 23:31:56,375	ERROR updater.py:140 -- NodeUpdater: i-0e5f65155b4a19f19: Error updating (Exit Status 1) ssh -i /root/.ssh/ray-autoscaler_us-west-2.pem -o ConnectTimeout=120s -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_sockets/%C -o ControlPersist=5m ubuntu@34.220.11.37 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && ray stop'
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/opt/conda/lib/python2.7/site-packages/ray/autoscaler/updater.py", line 143, in run
    raise e
CalledProcessError: Command '['ssh', '-i', '/root/.ssh/ray-autoscaler_us-west-2.pem', '-o', 'ConnectTimeout=120s', '-o', 'StrictHostKeyChecking=no', '-o', 'ControlMaster=auto', '-o', 'ControlPath=/tmp/ray_ssh_sockets/%C', '-o', 'ControlPersist=5m', 'ubuntu@34.220.11.37', "bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && ray stop'"]' returned non-zero exit status 1

2019-03-14 23:31:56,591	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=update-failed on ['i-0e5f65155b4a19f19'] [LogTimer=215ms]
2019-03-14 23:31:56,654	ERROR commands.py:260 -- get_or_create_head_node: Updating 34.220.11.37 failed
+ echo 'FAIL: test_many_tasks_and_transfers'
+ '[' '' = '' ']'
+ ray down -y stress_testing_config.yaml --cluster-name test_many_tasks_and_transfers
2019-03-14 23:31:58,372	INFO commands.py:111 -- teardown_cluster: Terminating 1 nodes...
2019-03-14 23:31:59,674	INFO log_timer.py:21 -- teardown_cluster: Termination done. [LogTimer=1302ms]
+ run_test test_dead_actors
+ local test_name=test_dead_actors
+ local CLUSTER=stress_testing_config.yaml
+ echo 'Try running test_dead_actors.'
+ ray up -y stress_testing_config.yaml --cluster-name test_dead_actors
Try running test_dead_actors.
2019-03-14 23:32:01,349	INFO config.py:107 -- _configure_iam_role: Role not specified for head node, using arn:aws:iam::450145409201:instance-profile/ray-autoscaler-v1
2019-03-14 23:32:01,760	INFO config.py:151 -- _configure_key_pair: KeyName not specified for nodes, using ray-autoscaler_us-west-2
2019-03-14 23:32:02,084	INFO config.py:192 -- _configure_subnet: SubnetIds not specified for head node, using [('subnet-503c0e26', 'us-west-2a')]
2019-03-14 23:32:02,085	INFO config.py:198 -- _configure_subnet: SubnetId not specified for workers, using [('subnet-503c0e26', 'us-west-2a')]
2019-03-14 23:32:02,736	INFO config.py:244 -- _configure_security_group: SecurityGroupIds not specified for head node, using ray-autoscaler-test_dead_actors
2019-03-14 23:32:02,737	INFO config.py:251 -- _configure_security_group: SecurityGroupIds not specified for workers, using ray-autoscaler-test_dead_actors
2019-03-14 23:32:02,915	INFO commands.py:189 -- get_or_create_head_node: Launching new head node...
2019-03-14 23:32:04,135	INFO commands.py:202 -- get_or_create_head_node: Updating files on head node...
2019-03-14 23:32:04,137	INFO updater.py:128 -- NodeUpdater: i-00d96001515a9b9ee: Updating to 06e896be4f849af6ae26e97c27b921af7e2bbf1f
2019-03-14 23:32:04,137	INFO updater.py:90 -- NodeUpdater: Waiting for IP of i-00d96001515a9b9ee...
2019-03-14 23:32:04,273	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=waiting-for-ssh on ['i-00d96001515a9b9ee'] [LogTimer=134ms]
2019-03-14 23:32:14,413	INFO updater.py:90 -- NodeUpdater: Waiting for IP of i-00d96001515a9b9ee...
2019-03-14 23:32:14,595	INFO log_timer.py:21 -- NodeUpdater: i-00d96001515a9b9ee: Got IP [LogTimer=10458ms]
2019-03-14 23:32:14,609	INFO updater.py:155 -- NodeUpdater: i-00d96001515a9b9ee: Waiting for SSH...
2019-03-14 23:32:14,609	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:24,628	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:34,656	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:44,686	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:49,735	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:54,780	INFO updater.py:268 -- NodeUpdater: Running uptime on 54.244.17.246...
2019-03-14 23:32:55,613	INFO log_timer.py:21 -- NodeUpdater: i-00d96001515a9b9ee: Got SSH [LogTimer=41004ms]
2019-03-14 23:32:55,614	INFO updater.py:198 -- NodeUpdater: i-00d96001515a9b9ee: Syncing /tmp/ray-bootstrap-hcwBvp to ~/ray_bootstrap_config.yaml...
2019-03-14 23:32:55,618	INFO updater.py:268 -- NodeUpdater: Running mkdir -p ~ on 54.244.17.246...
2019-03-14 23:32:55,718	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=syncing-files on ['i-00d96001515a9b9ee'] [LogTimer=104ms]
2019-03-14 23:32:55,989	INFO log_timer.py:21 -- NodeUpdater i-00d96001515a9b9ee: Synced /tmp/ray-bootstrap-hcwBvp to ~/ray_bootstrap_config.yaml [LogTimer=371ms]
2019-03-14 23:32:55,990	INFO updater.py:198 -- NodeUpdater: i-00d96001515a9b9ee: Syncing /root/.ssh/ray-autoscaler_us-west-2.pem to ~/ray_bootstrap_key.pem...
2019-03-14 23:32:55,990	INFO updater.py:268 -- NodeUpdater: Running mkdir -p ~ on 54.244.17.246...
2019-03-14 23:32:56,331	INFO log_timer.py:21 -- NodeUpdater i-00d96001515a9b9ee: Synced /root/.ssh/ray-autoscaler_us-west-2.pem to ~/ray_bootstrap_key.pem [LogTimer=341ms]
2019-03-14 23:32:56,331	INFO log_timer.py:21 -- NodeUpdater: i-00d96001515a9b9ee: Initialization commands completed [LogTimer=0ms]
2019-03-14 23:32:56,331	INFO updater.py:268 -- NodeUpdater: Running sudo apt-get update on 54.244.17.246...
2019-03-14 23:33:00,822	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=setting-up on ['i-00d96001515a9b9ee'] [LogTimer=103ms]
2019-03-14 23:33:03,715	INFO updater.py:268 -- NodeUpdater: Running sudo apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip flex bison python on 54.244.17.246...
2019-03-14 23:33:26,777	INFO updater.py:268 -- NodeUpdater: Running wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh || true on 54.244.17.246...
2019-03-14 23:33:31,611	INFO updater.py:268 -- NodeUpdater: Running bash Anaconda3-5.0.1-Linux-x86_64.sh -b -p $HOME/anaconda3 || true on 54.244.17.246...
2019-03-14 23:34:47,636	INFO updater.py:268 -- NodeUpdater: Running echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc on 54.244.17.246...
2019-03-14 23:34:47,756	INFO updater.py:268 -- NodeUpdater: Running pip install boto3==1.4.8 cython==0.29.0 on 54.244.17.246...
2019-03-14 23:34:51,706	INFO updater.py:268 -- NodeUpdater: Running pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.7.0.dev1-cp36-cp36m-manylinux1_x86_64.whl on 54.244.17.246...
2019-03-14 23:35:02,205	INFO updater.py:268 -- NodeUpdater: Running echo "sudo halt" | at now + 60 minutes on 54.244.17.246...
2019-03-14 23:35:03,272	INFO updater.py:268 -- NodeUpdater: Running ray stop on 54.244.17.246...
2019-03-14 23:35:03,840	INFO log_timer.py:21 -- NodeUpdater: i-00d96001515a9b9ee: Setup commands completed [LogTimer=127508ms]
2019-03-14 23:35:03,840	INFO log_timer.py:21 -- NodeUpdater: i-00d96001515a9b9ee: Applied config 06e896be4f849af6ae26e97c27b921af7e2bbf1f [LogTimer=179703ms]
2019-03-14 23:35:03,840	ERROR updater.py:140 -- NodeUpdater: i-00d96001515a9b9ee: Error updating (Exit Status 1) ssh -i /root/.ssh/ray-autoscaler_us-west-2.pem -o ConnectTimeout=120s -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_sockets/%C -o ControlPersist=5m ubuntu@54.244.17.246 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && ray stop'
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/opt/conda/lib/python2.7/site-packages/ray/autoscaler/updater.py", line 143, in run
    raise e
CalledProcessError: Command '['ssh', '-i', '/root/.ssh/ray-autoscaler_us-west-2.pem', '-o', 'ConnectTimeout=120s', '-o', 'StrictHostKeyChecking=no', '-o', 'ControlMaster=auto', '-o', 'ControlPath=/tmp/ray_ssh_sockets/%C', '-o', 'ControlPersist=5m', 'ubuntu@54.244.17.246', "bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && ray stop'"]' returned non-zero exit status 1

2019-03-14 23:35:04,092	INFO log_timer.py:21 -- AWSNodeProvider: Set tag ray-node-status=update-failed on ['i-00d96001515a9b9ee'] [LogTimer=252ms]
2019-03-14 23:35:04,177	ERROR commands.py:260 -- get_or_create_head_node: Updating 54.244.17.246 failed
+ echo 'FAIL: test_dead_actors'
+ '[' '' = '' ']'
+ ray down -y stress_testing_config.yaml --cluster-name test_dead_actors
2019-03-14 23:35:05,951	INFO commands.py:111 -- teardown_cluster: Terminating 1 nodes...
2019-03-14 23:35:07,247	INFO log_timer.py:21 -- teardown_cluster: Termination done. [LogTimer=1296ms]
+ popd
+ cat /ray/ci/stress_tests/results-2019-03-14_23-28-23.log
/ray
FAIL: test_many_tasks_and_transfers
FAIL: test_dead_actors
+ '[' '!' -s /ray/ci/stress_tests/results-2019-03-14_23-28-23.log ']'
+ exit 1
Build step 'Execute shell' marked build as failure
Sending e-mails to: rkn@berkeley.edu pcm@berkeley.edu rliaw@berkeley.edu ekhliang@gmail.com wang.stephanie93@gmail.com
Finished: FAILURE