Revision history [back]

click to hide/show revision 1
initial version

cloud-init fails only in a particular tenant

We have a pretty basic OpenStack Juno with Neutron which you can build by following OpenStack Installation Guide for Ubuntu 14.04. It works great, but recently we've seen the following logs on the output of nova console-log <server>. It only happens in one particular project. Is it an issue in quota setting? If so, which quota should I look into?

2015-04-28 14:18:07,546 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTT
PConnectionPool object at 0x7fdf0af35450>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:18:58,600 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af355d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:19:16,621 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af350d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)')]
2015-04-28 14:19:17,623 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
2015-04-28 14:20:07,675 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTTPConnecti
onPool object at 0x7fdf0af35990>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:20:58,728 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af35210>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:21:16,749 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af351d0>, 'Connection to 10.23.0.3 timed out. (connect timeout=17.0)')]
2015-04-28 14:21:17,751 - DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from ['http://10.23.0.3//latest/meta-data/instance-id'] after 120 seconds
Apr 28 14:21:58 ubuntu pollinate[713]: ERROR: Network communication failed [28]\n14:21:18.264414 * Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0
   0     0      0      0 --:--:--  0:00:02 --:--:--     014:21:21.778354 * Resolving timed out after 3513 milliseconds
  0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0
14:21:58.303965 * Closing connection 0
curl: (28) Resolving timed out after 3513 milliseconds
2015-04-28 14:21:58,364 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random
.pyc'>) failed

cloud-init fails only in a particular tenant

We have a pretty basic OpenStack Juno with Neutron which you can build by following OpenStack "OpenStack Installation Guide for Ubuntu 14.04. 14.04". It works great, but recently we've seen the following logs on the output of nova console-log <server>. It only happens in one particular project. Is it an issue in quota setting? If so, which quota should I look into?

2015-04-28 14:18:07,546 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTT
PConnectionPool object at 0x7fdf0af35450>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:18:58,600 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af355d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:19:16,621 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af350d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)')]
2015-04-28 14:19:17,623 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
2015-04-28 14:20:07,675 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTTPConnecti
onPool object at 0x7fdf0af35990>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:20:58,728 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af35210>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:21:16,749 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af351d0>, 'Connection to 10.23.0.3 timed out. (connect timeout=17.0)')]
2015-04-28 14:21:17,751 - DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from ['http://10.23.0.3//latest/meta-data/instance-id'] after 120 seconds
Apr 28 14:21:58 ubuntu pollinate[713]: ERROR: Network communication failed [28]\n14:21:18.264414 * Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0
   0     0      0      0 --:--:--  0:00:02 --:--:--     014:21:21.778354 * Resolving timed out after 3513 milliseconds
  0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0
14:21:58.303965 * Closing connection 0
curl: (28) Resolving timed out after 3513 milliseconds
2015-04-28 14:21:58,364 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random
.pyc'>) failed

cloud-init fails only in a particular tenant

We have a pretty basic OpenStack Juno with Neutron which you can build by following "OpenStack Installation Guide for Ubuntu 14.04". It works great, but recently we've seen the following logs on the output of nova console-log <server>. It only happens in one particular project. tenant. Is it an issue in quota setting? If so, which quota should I look into?

2015-04-28 14:18:07,546 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTT
PConnectionPool object at 0x7fdf0af35450>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:18:58,600 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af355d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:19:16,621 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af350d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)')]
2015-04-28 14:19:17,623 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
2015-04-28 14:20:07,675 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTTPConnecti
onPool object at 0x7fdf0af35990>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:20:58,728 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af35210>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:21:16,749 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af351d0>, 'Connection to 10.23.0.3 timed out. (connect timeout=17.0)')]
2015-04-28 14:21:17,751 - DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from ['http://10.23.0.3//latest/meta-data/instance-id'] after 120 seconds
Apr 28 14:21:58 ubuntu pollinate[713]: ERROR: Network communication failed [28]\n14:21:18.264414 * Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0
   0     0      0      0 --:--:--  0:00:02 --:--:--     014:21:21.778354 * Resolving timed out after 3513 milliseconds
  0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0
14:21:58.303965 * Closing connection 0
curl: (28) Resolving timed out after 3513 milliseconds
2015-04-28 14:21:58,364 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random
.pyc'>) failed

cloud-init fails only in a particular tenant

We have a pretty basic OpenStack Juno with Neutron which you can build by following "OpenStack Installation Guide for Ubuntu 14.04". It works great, but recently we've seen the following logs on the output of cloud-init started to fail ssh key injection, and nova console-log <server>. It shows following errors. Even more strangely, this problem only happens in one particular tenant. Is it an issue in quota setting? If so, which quota should I look into?setting, or something else? Any help or comment would be appreciated.

2015-04-28 14:18:07,546 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTT
PConnectionPool object at 0x7fdf0af35450>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:18:58,600 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af355d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)')]
2015-04-28 14:19:16,621 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HT
TPConnectionPool object at 0x7fdf0af350d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)')]
2015-04-28 14:19:17,623 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
2015-04-28 14:20:07,675 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [50/120s]: request error [(<urllib3.connectionpool.HTTPConnecti
onPool object at 0x7fdf0af35990>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:20:58,728 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [101/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af35210>, 'Connection to 10.23.0.3 timed out. (connect timeout=50.0)')]
2015-04-28 14:21:16,749 - url_helper.py[WARNING]: Calling 'http://10.23.0.3//latest/meta-data/instance-id' failed [119/120s]: request error [(<urllib3.connectionpool.HTTPConnect
ionPool object at 0x7fdf0af351d0>, 'Connection to 10.23.0.3 timed out. (connect timeout=17.0)')]
2015-04-28 14:21:17,751 - DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from ['http://10.23.0.3//latest/meta-data/instance-id'] after 120 seconds
Apr 28 14:21:58 ubuntu pollinate[713]: ERROR: Network communication failed [28]\n14:21:18.264414 * Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  0     0    0     0
   0     0      0      0 --:--:--  0:00:02 --:--:--     014:21:21.778354 * Resolving timed out after 3513 milliseconds
  0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0
14:21:58.303965 * Closing connection 0
curl: (28) Resolving timed out after 3513 milliseconds
2015-04-28 14:21:58,364 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random
.pyc'>) failed