Ask Your Question
0

Can not access second and next instances via ssh/ping etc

asked 2011-02-25 20:04:12 -0600

arturo-lorenzo gravatar image

Launched an instance, runs OK, I can access it via ssh/ping etc. Launch a second instance from the same image and I can not access it via ssh/ping etc. Both instances are running OK.

On the second instance console shows:

2011-02-25 16:09:20,916 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id (http://169.254.169.254/2009-04-04/met...)

2011-02-25 16:09:20,916 - DataSourceEc2.py[WARNING]: 16:09:20 [ 1/100]: url error [timed out]

2011-02-25 16:50:42,117 - DataSourceEc2.py[CRITICAL]: giving up on md after 1213 seconds

2011-02-25 16:10:26,875 - DataSourceEc2.py[WARNING]: 16:10:26 [17/100]: url error [[Errno 113] No route to host]

Traceback (most recent call last): File "/usr/bin/cloud-init-cfg", line 116, in <module> main() File "/usr/bin/cloud-init-cfg", line 63, in main cc = cloudinit.CloudConfig.CloudConfig(cfg_path) File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig/__init__.py", line 35, in __init__ self.cloud.get_data_source() File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 270, in get_data_source log.debug("did not find data source from %s" % dslist) TypeError: not all arguments converted during string formatting

=============================================

I have not been able to access any new instance via ssh/ping after the first one which is still running OK.

edit retag flag offensive close merge delete

11 answers

Sort by ยป oldest newest most voted
0

answered 2011-02-25 23:22:09 -0600

berendt gravatar image

Please paste your nova.conf and your logfiles (I think the logs of nova-network and nova-compute are interesting in this case) to http://paste.openstack.org and post the URLs here.

edit flag offensive delete link more
0

answered 2011-02-28 14:03:48 -0600

arturo-lorenzo gravatar image

here is the nova.conf file, failed to post the logs using http://pasted.openstack.org , another way? --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --verbose --sql_connection=mysql://root:nova@13.141.230.64/nova --s3_host=13.141.230.64 --rabbit_host=13.141.230.64 --ec2_host=13.141.230.64 --ec2_url=http://13.141.230.64:8773/services/Cloud --network_manager=nova.network.manager.FlatDHCPManager --fixed_range=13.141.230.96/28 --flat_network_dhcp_start=13.141.230.96 --flat_network_dns=13.141.230.8 --dns_server=13.141.230.8 --flat_injected=False --network_size=16 --fixed_ip_disassociate_timeout=60

thanks

edit flag offensive delete link more
0

answered 2011-02-28 19:53:29 -0600

vishvananda gravatar image

are you seeing the second ip get leased properly? (You should see leased reports every two minutes from each instance in nova-network) It seems like you might have an ip conflict. If you describe_instances do they show that they have different ips? Are you sure there isn't an old instance (or another machine on the network) trying to claim the same ip?

Vish

On Feb 28, 2011, at 6:03 AM, arturo lorenzo wrote:

Question #146894 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Status: Needs information => Open

arturo lorenzo gave more information on the question: here is the nova.conf file, failed to post the logs using pasted.openstack.org , another way? --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --verbose --sql_connection=mysql://root:nova@13.141.230.64/nova --s3_host=13.141.230.64 --rabbit_host=13.141.230.64 --ec2_host=13.141.230.64 --ec2_url=http://13.141.230.64:8773/services/Cloud --network_manager=nova.network.manager.FlatDHCPManager --fixed_range=13.141.230.96/28 --flat_network_dhcp_start=13.141.230.96 --flat_network_dns=13.141.230.8 --dns_server=13.141.230.8 --flat_injected=False --network_size=16 --fixed_ip_disassociate_timeout=60

thanks


You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-03-01 04:43:51 -0600

arturo-lorenzo gravatar image

Vish, now my first instance is not running anymore. I think the problem is that the instance is trying to access this url http://169.254.169.254/2009-04-04/meta-data/instance-id (http://169.254.169.254/2009-04-04/met...) and after a while it changes to CRITICAL and it stays there forever. The range of IPs is unique and they are dedicated to the instances only. I am also seeing a progressive IP # on new instances but all of them try to access that url. I have entered the iptables cmd on the compute node.

see the beginning and the end of the attached log:

cloud-init start running: Tue, 01 Mar 2011 03:36:54 +0000. up 1.96 seconds 2011-03-01 03:36:56,243 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id (http://169.254.169.254/2009-04-04/met...)

2011-03-01 03:36:56,244 - DataSourceEc2.py[WARNING]: 03:36:56 [ 1/100]: url error [timed out]

2011-03-01 03:36:57,250 - DataSourceEc2.py[WARNING]: 03:36:57 [ 2/100]: url error [[Errno 113] No route to host]

2011-03-01 03:37:00,256 - DataSourceEc2.py[WARNING]: 03:37:00 [ 3/100]: url error [timed out]

......

2011-03-01 04:17:48,760 - DataSourceEc2.py[WARNING]: 04:17:48 [100/100]: url error [timed out]

2011-03-01 04:18:08,782 - DataSourceEc2.py[CRITICAL]: giving up on md after 1208 seconds

Traceback (most recent call last): File "/usr/bin/cloud-init-cfg", line 116, in <module> main() File "/usr/bin/cloud-init-cfg", line 63, in main cc = cloudinit.CloudConfig.CloudConfig(cfg_path) File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig/__init__.py", line 35, in __init__ self.cloud.get_data_source() File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 270, in get_data_source log.debug("did not find data source from %s" % dslist) TypeError: not all arguments converted during string formatting

======== here is my instance: euca-describe-instances RESERVATION r-gvmuusmn simple_cloud default INSTANCE i-00000002 ami-zjnxlmfr 13.141.230.98 13.141.230.98 running mykey (simple_cloud, cc1) 0 m1.small 2011-03-01T03:36:25Z nova

thanks!

edit flag offensive delete link more
0

answered 2011-03-01 17:02:16 -0600

vishvananda gravatar image

The iptables command should be on the network node, and it is done automatically by flatdhcp. Also, you may need to give 169.254.169.254 to the network node to allow some guests to reach it properly (this generally happens with desktop images). for example: ip addr add 169.254.169.254/32 link local dev br100

On Feb 28, 2011, at 8:43 PM, arturo lorenzo wrote:

Question #146894 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Status: Answered => Open

arturo lorenzo is still having a problem: Vish, now my first instance is not running anymore. I think the problem is that the instance is trying to access this url http://169.254.169.254/2009-04-04/met... and after a while it changes to CRITICAL and it stays there forever. The range of IPs is unique and they are dedicated to the instances only. I am also seeing a progressive IP # on new instances but all of them try to access that url. I have entered the iptables cmd on the compute node.

see the beginning and the end of the attached log:

cloud-init start running: Tue, 01 Mar 2011 03:36:54 +0000. up 1.96 seconds 2011-03-01 03:36:56,243 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/met...

2011-03-01 03:36:56,244 - DataSourceEc2.py[WARNING]: 03:36:56 [ 1/100]: url error [timed out]

2011-03-01 03:36:57,250 - DataSourceEc2.py[WARNING]: 03:36:57 [ 2/100]: url error [[Errno 113] No route to host]

2011-03-01 03:37:00,256 - DataSourceEc2.py[WARNING]: 03:37:00 [ 3/100]: url error [timed out]

......

2011-03-01 04:17:48,760 - DataSourceEc2.py[WARNING]: 04:17:48 [100/100]: url error [timed out]

2011-03-01 04:18:08,782 - DataSourceEc2.py[CRITICAL]: giving up on md after 1208 seconds

Traceback (most recent call last): File "/usr/bin/cloud-init-cfg", line 116, in <module> main() File "/usr/bin/cloud-init-cfg", line 63, in main cc = cloudinit.CloudConfig.CloudConfig(cfg_path) File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig/__init__.py", line 35, in __init__ self.cloud.get_data_source() File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 270, in get_data_source log.debug("did not find data source from %s" % dslist) TypeError: not all arguments converted during string formatting

======== here is my instance: euca-describe-instances RESERVATION r-gvmuusmn simple_cloud default INSTANCE i-00000002 ami-zjnxlmfr 13.141.230.98 13.141.230.98 running mykey (simple_cloud, cc1) 0 m1.small 2011-03-01T03:36:25Z nova

thanks!


You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-03-02 02:38:24 -0600

arturo-lorenzo gravatar image

Still can't run an instance. I have reinstalled nova from zero and I am using this nova.conf: --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --verbose --sql_connection=mysql://root:nova@192.168.1.194/nova --s3_host=192.168.1.194 --rabbit_host=192.168.1.194 --ec2_host=192.168.1.194 --ec2_url=http://192.168.1.194:8773/services/Cloud --network_manager=nova.network.manager.FlatDHCPManager --fixed_range=10.0.0.0/24 --flat_network_dhcp_start=10.0.0.2 --flat_network_dns=209.18.47.61 --flat_injected=False --fixed_ip_disassociate_timeout=60 --network_size=16 --dns_server=209.18.47.61

and when I run an instance I get this:

2011-03-01 20:53:28,561 DEBUG nova.rpc [-] Initing the Adapter Consumer for compute from (pid=8161) __init__ /usr/lib/pymodules/python2.6/nova/rpc.py:160 2011-03-01 20:53:28,607 DEBUG nova.rpc [-] Initing the Adapter Consumer for compute.ubuntu-svr1 from (pid=8161) __init__ /usr/lib/pymodules/python2.6/nova/rpc.py:160 2011-03-01 21:09:47,520 DEBUG nova.rpc [-] received {u'_context_request_id': u'Y2LX5RC-4MKBFUPRCGU7', u'_context_read_deleted': False, u'args': {u'instance_id': 1, u'onset_files': None, u'availability_zone': None}, u'_context_is_admin': True, u'_context_timestamp': u'2011-03-02T02:09:46Z', u'_context_user': u'admin', u'method': u'run_instance', u'_context_project': u'simple_cloud', u'_context_remote_address': u'192.168.1.194'} from (pid=8161) _receive /usr/lib/pymodules/python2.6/nova/rpc.py:179 2011-03-01 21:09:47,521 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-03-02T02:09:46Z', 'remote_address': u'192.168.1.194', 'project': u'simple_cloud', 'is_admin': True, 'user': u'admin', 'request_id': u'Y2LX5RC-4MKBFUPRCGU7', 'read_deleted': False} from (pid=8161) _unpack_context /usr/lib/pymodules/python2.6/nova/rpc.py:295 2011-03-01 21:09:47,581 DEBUG nova.virt.libvirt_conn [-] Connecting to libvirt: qemu:///system from (pid=8161) _get_connection /usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py:183 2011-03-01 21:09:47,770 AUDIT nova.compute.manager [Y2LX5RC-4MKBFUPRCGU7 admin simple_cloud] instance 1: starting... 2011-03-01 21:09:47,900 DEBUG nova.rpc [-] Making asynchronous call... from (pid=8161) call /usr/lib/pymodules/python2.6/nova/rpc.py:314 2011-03-01 21:09:47,900 DEBUG nova.rpc [-] MSG_ID is 6324edb1992c48c6a04dc70c1784b372 from (pid=8161) call /usr/lib/pymodules/python2.6/nova/rpc.py:317 2011-03-01 21:09:49,440 DEBUG nova.rpc [-] Making asynchronous call... from (pid=8161) call /usr/lib/pymodules/python2.6/nova/rpc.py:314 2011-03-01 21:09:49,441 DEBUG nova.rpc [-] MSG_ID is 310a43e781c643878c2dbaa90df86c82 from (pid=8161) call /usr/lib/pymodules/python2.6/nova/rpc.py:317 2011-03-01 21:09:49,944 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev br100 from (pid=8161) execute /usr/lib/pymodules/python2.6/nova/utils.py:132 2011-03-01 21:09:49,971 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables --delete FORWARD --in-interface br100 -j ACCEPT from (pid=8161) execute /usr/lib/pymodules/python2.6/nova/utils.py:132 2011-03-01 21:09:49,982 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables -I FORWARD --in-interface br100 -j ACCEPT ... (more)

edit flag offensive delete link more
0

answered 2011-03-02 22:28:53 -0600

Try: chmod a+rw instances/i-00000001/console.log

It worked for me. I suspect is a bug...

indro

edit flag offensive delete link more
0

answered 2011-03-02 23:32:51 -0600

arturo-lorenzo gravatar image

Indro, thanks so much for figuring out this problem. you are right. I've tried changing the permissions to the console and run it with the virsh create command successfully. But I am running this instance with euca-run-instances and there is no way to specify that permission because the instances directories are created when euca-run-instances is launched. So I guess I need to wait for the fix. Thanks!

edit flag offensive delete link more
0

answered 2011-03-02 23:54:55 -0600

Or you can just change the permission and use euca-reboot-instances...

edit flag offensive delete link more
0

answered 2011-03-03 04:23:40 -0600

arturo-lorenzo gravatar image

Yep, that is a temporary working around which works OK. Do you know if this is reported as a bug? should I create a bug report? Thanks again Indro!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-02-25 20:04:12 -0600

Seen: 187 times

Last updated: Mar 09 '11