Ask Your Question
0

Nova Netowrk Multihost Issue

asked 2013-01-25 21:29:29 -0500

stephen-kahmann gravatar image

I am setting up a FlatDHCPManager network using multi host. I am having an issue where my cloud vms don't boot correctly because they can't communicate with the metadata service. Nova network is installed on both the controller and the compute node. The vm receives an IP address, but boots with the following errors:

cloud-init start-local running: Fri, 25 Jan 2013 21:19:24 +0000. up 2.58 seconds no instance data found in start-local ci-info: lo : 1 127.0.0.1 255.0.0.0 . ci-info: eth0 : 1 10.0.1.130 255.255.255.128 fa:16:3e:41:08:3e ci-info: route-0: 0.0.0.0 10.0.1.129 0.0.0.0 eth0 UG ci-info: route-1: 10.0.1.128 0.0.0.0 255.255.255.128 eth0 U cloud-init start running: Fri, 25 Jan 2013 21:19:25 +0000. up 2.98 seconds 2013-01-25 21:19:47,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:19:51,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [5/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:19:54,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [8/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:19:57,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [11/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:00,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [14/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:03,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [17/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:06,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [20/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:09,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [23/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:12,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [26/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:15,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [29/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:18,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [32/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:24,150 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [38/120s]: url error [[Errno 113] No route to host] 2013-01-25 21:20:30,154 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [44/120s]: url error [[Errno 113] No ... (more)

edit retag flag offensive close merge delete

7 answers

Sort by » oldest newest most voted
0

answered 2013-01-30 02:32:48 -0500

stephen-kahmann gravatar image

The only IPs I configured were the ones in the interface files as this was a fresh install. I'm not sure what caused the issue. Thanks again for your help. I'm just glad the issue is resolved.

edit flag offensive delete link more
0

answered 2013-01-30 01:40:45 -0500

keith-tobin gravatar image

Would it be possible that at some stage you hard 10.0.1.131 in config and started nova-networking. Nova-networking would have then populated the network/interfaces with br100 and the incorrect IP. later when you change it in config, it would not changed in network/interface file, even after restarting nova-networking as it would see their was a br100 already configured.

Thanks for the feed back, it helped me understand you problem.

edit flag offensive delete link more
0

answered 2013-01-28 20:39:05 -0500

stephen-kahmann gravatar image

I solved the problem. As it turns out, the default gateway for the nova network was 10.0.1.129, but nova had assigned 10.0.1.131 to br100. All I had to do was 'ip addr add 10.0.1.129 dev br100' and the machine was capable of booting and I can ssh into it. Not sure if this is a bug or known behavior.

Thank you!

edit flag offensive delete link more
0

answered 2013-01-28 18:41:06 -0500

stephen-kahmann gravatar image

Also here's my iptables config:

irad-compute@irad-compute:~$ sudo iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N nova-api-metadat-OUTPUT -N nova-api-metadat-POSTROUTING -N nova-api-metadat-PREROUTING -N nova-api-metadat-float-snat -N nova-api-metadat-snat -N nova-compute-OUTPUT -N nova-compute-POSTROUTING -N nova-compute-PREROUTING -N nova-compute-float-snat -N nova-compute-snat -N nova-network-OUTPUT -N nova-network-POSTROUTING -N nova-network-PREROUTING -N nova-network-float-snat -N nova-network-snat -N nova-postrouting-bottom -A PREROUTING -j nova-network-PREROUTING -A PREROUTING -j nova-compute-PREROUTING -A PREROUTING -j nova-api-metadat-PREROUTING -A OUTPUT -j nova-network-OUTPUT -A OUTPUT -j nova-compute-OUTPUT -A OUTPUT -j nova-api-metadat-OUTPUT -A POSTROUTING -j nova-network-POSTROUTING -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-api-metadat-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A nova-api-metadat-snat -j nova-api-metadat-float-snat -A nova-compute-snat -j nova-compute-float-snat -A nova-network-POSTROUTING -s 10.0.1.128/25 -d 10.0.1.4/32 -j ACCEPT -A nova-network-POSTROUTING -s 10.0.1.128/25 -d 10.0.1.128/25 -m conntrack ! --ctstate DNAT -j ACCEPT -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.1.4:8775 -A nova-network-snat -j nova-network-float-snat -A nova-network-snat -s 10.0.1.128/25 -o p255p2 -j SNAT --to-source 7.7.7.102 -A nova-postrouting-bottom -j nova-network-snat -A nova-postrouting-bottom -j nova-compute-snat -A nova-postrouting-bottom -j nova-api-metadat-snat

edit flag offensive delete link more
0

answered 2013-01-28 18:29:55 -0500

stephen-kahmann gravatar image

Looking through the nova network config I see that the gateway is 10.0.1.129, but that ip is never assigned to an interface:

+---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | bridge | br100 | | bridge_interface | p5p1 | | broadcast | 10.0.1.255 | | cidr | 10.0.1.128/25 | | cidr_v6 | None | | created_at | 2013-01-23T23:01:49.000000 | | deleted | False | | deleted_at | None | | dhcp_start | 10.0.1.130 | | dns1 | 8.8.4.4 | | dns2 | None | | gateway | 10.0.1.129 | | gateway_v6 | None | | host | None | | id | 74289662-65ed-4f36-90c4-67624e722063 | | injected | True | | label | private | | multi_host | True | | netmask | 255.255.255.128 | | netmask_v6 | None | | priority | None | | project_id | None | | rxtx_base | None | | updated_at | 2013-01-25T16:36:20.000000 | | vlan | None | | vpn_private_address | None | | vpn_public_address | None | | vpn_public_port | None | +---------------------+--------------------------------------+

edit flag offensive delete link more
0

answered 2013-01-28 13:01:21 -0500

stephen-kahmann gravatar image

Sure, here are the requested files:

Controller Config:

[DEFAULT]

iscsi_helper=tgtadm root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf ec2_private_dns_show_ip=True volumes_path=/var/lib/nova/volumes

LOGS/STATE

verbose=True logdir=/var/log/nova state_path=/var/lib/nova lock_path=/run/lock/nova

AUTHENTICATION

auth_strategy=keystone

SCHEDULER

scheduler_driver=nova.scheduler.multi.MultiScheduler compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

CINDER

volume_api_class=nova.volume.cinder.API

DATABASE

sql_connection=mysql://novaUser:novaPass@10.0.1.4/nova

COMPUTE

libvirt_type=kvm libvirt_use_virtio_for_bridges=True start_guests_on_host_boot=True resume_guests_state_on_host_boot=True api_paste_config=/etc/nova/api-paste.ini allow_admin_api=True use_deprecated_auth=False nova_url=http://10.0.1.4:8774/v1.1/ root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

APIS

ec2_host=10.0.1.4 ec2_url=http://10.0.1.4:8773/services/Cloud keystone_ec2_url=http://10.0.1.4:5000/v2.0/ec2tokens s3_host=10.0.1.4 cc_host=10.0.1.4 metadata_host=10.0.1.4 #metadata_listen=0.0.0.0 enabled_apis=ec2,osapi_compute,metadata

RABBITMQ

rabbit_host=10.0.1.4

GLANCE

image_service=nova.image.glance.GlanceImageService glance_api_servers=10.0.1.4:9292

NETWORK

network_manager=nova.network.manager.FlatDHCPManager force_dhcp_release=True dhcpbridge_flagfile=/etc/nova/nova.conf dhcpbridge=/usr/bin/nova-dhcpbridge firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver public_interface=p5p1 flat_interface=p4p1 flat_network_bridge=br100 fixed_range=10.0.1.129/25 network_size=128 flat_network_dhcp_start=10.0.1.129 flat_injected=False connection_type=libvirt multi_host=True

NOVNC CONSOLE

novnc_enabled=True novncproxy_base_url=http://7.7.7.101:6080/vnc_auto.html vncserver_proxyclient_address=10.0.1.4

vncserver_listen=10.0.1.4

Compute Config:

[DEFAULT] iscsi_helper=tgtadm ec2_private_dns_show_ip=True volumes_path=/var/lib/nova/volumes

[DEFAULT]

LOGS/STATE

verbose=True logdir=/var/log/nova state_path=/var/lib/nova lock_path=/run/lock/nova

AUTHENTICATION

auth_strategy=keystone

SCHEDULER

scheduler_driver=nova.scheduler.multi.MultiScheduler compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

CINDER

volume_api_class=nova.volume.cinder.API

DATABASE

sql_connection=mysql://novaUser:novaPass@10.0.1.4/nova

COMPUTE

libvirt_type=kvm libvirt_use_virtio_for_bridges=True start_guests_on_host_boot=True resume_guests_state_on_host_boot=True api_paste_config=/etc/nova/api-paste.ini allow_admin_api=True use_deprecated_auth=False nova_url=http://10.0.1.4:8774/v1.1/ root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

APIS

ec2_host=10.0.1.4 ec2_url=http://10.0.1.4:8773/services/Cloud keystone_ec2_url=http://10.0.1.4:5000/v2.0/ec2tokens s3_host=10.0.1.4 cc_host=10.0.1.4 metadata_host=10.0.1.4 enabled_apis=metadata

RABBITMQ

rabbit_host=10.0.1.4

GLANCE

image_service=nova.image.glance.GlanceImageService glance_api_servers=10.0.1.4:9292

NETWORK

network_manager=nova.network.manager.FlatDHCPManager force_dhcp_release=True dhcpbridge_flagfile=/etc/nova/nova.conf dhcpbridge=/usr/bin/nova-dhcpbridge firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver public_interface=p255p2 flat_interface=p255p1 flat_network_bridge=br100 fixed_range=10.0.1.129/25 network_size=128 flat_network_dhcp_start=10.0.1.129 flat_injected=False connection_type=libvirt multi_host=True

NOVNC CONSOLE

novnc_enabled=True novncproxy_base_url=http://7.7.7.101:6080/vnc_auto.html vncserver_proxyclient_address=10.0.1.2

vncserver_listen=10.0.1.2

Controller Interfaces:

This file describes the network interfaces available on your system

and how to activate them. For more information, see interfaces(5).

The loopback network interface

auto lo iface lo inet loopback ... (more)

edit flag offensive delete link more
0

answered 2013-01-27 22:25:16 -0500

keith-tobin gravatar image

I need to see config files for controller and compute, I also want to see you 'interfaces' file.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-01-25 21:29:29 -0500

Seen: 75 times

Last updated: Jan 30 '13