Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Network configuration / Can't ping or ssh to our instances

We have been trying to deploy nova (Bexar) for the past week on a single machine. We can install and configure nova on a fresh Ubuntu 10.10 server and even successfully start VMs but we have not been able to connect to those VMs. Neither ping nor ssh work (yes, we have done the euca-authorize steps).

We've tried all of the steps from the http://wiki.openstack.org/TroubleshootingNova. We've tried asking other questions, https://answers.launchpad.net/nova/+question/145062 and https://answers.launchpad.net/nova/+question/145063 (no replies). We've tried going on IRC, giving bits and pieces of information about our problems and getting good troubleshooting tips in return but, ultimately, nothing that has worked for us. We've read about a similar problems: https://answers.launchpad.net/nova/+question/141960 was already solved and the patch merged in Bexar and https://answers.launchpad.net/nova/+question/145567 doesn't seem to be relevant anymore (can't find --flat_interface reference in any of the nova-<service> --help).

So, out of desperation, here is all of the information about our configuration (IP addresses changed) in the hopes that someone can pinpoint what the problem is.

Is our networking configured properly for Nova? Why can't we ping or ssh to our instances?

Any answers or network troubleshooting tips are greatly appreciated!

** Installation Architecture ** By design our network architecture very closely resembles the Nova example installation architecture.

http://docs.openstack.org/openstack-compute/admin/content/figures/NOVA_install_arch.png

Keep in mind that, at the moment, we are only trying to install to a single machine.

** Installation Procedure ** We are doing the scripted installation (http://docs.openstack.org/openstack-compute/admin/content/ch03s02.html#d5e161). When we run nova-CC-install-v1.1.sh here are the values we input:

S3 Host IP (Default is 99.99.99.185 -- ENTER to accept): RabbitMQ Host IP (Default is 99.99.99.185 -- ENTER to accept): Cloud Controller Host IP (Default is 99.99.99.185 -- ENTER to accept): mySQL Host IP (Default is 99.99.99.185 -- ENTER to accept):

Controller network range for ALL projects (normally x.x.x.x/12):192.168.0.0/16 Total amount of usable IPs for ALL projects:64 Nova project user name:admin Nova project name:admin-project Desired network + CIDR for project (normally x.x.x.x/24):192.168.0.0/24 How many networks for project:1 How many availible IPs per project network:8

Please enter your local server IP (Default is 99.99.99.185 -- ENTER to accept): Please enter your broadcast IP (Default is 99.99.99.191 -- ENTER to accept): Please enter your netmask (Default is 255.255.255.224 -- ENTER to accept): Please enter your gateway (Default is 99.99.99.161 -- ENTER to accept): Please enter your default nameserver (Default is 99.99.99.194 -- ENTER to accept):

** Installation Output ** Installing packages

#

Installing package 'python-software-properties' ... ok Installing package 'python-mysqldb' ... ok Installing package 'mysql-server' ... ok Installing package 'nova-api' ... ok Installing package 'nova-network' ... ok Installing package 'nova-objectstore' ... ok Installing package 'nova-scheduler' ... ok Installing package 'nova-compute' ... ok Installing package 'unzip' ... ok Installing package 'vim' ... ok Installing package 'euca2ools' ... ok Finalizing mySQL setup

#

...done...

Setting up Nova configuration files

#

Initializing database ...done...

...done...

Generating Nova credentials

#

...done...

Creating br100 bridge device

#

...done...

Restarting networking * Reconfiguring network interfaces... ssh stop/waiting ssh start/running, process 4472 ssh stop/waiting ssh start/running, process 4500 ssh stop/waiting ssh start/running, process 4562 [ OK ] ...done...

###########NOTE#####NOTE#####NOTE#####NOTE#####NOTE#####NOTE

Be sure to source your credential file into your environment after config changes

#####################e.g. source /root/creds/novarc

Ensure all five Nova services are running

#

root 3763 0.0 0.0 12228 1884 pts/0 S+ 17:15 0:00 /bin/bash ./nova-CC-install-v1.1.sh nova 4623 0.0 0.0 35632 1264 ? Ss 17:17 0:00 su -c nova-network --flagfile=/etc/nova/nova.conf nova nova 4625 6.8 0.3 105064 27180 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-network --flagfile=/etc/nova/nova.conf nova 4637 0.0 0.0 35632 1268 ? Ss 17:17 0:00 su -c nova-compute --flagfile=/etc/nova/nova.conf nova nova 4639 13.2 0.4 157360 38864 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-compute --flagfile=/etc/nova/nova.conf nova 4662 0.0 0.0 35632 1264 ? Ss 17:17 0:00 su -c nova-api --flagfile=/etc/nova/nova.conf nova nova 4663 16.5 0.2 80224 23948 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf nova 4676 15.5 0.3 96716 24604 ? S 17:17 0:00 /usr/bin/python /usr/bin/nova-objectstore --uid 106 --gid 65534 --pidfile /var/run/nova/nova-objectstore.pid --flagfile=/etc/nova/nova.conf --nodaemon --logfile=/var/log/nova/nova-objectstore.log nova 4692 0.0 0.0 35632 1268 ? Ss 17:17 0:00 su -c nova-scheduler --flagfile=/etc/nova/nova.conf nova root 4694 0.0 0.0 8952 888 pts/0 S+ 17:17 0:00 grep -i nova nova 4695 0.0 0.0 27948 4408 ? R 17:17 0:00 /usr/bin/python /usr/bin/nova-scheduler --flagfile=/etc/nova/nova.conf

Setup default ICMP and SSH access to your future VMs

#

Allowing ping and SSH to your running instances ...done...

#

You /MUST/ re-source your 'novarc' to use the API commands since the

script cannot pass the source information out of it's own process

#

dnsmasq: no process found

The next thing you are going to want to do it get a VM to test with. You can find a test VM how-to, and read more about custom image creation at "http://nova.openstack.org/adminguide/multi.node.install.html" and "http://wiki.openstack.org/GettingImages

Enjoy your new private cloud!

** Running Instances Procedure ** We followed the instructions from Starting Images (http://docs.openstack.org/openstack-compute/admin/content/ch05s01.html). The instance runs and here is the result of euca-describe-instances,

RESERVATION r-et60ec7l admin-project default INSTANCE i-00000001 ami-3if67tvi 192.168.0.2 192.168.0.2 running admin (admin-project, ubuntu) 0 m1.tiny 2011-02-17 22:34:38 nova

** Ping and SSH ** root@ubuntu:~# ping 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. ^C --- 192.168.0.2 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3007ms

root@ubuntu:~# ssh 192.168.0.2 ssh: connect to host 192.168.0.2 port 22: Connection timed out

** traceroute ** root@ubuntu:~# traceroute 192.168.0.2 traceroute to 192.168.0.2 (192.168.0.2), 30 hops max, 60 byte packets send: Operation not permitted

** euca-get-console-output ** As described in these other questions, https://answers.launchpad.net/nova/+question/145062 and https://answers.launchpad.net/nova/+question/145063

** Logs ** It is worthwhile to note there are no ERRORs in the logs.

** nova.conf ** --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --verbose --s3_host=99.99.99.185 --rabbit_host=99.99.99.185 --cc_host=99.99.99.185 --ec2_url=http://99.99.99.185:8773/services/Cloud --fixed_range=192.168.0.0/16 --network_size=64 --FAKE_subdomain=ec2 --routing_source_ip=99.99.99.185 --verbose --sql_connection=mysql://root:nova@99.99.99.185/nova --network_manager=nova.network.manager.FlatManager

** ifconfig ** br100 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9a
inet addr:99.99.99.185 Bcast:99.99.99.191 Mask:255.255.255.224 inet6 addr: fe80::21b:78ff:fed2:ab9a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:145536 errors:0 dropped:0 overruns:0 frame:0 TX packets:94481 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:380460510 (380.4 MB) TX bytes:6805858 (6.8 MB)

eth0 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9a
inet6 addr: fe80::21b:78ff:fed2:ab9a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:271190 errors:0 dropped:0 overruns:0 frame:0 TX packets:95283 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:391928462 (391.9 MB) TX bytes:7236250 (7.2 MB) Interrupt:16

eth1 Link encap:Ethernet HWaddr 00:1b:78:d2:ab:9b
inet addr:192.168.3.185 Bcast:192.168.255.255 Mask:255.255.0.0 inet6 addr: fe80::21b:78ff:fed2:ab9b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:49 errors:0 dropped:0 overruns:0 frame:0 TX packets:17 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3136 (3.1 KB) TX bytes:1418 (1.4 KB) Interrupt:17

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1020865 errors:0 dropped:0 overruns:0 frame:0 TX packets:1020865 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1738335671 (1.7 GB) TX bytes:1738335671 (1.7 GB)

virbr0 Link encap:Ethernet HWaddr 06:99:07:25:0b:9c
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::499:7ff:fe25:b9c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:468 (468.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:2b:32:49
inet6 addr: fe80::fc16:3eff:fe2b:3249/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:720 errors:0 dropped:0 overruns:0 frame:0 TX packets:1635 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:30456 (30.4 KB) TX bytes:98504 (98.5 KB)

** iptables-save **

Generated by iptables-save v1.4.4 on Thu Feb 17 18:27:59 2011

*nat :PREROUTING ACCEPT [276:35596] :OUTPUT ACCEPT [380:24818] :POSTROUTING ACCEPT [386:25062] :SNATTING - [0:0] -A PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 99.99.99.185:8773 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE -A POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -j ACCEPT -A POSTROUTING -s 10.0.0.0/8 -d 10.128.0.0/24 -j ACCEPT -A POSTROUTING -j SNATTING -A SNATTING -s 10.0.0.0/8 -j SNAT --to-source 99.99.99.185 COMMIT

Completed on Thu Feb 17 18:27:59 2011

Generated by iptables-save v1.4.4 on Thu Feb 17 18:27:59 2011

*filter :INPUT ACCEPT [643452:1525282513] :FORWARD ACCEPT [8:352] :OUTPUT ACCEPT [643396:1525477930] :nova-fallback - [0:0] :nova-inst-1 - [0:0] :nova-local - [0:0] :nova-sg-1 - [0:0] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -j nova-local -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -j nova-local -A nova-fallback -j DROP -A nova-inst-1 -m state --state INVALID -j DROP -A nova-inst-1 -m state --state RELATED,ESTABLISHED -j ACCEPT -A nova-inst-1 -j nova-sg-1 -A nova-inst-1 -s 192.168.0.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT -A nova-inst-1 -s 192.168.0.0/29 -j ACCEPT -A nova-inst-1 -j nova-fallback -A nova-local -d 192.168.0.2/32 -j nova-inst-1 -A nova-sg-1 -p icmp -j ACCEPT -A nova-sg-1 -p tcp -m tcp --dport 22 -j ACCEPT COMMIT

Completed on Thu Feb 17 18:27:59 2011

** brctl show ** bridge name bridge id STP enabled interfaces br100 8000.001b78d2ab9a no eth0 vnet0 virbr0 8000.000000000000 yes

** TL;DR ** Is our networking configured properly for Nova? Why can't we ping or ssh to our instances?