Ask Your Question
0

No connection to metadata service - nova-network

asked 2013-12-12 06:46:53 -0500

Mathias Ewald gravatar image

updated 2014-01-02 14:26:28 -0500

smaffulli gravatar image

Hi, I just ran through the installation guide for Havana on Ubuntu 12.04 and got stuck at the point where I should be able to login to my instance using SSH public key authentication. I could find out, that from within the VM (cirros) a wget to the URL is not possible. I received a connection timeout. So I found out that a couple more options in nova.conf on the controller system were necessary. But as it still doent work, I figure something else / more is wrong.

on the controller node

root@controller:~# netstat -nap | grep 8775 tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 938/python 
root@controller:~# 
root@controller:~# iptables-save | grep 169 
root@controller:~#

On the compute node:

root@compute1:~# iptables-save | grep 169 -A 
nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.0.10:8775 root@compute1:~#

So the netfilter rule seems to be there - cool! From the VM I can ping 169.254.169.254 but wget does not work on it. From the VM I can ping 169.254.169.254 but wget does not work on it.

controller: nova.conf

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
rpc_backend=nova.rpc.impl_kombu
rabbit_host = localhost
rabbit_port = 5672
rabbit_use_ssl = false
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
my_ip=192.168.0.10
vncserver_listen=192.168.0.10
vncserver_proxyclient_address=192.168.0.10
auth_strategy=keystone
metadata_host=192.168.0.10
metadata_listen=0.0.0.0
metadata_listen_port=8775
metadata_manager=nova.api.manager.MetadataManager
[database]
connection = mysql://nova:nova@controller/nova
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = vXLT1234!

compute1: nova.conf

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
rpc_backend=nova.rpc.impl_kombu
rabbit_host = controller
rabbit_port = 5672
rabbit_use_ssl = false
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
auth_strategy=keystone
my_ip=192.168.0.11
vnc_enabled=True
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=192.168.0.11
novncproxy_base_url=http://controller:6080/vnc_auto.html
glance_host=controller

metadata_host=192.168.0.10
metadata_port=8775
metadata_manager=nova.api.manager.MetadataManager

network_manager=nova.network.manager.FlatDHCPManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
network_size=254
allow_same_net_traffic=False
multi_host=True
send_arp_for_ha=True
share_dhcp_address=True
force_dhcp_release=True
flat_network_bridge=br100
flat_interface=eth1
public_interface=eth1

[database]
connection = mysql://nova:nova@controller/nova

From the CirrOS Instance, this happens calling wget:

$ wget http://169.254.169.254
Connecting to 169.254.169.254 (169.254.169.254:80)
wget ...
(more)
edit retag flag offensive close merge delete

Comments

Can you confirm that your CirrOS instance is receiving an IP from DHCP? If so, can you try removing all metadata_* settings from both servers and restart all nova-* services? The default metadata_* settings should allow things to just work for multinode environments.

jtopjian gravatar imagejtopjian ( 2013-12-12 09:50:48 -0500 )edit

Can you give futher assistance with this? How would I confirm the IP comes from DHCP?

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-12 10:33:45 -0500 )edit

Just a simple "ip a" at the command prompt will do it. If the eth0 interface has an IP, then DHCP communication is working.

jtopjian gravatar imagejtopjian ( 2013-12-12 14:01:03 -0500 )edit

Yep, IP is there - I can SSH in providing username and password (key auth does not work).

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-13 02:25:04 -0500 )edit

3 answers

Sort by ยป oldest newest most voted
1

answered 2013-12-23 12:54:36 -0500

ajaya gravatar image

updated 2014-01-02 14:28:05 -0500

smaffulli gravatar image

Please install nova-api in the compute node and make sure the /etc/nova/nova.conf file has "enabled_apis=ec2,osapi_compute,metadata" entry.

As per documentation, if you are running multi_host true and using nova-networking, all compute nodes must have nova-api-metadata running.

I will submit a bug to fix the docs.

On Compute node

apt-get install nova-api

/etc/nova/nova.conf

enabled_apis=ec2,osapi_compute,metadata
edit flag offensive delete link more

Comments

Hi, I believe to see some inconsistency in your answer above. You are saying " all compute nodes must have nova-api-metadata running" before telling me to run "apt-get install nova-api" on the compute node. Shouldn't that be "apt-get install nova-api-metadata" instead? Reading this "The nova-api-metadata service is generally only used when running in multi-host mode" I thought running it wasn't a general requirement when in multi host mode but an option that would maybe increase performance or something...

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-25 05:23:19 -0500 )edit

I just installed nova-api-metadata on the compute node. Please check http://pastebin.com/ieDj4PwJ to see current behavior. The instance cannot reach metadata service during boot time yet, as it responds to port 8775, not 80.

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-25 05:44:04 -0500 )edit
1

answered 2013-12-16 15:11:34 -0500

jtopjian gravatar image

Can you try removing the metadata_* options from both servers' nova.conf file?

The default values of those options when they are not specified should be sufficient to get metadata working in most environments.

edit flag offensive delete link more

Comments

Hi, I removed all lines as you said and restarted all nova-* services. I am still getting a connection refused for a wget http://169.254.169.254 from one of the instances and there is still not netfilter rule for the forwarding :(

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-18 10:07:28 -0500 )edit

I'm sorry to hear you're still having issues. Can you confirm that something is listening on port 8775 (netstat -nap | grep 8775)? And if you do "iptables-save | grep 169", you get no results, correct?

jtopjian gravatar imagejtopjian ( 2013-12-18 20:48:02 -0500 )edit

Hi, this is what I see on the controller node: root@controller:~# netstat -nap | grep 8775 tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 938/python root@controller:~# root@controller:~# iptables-save | grep 169 root@controller:~# On the compute node: root@compute1:~# iptables-save | grep 169 -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.0.10:8775 root@compute1:~# So the netfilter rule seems to be there - cool! From the VM I can ping 169.254.169.254 but wget does not work on it.

Mathias Ewald gravatar imageMathias Ewald ( 2013-12-19 02:53:25 -0500 )edit

hm. Unfortunately I'm not sure where to go from here. I don't have a lot of experience with nova-network multi-node environments. If you do a google search for "nova-api-metadata", you'll see results for something that should be running in that type of environment. Not sure if that is deprecated...

jtopjian gravatar imagejtopjian ( 2013-12-19 12:45:50 -0500 )edit

I can confirm the same problem following the Havana on Ubuntu guide. I see that the metadata service running by accessing the curl http://ace:8775 which outputs ============ 1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 2009-04-04 latest ========= But I checked my compute nodes and I see iptables forwards iptables -t nat -L -v | grep -n3 169.254.169.254 46- 47-Chain nova-network-PREROUTING (1 references) 48- pkts bytes target prot opt in out source destination 49: 0 0 DNAT tcp -- any any anywhere 169.254.169.254 tcp dpt:http to:192.168.0.21:8775 50- 51-Chain nova-network-float-snat (1 references) 52- pkts bytes target prot opt in out source destination

ajaya gravatar imageajaya ( 2013-12-21 19:02:22 -0500 )edit
-1

answered 2013-12-12 11:57:10 -0500

Mathias Ewald gravatar image

updated 2014-01-09 14:01:01 -0500

Hi, I could gather some more information: According to the guide according to which I was installing OpenStack with KVM on Ubuntu 12.04, the controller node was supposed to get an IP address in the 10.0.0.0/24 network together with the instances and the compute node. Sniffing on the controller node's eth0 interface (192.168.0.10) I could see incoming requests from 10.0.0.2 (my instance IP) to 192.168.0.10:8775, BUT: not responses. I figured that the existence of eth1 (10.0.0.10/24) on the controller seemed to confuse the system somehow. Shutting this interface down, I could suddenly see requests and responses from and to my instance:

root@controller:~# tcpdump -n -i eth0 net 10.0.0.0/24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:24:08.325058 IP 10.0.0.2.53182 > 192.168.0.10.8775: Flags [S], seq 1901335267, win 14600, options [mss 1460,sackOK,TS val 60356 ecr 0,nop,wscale 3], length 0
17:24:08.325143 IP 192.168.0.10.8775 > 10.0.0.2.53182: Flags [S.], seq 1763350780, ack 1901335268, win 14480, options [mss 1460,sackOK,TS val 1318617 ecr 60356,nop,wscale 7], length 0
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
root@controller:~#

Now the response goes to 10.0.0.2 out interface eth0. But my instance cannot see the response and keeps sending requests. Does that help anyone to help me? :D

compute node: /etc/nova/nova.conf

[DEFAULT]
state_path=/var/lib/nova
lock_path=/var/lock/nova
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
debug=False
my_ip=192.168.0.11

# API
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=ec2,osapi_compute,metadata

# LOGGING
logdir=/var/log/nova

# NETWORKING
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
libvirt_use_virtio_for_bridges=True
network_manager=nova.network.manager.FlatDHCPManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
force_dhcp_release=True
flat_network_bridge=br100
flat_interface=eth1
public_interface=eth0
multi_host=True
dhcp_domain=vxlt.invalid
flat_network_dns=8.8.8.8
gateway=10.0.0.1
network_size=256
allow_same_net_traffic=False
send_arp_for_ha=True


# QUEUE
rabbit_host = controller
rabbit_port = 5672
rabbit_use_ssl = false
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /

#AUTH
auth_strategy=keystone

# HYPERVISOR
compute_driver=libvirt.LibvirtDriver

# VNC
vnc_enabled=True
vncserver_listen=192.168.0.11
vncserver_proxyclient_address=192.168.0.10
novncproxy_base_url=http://controller:6080/vnc_auto.html

# GLANCE
glance_host=controller

# METADATA
metadata_host=192.168.0.10
metadata_port=8775
metadata_listen=0.0.0.0
metadata_listen_port=8775

# DATABASE
[database]
connection = mysql://nova:nova@controller/nova

# KEYSTONE
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = vXLT1234!

EDIT: I can see that my instance can ping 192.168.0.1 and any IP on the internet easily. From this I know that NAT must be working. But the fact that when I ping 192.168.0.10 it responds to 10.0.0.2 tells ... (more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2013-12-12 06:46:53 -0500

Seen: 5,880 times

Last updated: Jan 09 '14