Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

juno neutron nova metadata network configuration issue (request failed)

I've read a ton about this metadata service issue on multinode environments and nothing seems to quite hit the nail into the coffin.

I have a true multinode configuration (separated controller, network, compute, and storage nodes).

I've read in the admin guide the following.
For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

However I'm not using nova-network I'm using neutron.

The install guide does not seem to cover this either, outside of just point the metadata server to the controller's IP in the nova and neutron configs.

However when I boot up a cirros image, it still reaches out to 169.254.169.254 which seems to be a magic number over an actual configuration item.

Do I still need the NAT? If yes, please show the correct command per the juno neutron and juno setup. I've found one I think for nova-network through a bunch of posts. If no, I'm kind of at a loss in direction at this point.

I've also tried this instruction per the same admin guide (even though again I'm not using nova-netowork) The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

The actual error as follows. *As shown I get a private IP. I can also ping my external net. I can however on the controller hit http://controller:8775 and get the listing of dates. However when I try to do anything more than that with the instance id from the dashboard, I keep getting malformed request. Sending discover... Sending select for 192.168.1.22... Lease of 192.168.1.22 obtained, lease time 86400 cirros-ds 'net' up at 1.01 checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 1.01. request failed

juno neutron nova metadata network configuration issue (request failed)

I've read a ton about this metadata service issue on multinode environments and nothing seems to quite hit the nail into the coffin.

I have a true multinode configuration (separated controller, network, compute, and storage nodes).

I've read in the admin guide the following.
For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

However I'm not using nova-network I'm using neutron.

The install guide does not seem to cover this either, outside of just point the metadata server to the controller's IP in the nova and neutron configs.

However when I boot up a cirros image, it still reaches out to 169.254.169.254 which seems to be a magic number over an actual configuration item.

Do I still need the NAT? If yes, please show the correct command per the juno neutron and juno setup. I've found one I think for nova-network through a bunch of posts. If no, I'm kind of at a loss in direction at this point.

I've also tried this instruction per the same admin guide (even though again I'm not using nova-netowork) The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

The actual error as follows. *As shown I get a private IP. I can also ping my external net. I can however on the controller hit http://controller:8775 and get the listing of dates. However when I try to do anything more than that with the instance id from the dashboard, I keep getting malformed request. request.

 Sending discover...
 Sending select for 192.168.1.22...
 Lease of 192.168.1.22 obtained, lease time 86400
 cirros-ds 'net' up at 1.01
 checking http://169.254.169.254/2009-04-04/instance-id
 failed 1/20: up 1.01. request failed

failed

juno neutron nova metadata network configuration issue (request failed)

I've read a ton about this metadata service issue on multinode environments and nothing seems to quite hit the nail into the coffin.

I have a true multinode configuration (separated controller, network, compute, and storage nodes).

I've read in the admin guide the following.
For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

However I'm not using nova-network I'm using neutron.

The install guide does not seem to cover this either, outside of just point the metadata server to the controller's IP in the nova and neutron configs.

However when I boot up a cirros image, it still reaches out to 169.254.169.254 which seems to be a magic number over an actual configuration item.

Do I still need the NAT? If yes, please show the correct command per the juno neutron and juno setup. I've found one I think for nova-network through a bunch of posts. If no, I'm kind of at a loss in direction at this point.

I've also tried this instruction per the same admin guide (even though again I'm not using nova-netowork) The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

The actual error as follows. *As shown I get a private IP. I can also ping my external net. I can however on the controller hit http://controller:8775 and get the listing of dates. However when I try to do anything more than that with the instance id from the dashboard, I keep getting malformed request.

 Sending discover...
 Sending select for 192.168.1.22...
 Lease of 192.168.1.22 obtained, lease time 86400
 cirros-ds 'net' up at 1.01
 checking http://169.254.169.254/2009-04-04/instance-id
 failed 1/20: up 1.01. request failed

Here is the ip netns iptables

ip netns exec qrouter-3e7bfe72-d4d7-4f7d-a3ea-4d22615cc19a iptables -S -t nat

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 202.0.113.105/32 -j DNAT --to-destination 192.168.1.22
-A neutron-l3-agent-OUTPUT -d 202.0.113.106/32 -j DNAT --to-destination 192.168.1.23
-A neutron-l3-agent-OUTPUT -d 202.0.113.104/32 -j DNAT --to-destination 192.168.1.6
-A neutron-l3-agent-OUTPUT -d 202.0.113.102/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-OUTPUT -d 202.0.113.103/32 -j DNAT --to-destination 192.168.1.5
-A neutron-l3-agent-POSTROUTING ! -i qg-f9abab5e-67 ! -o qg-f9abab5e-67 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 202.0.113.105/32 -j DNAT --to-destination 192.168.1.22
-A neutron-l3-agent-PREROUTING -d 202.0.113.106/32 -j DNAT --to-destination 192.168.1.23
-A neutron-l3-agent-PREROUTING -d 202.0.113.104/32 -j DNAT --to-destination 192.168.1.6
-A neutron-l3-agent-PREROUTING -d 202.0.113.102/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-PREROUTING -d 202.0.113.103/32 -j DNAT --to-destination 192.168.1.5
-A neutron-l3-agent-float-snat -s 192.168.1.22/32 -j SNAT --to-source 202.0.113.105
-A neutron-l3-agent-float-snat -s 192.168.1.23/32 -j SNAT --to-source 202.0.113.106
-A neutron-l3-agent-float-snat -s 192.168.1.6/32 -j SNAT --to-source 202.0.113.104
-A neutron-l3-agent-float-snat -s 192.168.1.2/32 -j SNAT --to-source 202.0.113.102
-A neutron-l3-agent-float-snat -s 192.168.1.5/32 -j SNAT --to-source 202.0.113.103
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -s 192.168.1.0/24 -j SNAT --to-source 202.0.113.101
-A neutron-postrouting-bottom -j neutron-l3-agent-snat

juno neutron nova metadata network configuration issue (request failed)

I've read a ton about this metadata service issue on multinode environments and nothing seems to quite hit the nail into the coffin.

I have a true multinode configuration (separated controller, network, compute, and storage nodes).

I've read in the admin guide the following.
For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

However I'm not using nova-network I'm using neutron.

The install guide does not seem to cover this either, outside of just point the metadata server to the controller's IP in the nova and neutron configs.

However when I boot up a cirros image, it still reaches out to 169.254.169.254 which seems to be a magic number over an actual configuration item.

Do I still need the NAT? If yes, please show the correct command per the juno neutron and juno setup. I've found one I think for nova-network through a bunch of posts. If no, I'm kind of at a loss in direction at this point.

I've also tried this instruction per the same admin guide (even though again I'm not using nova-netowork) The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

The actual error as follows. *As shown I get a private IP. I can also ping my external net. I can however on the controller hit http://controller:8775 and get the listing of dates. However when I try to do anything more than that with the instance id from the dashboard, I keep getting malformed request.

 Sending discover...
 Sending select for 192.168.1.22...
 Lease of 192.168.1.22 obtained, lease time 86400
 cirros-ds 'net' up at 1.01
 checking http://169.254.169.254/2009-04-04/instance-id
 failed 1/20: up 1.01. request failed

Here is the ip netns iptables

ip netns exec qrouter-3e7bfe72-d4d7-4f7d-a3ea-4d22615cc19a iptables -S -t nat

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 202.0.113.105/32 -j DNAT --to-destination 192.168.1.22
-A neutron-l3-agent-OUTPUT -d 202.0.113.106/32 -j DNAT --to-destination 192.168.1.23
-A neutron-l3-agent-OUTPUT -d 202.0.113.104/32 -j DNAT --to-destination 192.168.1.6
-A neutron-l3-agent-OUTPUT -d 202.0.113.102/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-OUTPUT -d 202.0.113.103/32 -j DNAT --to-destination 192.168.1.5
-A neutron-l3-agent-POSTROUTING ! -i qg-f9abab5e-67 ! -o qg-f9abab5e-67 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 202.0.113.105/32 -j DNAT --to-destination 192.168.1.22
-A neutron-l3-agent-PREROUTING -d 202.0.113.106/32 -j DNAT --to-destination 192.168.1.23
-A neutron-l3-agent-PREROUTING -d 202.0.113.104/32 -j DNAT --to-destination 192.168.1.6
-A neutron-l3-agent-PREROUTING -d 202.0.113.102/32 -j DNAT --to-destination 192.168.1.2
-A neutron-l3-agent-PREROUTING -d 202.0.113.103/32 -j DNAT --to-destination 192.168.1.5
-A neutron-l3-agent-float-snat -s 192.168.1.22/32 -j SNAT --to-source 202.0.113.105
-A neutron-l3-agent-float-snat -s 192.168.1.23/32 -j SNAT --to-source 202.0.113.106
-A neutron-l3-agent-float-snat -s 192.168.1.6/32 -j SNAT --to-source 202.0.113.104
-A neutron-l3-agent-float-snat -s 192.168.1.2/32 -j SNAT --to-source 202.0.113.102
-A neutron-l3-agent-float-snat -s 192.168.1.5/32 -j SNAT --to-source 202.0.113.103
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -s 192.168.1.0/24 -j SNAT --to-source 202.0.113.101
-A neutron-postrouting-bottom -j neutron-l3-agent-snat

ANSWER: Just figured it out, it looks like by default the metadata configuration file has the auth services uncommented so my added configuration was being bashed by the default configuration (localhost). I found that issue in one of the log files. No iptables NAT additional steps were needed. Might be a good thing to have those commented back out as part of the Juno package as all the other configs act as if you have to uncomment something for it to become active.