I've read a ton about this metadata service issue on multinode environments and nothing seems to quite hit the nail into the coffin.

I have a true multinode configuration (separated controller, network, compute, and storage nodes).

I've read in the admin guide the following.
For instances to reach the metadata service, the nova-network service must configure iptables to NAT port 80 of the 169.254.169.254 address to the IP address specified in metadata_host (this defaults to $my_ip, which is the IP address of the nova-network service) and port specified in metadata_port (which defaults to 8775) in /etc/nova/nova.conf.

However I'm not using nova-network I'm using neutron.

The install guide does not seem to cover this either, outside of just point the metadata server to the controller's IP in the nova and neutron configs.

However when I boot up a cirros image, it still reaches out to 169.254.169.254 which seems to be a magic number over an actual configuration item.

Do I still need the NAT? If yes, please show the correct command per the juno neutron and juno setup. I've found one I think for nova-network through a bunch of posts. If no, I'm kind of at a loss in direction at this point.

I've also tried this instruction per the same admin guide (even though again I'm not using nova-netowork) The default Compute service settings assume that nova-network and nova-api are running on the same host. If this is not the case, in the /etc/nova/nova.conf file on the host running nova-network, set the metadata_host configuration option to the IP address of the host where nova-api is running.

The actual error as follows. *As shown I get a private IP. I can also ping my external net. I can however on the controller hit http://controller:8775 and get the listing of dates. However when I try to do anything more than that with the instance id from the dashboard, I keep getting malformed request.

 Sending discover...
 Sending select for 192.168.1.22...
 Lease of 192.168.1.22 obtained, lease time 86400
 cirros-ds 'net' up at 1.01
 checking http://169.254.169.254/2009-04-04/instance-id
 failed 1/20: up 1.01. request failed

Here is the ip netns iptables

ip netns exec qrouter-3e7bfe72-d4d7-4f7d-a3ea-4d22615cc19a iptables -S -t nat

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 202.0.113.105/32 -j DNAT --to-destination 192.168.1.22
-A neutron-l3-agent-OUTPUT -d 202.0.113.106/32 -j DNAT --to-destination 192.168.1.23
-A neutron-l3-agent-OUTPUT -d 202.0.113.104/32 -j DNAT --to-destination 192.168.1.6
-A neutron-l3-agent-OUTPUT -d 202.0.113.102/32 -j DNAT --to-destination 192.168.1.2
-A ...