Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

metadata server is not reachable 404 error

Hello I created an Openstack newton cluster I have cirros and ubuntu VMs running in the cluster. I can ssh into cirros instance using cirros and cubswin:) . I can ping the 8.8.8.8 from the cirros instance and I can ssh into the cirros instance from any node in my network. But when I tried to ssh into ubuntu instance using mykey it is giving me error as below

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 141.30.74.132 [141.30.74.132] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 141.30.74.132:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oMaRnsZ9/l6vjz8hyudV6TjOaXG1n1Ytxu2f8/QsU50
debug1: Host '141.30.74.132' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I found that in the console.log of ubuntu instance I have a warning like below

[   15.417153] cloud-init[996]: Cloud-init v. 0.7.8 running 'init' at Wed, 14 Dec 2016 10:56:38 +0000. Up 14.36 seconds.
[   15.421593] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   15.424650] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.427664] cloud-init[996]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[   15.430657] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.433658] cloud-init[996]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[   15.436690] cloud-init[996]: ci-info: |   lo   | True |           ::1/128            |       .       |  host |         .         |
[   15.439663] cloud-init[996]: ci-info: |  ens3  | True |         192.168.1.8          | 255.255.255.0 |   .   | fa:16:3e:25:e8:ac |
[   15.442646] cloud-init[996]: ci-info: |  ens3  | True | fe80::f816:3eff:fe25:e8ac/64 |       .       |  link | fa:16:3e:25:e8:ac |
[   15.445698] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.448729] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[   15.451571] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.454385] cloud-init[996]: ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
[   15.457254] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.460141] cloud-init[996]: ci-info: |   0   |     0.0.0.0     | 192.168.1.1 |     0.0.0.0     |    ens3   |   UG  |
[   15.462981] cloud-init[996]: ci-info: |   1   | 169.254.169.254 | 192.168.1.1 | 255.255.255.255 |    ens3   |  UGH  |
[   15.465796] cloud-init[996]: ci-info: |   2   |   192.168.1.0   |   0.0.0.0   |  255.255.255.0  |    ens3   |   U   |
[   15.468643] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.471468] cloud-init[996]: 2016-12-14 10:56:39,954 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [404]
[   16.589183] cloud-init[996]: 2016-12-14 10:56:41,126 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [404]

my ip netns exec qrouter-baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c iptables -t nat -L is as below

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.1.5          anywhere             to:141.30.74.132
SNAT       all  --  192.168.1.6          anywhere             to:141.30.74.138

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  anywhere             anywhere             to:141.30.74.134
SNAT       all  --  anywhere             anywhere             mark match ! 0x2/0xffff ctstate DNAT to:141.30.74.134

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere             /* Perform source NAT on outgoing traffic. */

in nova-api.log it seems like metadata server is properly started..

2016-12-13 18:02:04.804 6920 INFO nova.wsgi [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] metadata listening on 0.0.0.0:8772
2016-12-13 18:02:04.805 6920 INFO oslo_service.service [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Starting 32 workers
2016-12-13 18:02:05.054 6920 WARNING oslo_config.cfg [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Option "rpc_backend" from group "DEFAULT" is deprecated for
2016-12-13 18:02:05.234 7905 INFO nova.metadata.wsgi.server [req-5990f777-630b-429c-b741-fd0b7ee2220f - - - - -] (7905) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.235 7906 INFO nova.metadata.wsgi.server [req-b081eaae-b009-45f4-90a9-191f36e0fecf - - - - -] (7906) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.267 7907 INFO nova.metadata.wsgi.server [req-e939ec30-1de1-4d04-80c8-99dba46aab3d - - - - -] (7907) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.281 7909 INFO nova.metadata.wsgi.server [req-14bffcd5-42a2-4588-a935-7afb7180ab02 - - - - -] (7909) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.312 7911 INFO nova.metadata.wsgi.server [req-9c783d6f-5910-4fa7-86a7-fb4ed4b52faf - - - - -] (7911) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.323 7915 INFO nova.metadata.wsgi.server [req-db8248a1-87d2-4125-bc9d-73e8d7837dab - - - - -] (7915) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.330 7927 INFO nova.metadata.wsgi.server [req-0c32a119-47e7-471a-9ed2-11acac4b214b - - - - -] (7927) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.335 7925 INFO nova.metadata.wsgi.server [req-fa4d0e5e-d7cf-4b1e-bc64-e4901d173847 - - - - -] (7925) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.338 7931 INFO nova.metadata.wsgi.server [req-e4109117-5b71-45a0-9dbb-edaa165bca59 - - - - -] (7931) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.344 7930 INFO nova.metadata.wsgi.server [req-20e06336-9671-4e6e-8d52-a1e664a74c6f - - - - -] (7930) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.348 7916 INFO nova.metadata.wsgi.server [req-6668dc9a-ccc4-47cc-ac01-76bea7540225 - - - - -] (7916) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.351 7919 INFO nova.metadata.wsgi.server [req-27625ad2-54e9-4add-80da-915550c809a5 - - - - -] (7919) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.352 7923 INFO nova.metadata.wsgi.server [req-6fcb8d8d-e166-4403-b071-50f438d5edb8 - - - - -] (7923) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.362 7934 INFO nova.metadata.wsgi.server [req-d101b3e2-89da-48ff-99ef-db8ec8d25482 - - - - -] (7934) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.377 7908 INFO nova.metadata.wsgi.server [req-65bc543d-14de-460b-8320-00b44dadde09 - - - - -] (7908) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.383 7936 INFO nova.metadata.wsgi.server [req-2c9a3635-79ea-4792-a851-3e01f0769cb2 - - - - -] (7936) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.397 7942 INFO nova.metadata.wsgi.server [req-b8912d2c-b175-4b3f-93b1-804bf10b89cb - - - - -] (7942) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.403 7913 INFO nova.metadata.wsgi.server [req-27d0fcaa-514f-4ac1-902b-012732a0333f - - - - -] (7913) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.416 7945 INFO nova.metadata.wsgi.server [req-2c4de1f7-7750-44c7-ba45-3be0c6502cd3 - - - - -] (7945) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.417 7940 INFO nova.metadata.wsgi.server [req-d56dc28b-cf01-4fc6-9fb0-1ad45c3f7d2b - - - - -] (7940) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.418 7921 INFO nova.metadata.wsgi.server [req-0e43662e-a41d-48c8-9ae0-18abe28f335b - - - - -] (7921) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.435 7952 INFO nova.metadata.wsgi.server [req-7b6e53de-4b45-48b5-ac6c-2f2dc882c783 - - - - -] (7952) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7957 INFO nova.metadata.wsgi.server [req-901e8816-e43c-4446-a081-9b5c7a2cabb5 - - - - -] (7957) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7954 INFO nova.metadata.wsgi.server [req-f1269c16-73ff-42f7-91f5-e99ace884c8f - - - - -] (7954) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.463 7956 INFO nova.metadata.wsgi.server [req-24bd6016-6e81-4e99-80df-e833f3620e01 - - - - -] (7956) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.479 7949 INFO nova.metadata.wsgi.server [req-06b7e743-1602-477d-9ad2-97fa356d973d - - - - -] (7949) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.480 7939 INFO nova.metadata.wsgi.server [req-109b03d7-3493-4d08-b6b2-2769f6977917 - - - - -] (7939) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.499 7963 INFO nova.metadata.wsgi.server [req-a5efa08c-1c54-4428-bfac-35d05cf7191c - - - - -] (7963) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.503 7946 INFO nova.metadata.wsgi.server [req-860da4fe-77c5-41f2-bca3-f0f475cfb82b - - - - -] (7946) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.504 7960 INFO nova.metadata.wsgi.server [req-2cc25bfb-31f6-48e1-9a12-93d6b44246a1 - - - - -] (7960) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.535 7948 INFO nova.metadata.wsgi.server [req-4804e557-1004-4b1a-9701-4941c4555324 - - - - -] (7948) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.546 7961 INFO nova.metadata.wsgi.server [req-aad8654e-4c4c-4a9a-b000-76be2f1ee562 - - - - -] (7961) wsgi starting up on http://0.0.0.0:8772

Please help .. Any help to solve this issue is really helpful..

metadata server is not reachable 404 error

Hello I created an Openstack newton cluster I have cirros and ubuntu VMs running in the cluster. I can ssh into cirros instance using cirros and cubswin:) . I can ping the 8.8.8.8 from the cirros instance and I can ssh into the cirros instance from any node in my network. But when I tried to ssh into ubuntu instance using mykey it is giving me error as below

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 141.30.74.132 [141.30.74.132] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 141.30.74.132:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oMaRnsZ9/l6vjz8hyudV6TjOaXG1n1Ytxu2f8/QsU50
debug1: Host '141.30.74.132' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I found that in the console.log of ubuntu instance I have a warning like below

[   15.417153] cloud-init[996]: Cloud-init v. 0.7.8 running 'init' at Wed, 14 Dec 2016 10:56:38 +0000. Up 14.36 seconds.
[   15.421593] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   15.424650] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.427664] cloud-init[996]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[   15.430657] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.433658] cloud-init[996]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[   15.436690] cloud-init[996]: ci-info: |   lo   | True |           ::1/128            |       .       |  host |         .         |
[   15.439663] cloud-init[996]: ci-info: |  ens3  | True |         192.168.1.8          | 255.255.255.0 |   .   | fa:16:3e:25:e8:ac |
[   15.442646] cloud-init[996]: ci-info: |  ens3  | True | fe80::f816:3eff:fe25:e8ac/64 |       .       |  link | fa:16:3e:25:e8:ac |
[   15.445698] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.448729] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[   15.451571] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.454385] cloud-init[996]: ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
[   15.457254] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.460141] cloud-init[996]: ci-info: |   0   |     0.0.0.0     | 192.168.1.1 |     0.0.0.0     |    ens3   |   UG  |
[   15.462981] cloud-init[996]: ci-info: |   1   | 169.254.169.254 | 192.168.1.1 | 255.255.255.255 |    ens3   |  UGH  |
[   15.465796] cloud-init[996]: ci-info: |   2   |   192.168.1.0   |   0.0.0.0   |  255.255.255.0  |    ens3   |   U   |
[   15.468643] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.471468] cloud-init[996]: 2016-12-14 10:56:39,954 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [404]
[   16.589183] cloud-init[996]: 2016-12-14 10:56:41,126 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [404]

my ip netns exec qrouter-baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c iptables -t nat -L -L is as below

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.1.5          anywhere             to:141.30.74.132
SNAT       all  --  192.168.1.6          anywhere             to:141.30.74.138

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  anywhere             anywhere             to:141.30.74.134
SNAT       all  --  anywhere             anywhere             mark match ! 0x2/0xffff ctstate DNAT to:141.30.74.134

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere             /* Perform source NAT on outgoing traffic. */

in nova-api.log it seems like metadata server is properly started..

2016-12-13 18:02:04.804 6920 INFO nova.wsgi [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] metadata listening on 0.0.0.0:8772
2016-12-13 18:02:04.805 6920 INFO oslo_service.service [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Starting 32 workers
2016-12-13 18:02:05.054 6920 WARNING oslo_config.cfg [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Option "rpc_backend" from group "DEFAULT" is deprecated for
2016-12-13 18:02:05.234 7905 INFO nova.metadata.wsgi.server [req-5990f777-630b-429c-b741-fd0b7ee2220f - - - - -] (7905) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.235 7906 INFO nova.metadata.wsgi.server [req-b081eaae-b009-45f4-90a9-191f36e0fecf - - - - -] (7906) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.267 7907 INFO nova.metadata.wsgi.server [req-e939ec30-1de1-4d04-80c8-99dba46aab3d - - - - -] (7907) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.281 7909 INFO nova.metadata.wsgi.server [req-14bffcd5-42a2-4588-a935-7afb7180ab02 - - - - -] (7909) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.312 7911 INFO nova.metadata.wsgi.server [req-9c783d6f-5910-4fa7-86a7-fb4ed4b52faf - - - - -] (7911) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.323 7915 INFO nova.metadata.wsgi.server [req-db8248a1-87d2-4125-bc9d-73e8d7837dab - - - - -] (7915) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.330 7927 INFO nova.metadata.wsgi.server [req-0c32a119-47e7-471a-9ed2-11acac4b214b - - - - -] (7927) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.335 7925 INFO nova.metadata.wsgi.server [req-fa4d0e5e-d7cf-4b1e-bc64-e4901d173847 - - - - -] (7925) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.338 7931 INFO nova.metadata.wsgi.server [req-e4109117-5b71-45a0-9dbb-edaa165bca59 - - - - -] (7931) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.344 7930 INFO nova.metadata.wsgi.server [req-20e06336-9671-4e6e-8d52-a1e664a74c6f - - - - -] (7930) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.348 7916 INFO nova.metadata.wsgi.server [req-6668dc9a-ccc4-47cc-ac01-76bea7540225 - - - - -] (7916) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.351 7919 INFO nova.metadata.wsgi.server [req-27625ad2-54e9-4add-80da-915550c809a5 - - - - -] (7919) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.352 7923 INFO nova.metadata.wsgi.server [req-6fcb8d8d-e166-4403-b071-50f438d5edb8 - - - - -] (7923) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.362 7934 INFO nova.metadata.wsgi.server [req-d101b3e2-89da-48ff-99ef-db8ec8d25482 - - - - -] (7934) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.377 7908 INFO nova.metadata.wsgi.server [req-65bc543d-14de-460b-8320-00b44dadde09 - - - - -] (7908) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.383 7936 INFO nova.metadata.wsgi.server [req-2c9a3635-79ea-4792-a851-3e01f0769cb2 - - - - -] (7936) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.397 7942 INFO nova.metadata.wsgi.server [req-b8912d2c-b175-4b3f-93b1-804bf10b89cb - - - - -] (7942) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.403 7913 INFO nova.metadata.wsgi.server [req-27d0fcaa-514f-4ac1-902b-012732a0333f - - - - -] (7913) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.416 7945 INFO nova.metadata.wsgi.server [req-2c4de1f7-7750-44c7-ba45-3be0c6502cd3 - - - - -] (7945) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.417 7940 INFO nova.metadata.wsgi.server [req-d56dc28b-cf01-4fc6-9fb0-1ad45c3f7d2b - - - - -] (7940) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.418 7921 INFO nova.metadata.wsgi.server [req-0e43662e-a41d-48c8-9ae0-18abe28f335b - - - - -] (7921) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.435 7952 INFO nova.metadata.wsgi.server [req-7b6e53de-4b45-48b5-ac6c-2f2dc882c783 - - - - -] (7952) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7957 INFO nova.metadata.wsgi.server [req-901e8816-e43c-4446-a081-9b5c7a2cabb5 - - - - -] (7957) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7954 INFO nova.metadata.wsgi.server [req-f1269c16-73ff-42f7-91f5-e99ace884c8f - - - - -] (7954) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.463 7956 INFO nova.metadata.wsgi.server [req-24bd6016-6e81-4e99-80df-e833f3620e01 - - - - -] (7956) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.479 7949 INFO nova.metadata.wsgi.server [req-06b7e743-1602-477d-9ad2-97fa356d973d - - - - -] (7949) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.480 7939 INFO nova.metadata.wsgi.server [req-109b03d7-3493-4d08-b6b2-2769f6977917 - - - - -] (7939) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.499 7963 INFO nova.metadata.wsgi.server [req-a5efa08c-1c54-4428-bfac-35d05cf7191c - - - - -] (7963) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.503 7946 INFO nova.metadata.wsgi.server [req-860da4fe-77c5-41f2-bca3-f0f475cfb82b - - - - -] (7946) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.504 7960 INFO nova.metadata.wsgi.server [req-2cc25bfb-31f6-48e1-9a12-93d6b44246a1 - - - - -] (7960) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.535 7948 INFO nova.metadata.wsgi.server [req-4804e557-1004-4b1a-9701-4941c4555324 - - - - -] (7948) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.546 7961 INFO nova.metadata.wsgi.server [req-aad8654e-4c4c-4a9a-b000-76be2f1ee562 - - - - -] (7961) wsgi starting up on http://0.0.0.0:8772

Please help .. Any help to solve this issue is really helpful..

metadata server is not reachable 404 error

Hello I created an Openstack newton cluster I have cirros and ubuntu VMs running in the cluster. I can ssh into cirros instance using cirros and cubswin:) . I can ping the 8.8.8.8 from the cirros instance and I can ssh into the cirros instance from any node in my network. But when I tried to ssh into ubuntu instance using mykey it is giving me error as below

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 141.30.74.132 [141.30.74.132] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 141.30.74.132:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oMaRnsZ9/l6vjz8hyudV6TjOaXG1n1Ytxu2f8/QsU50
debug1: Host '141.30.74.132' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I found that in the console.log of ubuntu instance I have a warning like below

[   15.417153] cloud-init[996]: Cloud-init v. 0.7.8 running 'init' at Wed, 14 Dec 2016 10:56:38 +0000. Up 14.36 seconds.
[   15.421593] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   15.424650] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.427664] cloud-init[996]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[   15.430657] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.433658] cloud-init[996]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[   15.436690] cloud-init[996]: ci-info: |   lo   | True |           ::1/128            |       .       |  host |         .         |
[   15.439663] cloud-init[996]: ci-info: |  ens3  | True |         192.168.1.8          | 255.255.255.0 |   .   | fa:16:3e:25:e8:ac |
[   15.442646] cloud-init[996]: ci-info: |  ens3  | True | fe80::f816:3eff:fe25:e8ac/64 |       .       |  link | fa:16:3e:25:e8:ac |
[   15.445698] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.448729] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[   15.451571] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.454385] cloud-init[996]: ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
[   15.457254] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.460141] cloud-init[996]: ci-info: |   0   |     0.0.0.0     | 192.168.1.1 |     0.0.0.0     |    ens3   |   UG  |
[   15.462981] cloud-init[996]: ci-info: |   1   | 169.254.169.254 | 192.168.1.1 | 255.255.255.255 |    ens3   |  UGH  |
[   15.465796] cloud-init[996]: ci-info: |   2   |   192.168.1.0   |   0.0.0.0   |  255.255.255.0  |    ens3   |   U   |
[   15.468643] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.471468] cloud-init[996]: 2016-12-14 10:56:39,954 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [404]
[   16.589183] cloud-init[996]: 2016-12-14 10:56:41,126 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [404]

my ip netns exec qrouter-baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c iptables -t nat -L is as below

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.1.5          anywhere             to:141.30.74.132
SNAT       all  --  192.168.1.6          anywhere             to:141.30.74.138

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  anywhere             anywhere             to:141.30.74.134
SNAT       all  --  anywhere             anywhere             mark match ! 0x2/0xffff ctstate DNAT to:141.30.74.134

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere             /* Perform source NAT on outgoing traffic. */

in nova-api.log it seems like metadata server is properly started..

2016-12-13 18:02:04.804 6920 INFO nova.wsgi [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] metadata listening on 0.0.0.0:8772
2016-12-13 18:02:04.805 6920 INFO oslo_service.service [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Starting 32 workers
2016-12-13 18:02:05.054 6920 WARNING oslo_config.cfg [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Option "rpc_backend" from group "DEFAULT" is deprecated for
2016-12-13 18:02:05.234 7905 INFO nova.metadata.wsgi.server [req-5990f777-630b-429c-b741-fd0b7ee2220f - - - - -] (7905) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.235 7906 INFO nova.metadata.wsgi.server [req-b081eaae-b009-45f4-90a9-191f36e0fecf - - - - -] (7906) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.267 7907 INFO nova.metadata.wsgi.server [req-e939ec30-1de1-4d04-80c8-99dba46aab3d - - - - -] (7907) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.281 7909 INFO nova.metadata.wsgi.server [req-14bffcd5-42a2-4588-a935-7afb7180ab02 - - - - -] (7909) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.312 7911 INFO nova.metadata.wsgi.server [req-9c783d6f-5910-4fa7-86a7-fb4ed4b52faf - - - - -] (7911) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.323 7915 INFO nova.metadata.wsgi.server [req-db8248a1-87d2-4125-bc9d-73e8d7837dab - - - - -] (7915) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.330 7927 INFO nova.metadata.wsgi.server [req-0c32a119-47e7-471a-9ed2-11acac4b214b - - - - -] (7927) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.335 7925 INFO nova.metadata.wsgi.server [req-fa4d0e5e-d7cf-4b1e-bc64-e4901d173847 - - - - -] (7925) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.338 7931 INFO nova.metadata.wsgi.server [req-e4109117-5b71-45a0-9dbb-edaa165bca59 - - - - -] (7931) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.344 7930 INFO nova.metadata.wsgi.server [req-20e06336-9671-4e6e-8d52-a1e664a74c6f - - - - -] (7930) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.348 7916 INFO nova.metadata.wsgi.server [req-6668dc9a-ccc4-47cc-ac01-76bea7540225 - - - - -] (7916) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.351 7919 INFO nova.metadata.wsgi.server [req-27625ad2-54e9-4add-80da-915550c809a5 - - - - -] (7919) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.352 7923 INFO nova.metadata.wsgi.server [req-6fcb8d8d-e166-4403-b071-50f438d5edb8 - - - - -] (7923) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.362 7934 INFO nova.metadata.wsgi.server [req-d101b3e2-89da-48ff-99ef-db8ec8d25482 - - - - -] (7934) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.377 7908 INFO nova.metadata.wsgi.server [req-65bc543d-14de-460b-8320-00b44dadde09 - - - - -] (7908) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.383 7936 INFO nova.metadata.wsgi.server [req-2c9a3635-79ea-4792-a851-3e01f0769cb2 - - - - -] (7936) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.397 7942 INFO nova.metadata.wsgi.server [req-b8912d2c-b175-4b3f-93b1-804bf10b89cb - - - - -] (7942) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.403 7913 INFO nova.metadata.wsgi.server [req-27d0fcaa-514f-4ac1-902b-012732a0333f - - - - -] (7913) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.416 7945 INFO nova.metadata.wsgi.server [req-2c4de1f7-7750-44c7-ba45-3be0c6502cd3 - - - - -] (7945) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.417 7940 INFO nova.metadata.wsgi.server [req-d56dc28b-cf01-4fc6-9fb0-1ad45c3f7d2b - - - - -] (7940) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.418 7921 INFO nova.metadata.wsgi.server [req-0e43662e-a41d-48c8-9ae0-18abe28f335b - - - - -] (7921) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.435 7952 INFO nova.metadata.wsgi.server [req-7b6e53de-4b45-48b5-ac6c-2f2dc882c783 - - - - -] (7952) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7957 INFO nova.metadata.wsgi.server [req-901e8816-e43c-4446-a081-9b5c7a2cabb5 - - - - -] (7957) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7954 INFO nova.metadata.wsgi.server [req-f1269c16-73ff-42f7-91f5-e99ace884c8f - - - - -] (7954) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.463 7956 INFO nova.metadata.wsgi.server [req-24bd6016-6e81-4e99-80df-e833f3620e01 - - - - -] (7956) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.479 7949 INFO nova.metadata.wsgi.server [req-06b7e743-1602-477d-9ad2-97fa356d973d - - - - -] (7949) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.480 7939 INFO nova.metadata.wsgi.server [req-109b03d7-3493-4d08-b6b2-2769f6977917 - - - - -] (7939) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.499 7963 INFO nova.metadata.wsgi.server [req-a5efa08c-1c54-4428-bfac-35d05cf7191c - - - - -] (7963) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.503 7946 INFO nova.metadata.wsgi.server [req-860da4fe-77c5-41f2-bca3-f0f475cfb82b - - - - -] (7946) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.504 7960 INFO nova.metadata.wsgi.server [req-2cc25bfb-31f6-48e1-9a12-93d6b44246a1 - - - - -] (7960) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.535 7948 INFO nova.metadata.wsgi.server [req-4804e557-1004-4b1a-9701-4941c4555324 - - - - -] (7948) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.546 7961 INFO nova.metadata.wsgi.server [req-aad8654e-4c4c-4a9a-b000-76be2f1ee562 - - - - -] (7961) wsgi starting up on http://0.0.0.0:8772

Please help .. Any help to solve this issue is really helpful..appreciated..

metadata server is not reachable 404 error

Hello I created an Openstack newton cluster I have cirros and ubuntu VMs running in the cluster. I can ssh into cirros instance using cirros and cubswin:) . I can ping the 8.8.8.8 from the cirros instance and I can ssh into the cirros instance from any node in my network. But when I run curl http://169.254.169.254 from inside the instance it is giving me

500 Internal Server Error

Remote metadata server experienced an internal server error.

My /var/log/neutron/neutron-metadata-agent.log has the below error also

2016-12-23 11:33:02.111 25058 DEBUG neutron.agent.metadata.agent [-] Request: GET / HTTP/1.0^M
Accept-Encoding: gzip, deflate^M
Content-Length: 0^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: Python-httplib2/0.9.1 (gzip)^M
X-Forwarded-For: 192.168.1.11^M
X-Neutron-Router-Id: baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c __call__ /usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py:86
2016-12-23 11:33:02.117 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 3996b10a61f7415ca8a4721c5ab008b0 exchange 'neutron' topic 'q-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-12-23 11:33:02.199 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 3996b10a61f7415ca8a4721c5ab008b0 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-12-23 11:33:02.201 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: f80218ffb0d2404281a02185d71d8125 exchange 'neutron' topic 'q-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-12-23 11:33:02.268 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f80218ffb0d2404281a02185d71d8125 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 90, in __call__
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent     return self._proxy_request(instance_id, tenant_id, req)
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 220, in _proxy_request
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent     raise Exception(_('Unexpected response code: %s') % resp.status)
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 300
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent
2016-12-23 11:33:02.285 25058 INFO eventlet.wsgi.server [-] 192.168.1.11,<local> - - [23/Dec/2016 11:33:02] "GET / HTTP/1.1" 500 229 0.173503

when I tried to ssh into ubuntu instance using mykey it is giving me error as below

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 141.30.74.132 [141.30.74.132] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 141.30.74.132:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oMaRnsZ9/l6vjz8hyudV6TjOaXG1n1Ytxu2f8/QsU50
debug1: Host '141.30.74.132' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I found that in the console.log of ubuntu instance I have a warning like below

[   15.417153] cloud-init[996]: Cloud-init v. 0.7.8 running 'init' at Wed, 14 Dec 2016 10:56:38 +0000. Up 14.36 seconds.
[   15.421593] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   15.424650] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.427664] cloud-init[996]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[   15.430657] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.433658] cloud-init[996]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[   15.436690] cloud-init[996]: ci-info: |   lo   | True |           ::1/128            |       .       |  host |         .         |
[   15.439663] cloud-init[996]: ci-info: |  ens3  | True |         192.168.1.8          | 255.255.255.0 |   .   | fa:16:3e:25:e8:ac |
[   15.442646] cloud-init[996]: ci-info: |  ens3  | True | fe80::f816:3eff:fe25:e8ac/64 |       .       |  link | fa:16:3e:25:e8:ac |
[   15.445698] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.448729] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[   15.451571] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.454385] cloud-init[996]: ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
[   15.457254] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.460141] cloud-init[996]: ci-info: |   0   |     0.0.0.0     | 192.168.1.1 |     0.0.0.0     |    ens3   |   UG  |
[   15.462981] cloud-init[996]: ci-info: |   1   | 169.254.169.254 | 192.168.1.1 | 255.255.255.255 |    ens3   |  UGH  |
[   15.465796] cloud-init[996]: ci-info: |   2   |   192.168.1.0   |   0.0.0.0   |  255.255.255.0  |    ens3   |   U   |
[   15.468643] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.471468] cloud-init[996]: 2016-12-14 10:56:39,954 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [404]
[   16.589183] cloud-init[996]: 2016-12-14 10:56:41,126 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [404]

my ip netns exec qrouter-baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c iptables -t nat -L is as below

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.1.5          anywhere             to:141.30.74.132
SNAT       all  --  192.168.1.6          anywhere             to:141.30.74.138

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  anywhere             anywhere             to:141.30.74.134
SNAT       all  --  anywhere             anywhere             mark match ! 0x2/0xffff ctstate DNAT to:141.30.74.134

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere             /* Perform source NAT on outgoing traffic. */

in nova-api.log it seems like metadata server is properly started..

2016-12-13 18:02:04.804 6920 INFO nova.wsgi [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] metadata listening on 0.0.0.0:8772
2016-12-13 18:02:04.805 6920 INFO oslo_service.service [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Starting 32 workers
2016-12-13 18:02:05.054 6920 WARNING oslo_config.cfg [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Option "rpc_backend" from group "DEFAULT" is deprecated for
2016-12-13 18:02:05.234 7905 INFO nova.metadata.wsgi.server [req-5990f777-630b-429c-b741-fd0b7ee2220f - - - - -] (7905) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.235 7906 INFO nova.metadata.wsgi.server [req-b081eaae-b009-45f4-90a9-191f36e0fecf - - - - -] (7906) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.267 7907 INFO nova.metadata.wsgi.server [req-e939ec30-1de1-4d04-80c8-99dba46aab3d - - - - -] (7907) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.281 7909 INFO nova.metadata.wsgi.server [req-14bffcd5-42a2-4588-a935-7afb7180ab02 - - - - -] (7909) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.312 7911 INFO nova.metadata.wsgi.server [req-9c783d6f-5910-4fa7-86a7-fb4ed4b52faf - - - - -] (7911) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.323 7915 INFO nova.metadata.wsgi.server [req-db8248a1-87d2-4125-bc9d-73e8d7837dab - - - - -] (7915) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.330 7927 INFO nova.metadata.wsgi.server [req-0c32a119-47e7-471a-9ed2-11acac4b214b - - - - -] (7927) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.335 7925 INFO nova.metadata.wsgi.server [req-fa4d0e5e-d7cf-4b1e-bc64-e4901d173847 - - - - -] (7925) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.338 7931 INFO nova.metadata.wsgi.server [req-e4109117-5b71-45a0-9dbb-edaa165bca59 - - - - -] (7931) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.344 7930 INFO nova.metadata.wsgi.server [req-20e06336-9671-4e6e-8d52-a1e664a74c6f - - - - -] (7930) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.348 7916 INFO nova.metadata.wsgi.server [req-6668dc9a-ccc4-47cc-ac01-76bea7540225 - - - - -] (7916) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.351 7919 INFO nova.metadata.wsgi.server [req-27625ad2-54e9-4add-80da-915550c809a5 - - - - -] (7919) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.352 7923 INFO nova.metadata.wsgi.server [req-6fcb8d8d-e166-4403-b071-50f438d5edb8 - - - - -] (7923) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.362 7934 INFO nova.metadata.wsgi.server [req-d101b3e2-89da-48ff-99ef-db8ec8d25482 - - - - -] (7934) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.377 7908 INFO nova.metadata.wsgi.server [req-65bc543d-14de-460b-8320-00b44dadde09 - - - - -] (7908) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.383 7936 INFO nova.metadata.wsgi.server [req-2c9a3635-79ea-4792-a851-3e01f0769cb2 - - - - -] (7936) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.397 7942 INFO nova.metadata.wsgi.server [req-b8912d2c-b175-4b3f-93b1-804bf10b89cb - - - - -] (7942) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.403 7913 INFO nova.metadata.wsgi.server [req-27d0fcaa-514f-4ac1-902b-012732a0333f - - - - -] (7913) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.416 7945 INFO nova.metadata.wsgi.server [req-2c4de1f7-7750-44c7-ba45-3be0c6502cd3 - - - - -] (7945) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.417 7940 INFO nova.metadata.wsgi.server [req-d56dc28b-cf01-4fc6-9fb0-1ad45c3f7d2b - - - - -] (7940) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.418 7921 INFO nova.metadata.wsgi.server [req-0e43662e-a41d-48c8-9ae0-18abe28f335b - - - - -] (7921) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.435 7952 INFO nova.metadata.wsgi.server [req-7b6e53de-4b45-48b5-ac6c-2f2dc882c783 - - - - -] (7952) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7957 INFO nova.metadata.wsgi.server [req-901e8816-e43c-4446-a081-9b5c7a2cabb5 - - - - -] (7957) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7954 INFO nova.metadata.wsgi.server [req-f1269c16-73ff-42f7-91f5-e99ace884c8f - - - - -] (7954) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.463 7956 INFO nova.metadata.wsgi.server [req-24bd6016-6e81-4e99-80df-e833f3620e01 - - - - -] (7956) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.479 7949 INFO nova.metadata.wsgi.server [req-06b7e743-1602-477d-9ad2-97fa356d973d - - - - -] (7949) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.480 7939 INFO nova.metadata.wsgi.server [req-109b03d7-3493-4d08-b6b2-2769f6977917 - - - - -] (7939) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.499 7963 INFO nova.metadata.wsgi.server [req-a5efa08c-1c54-4428-bfac-35d05cf7191c - - - - -] (7963) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.503 7946 INFO nova.metadata.wsgi.server [req-860da4fe-77c5-41f2-bca3-f0f475cfb82b - - - - -] (7946) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.504 7960 INFO nova.metadata.wsgi.server [req-2cc25bfb-31f6-48e1-9a12-93d6b44246a1 - - - - -] (7960) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.535 7948 INFO nova.metadata.wsgi.server [req-4804e557-1004-4b1a-9701-4941c4555324 - - - - -] (7948) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.546 7961 INFO nova.metadata.wsgi.server [req-aad8654e-4c4c-4a9a-b000-76be2f1ee562 - - - - -] (7961) wsgi starting up on http://0.0.0.0:8772

Please help .. Any help to solve this issue is really appreciated..

metadata server is not reachable 404 error

Hello I created an Openstack newton cluster I have cirros and ubuntu VMs running in the cluster. I can ssh into cirros instance using cirros and cubswin:) . I can ping the 8.8.8.8 from the cirros instance and I can ssh into the cirros instance from any node in my network. But when I run curl http://169.254.169.254 from inside the instance it is giving me

500 Internal Server Error

Remote metadata server experienced an internal server error.

My /var/log/neutron/neutron-metadata-agent.log has the below error also

2016-12-23 11:33:02.111 25058 DEBUG neutron.agent.metadata.agent [-] Request: GET / HTTP/1.0^M
Accept-Encoding: gzip, deflate^M
Content-Length: 0^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: Python-httplib2/0.9.1 (gzip)^M
X-Forwarded-For: 192.168.1.11^M
X-Neutron-Router-Id: baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c __call__ /usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py:86
2016-12-23 11:33:02.117 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 3996b10a61f7415ca8a4721c5ab008b0 exchange 'neutron' topic 'q-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-12-23 11:33:02.199 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 3996b10a61f7415ca8a4721c5ab008b0 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-12-23 11:33:02.201 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: f80218ffb0d2404281a02185d71d8125 exchange 'neutron' topic 'q-plugin' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-12-23 11:33:02.268 25058 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f80218ffb0d2404281a02185d71d8125 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 90, in __call__
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent     return self._proxy_request(instance_id, tenant_id, req)
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 220, in _proxy_request
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent     raise Exception(_('Unexpected response code: %s') % resp.status)
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 300
2016-12-23 11:33:02.282 25058 ERROR neutron.agent.metadata.agent
2016-12-23 11:33:02.285 25058 INFO eventlet.wsgi.server [-] 192.168.1.11,<local> - - [23/Dec/2016 11:33:02] "GET / HTTP/1.1" 500 229 0.173503

when I tried to ssh into ubuntu instance using mykey it is giving me error as below

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 141.30.74.132 [141.30.74.132] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 141.30.74.132:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oMaRnsZ9/l6vjz8hyudV6TjOaXG1n1Ytxu2f8/QsU50
debug1: Host '141.30.74.132' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:7
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I found that in the console.log of ubuntu instance I have a warning like below

[   15.417153] cloud-init[996]: Cloud-init v. 0.7.8 running 'init' at Wed, 14 Dec 2016 10:56:38 +0000. Up 14.36 seconds.
[   15.421593] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   15.424650] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.427664] cloud-init[996]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope |     Hw-Address    |
[   15.430657] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.433658] cloud-init[996]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |   .   |         .         |
[   15.436690] cloud-init[996]: ci-info: |   lo   | True |           ::1/128            |       .       |  host |         .         |
[   15.439663] cloud-init[996]: ci-info: |  ens3  | True |         192.168.1.8          | 255.255.255.0 |   .   | fa:16:3e:25:e8:ac |
[   15.442646] cloud-init[996]: ci-info: |  ens3  | True | fe80::f816:3eff:fe25:e8ac/64 |       .       |  link | fa:16:3e:25:e8:ac |
[   15.445698] cloud-init[996]: ci-info: +--------+------+------------------------------+---------------+-------+-------------------+
[   15.448729] cloud-init[996]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
[   15.451571] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.454385] cloud-init[996]: ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
[   15.457254] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.460141] cloud-init[996]: ci-info: |   0   |     0.0.0.0     | 192.168.1.1 |     0.0.0.0     |    ens3   |   UG  |
[   15.462981] cloud-init[996]: ci-info: |   1   | 169.254.169.254 | 192.168.1.1 | 255.255.255.255 |    ens3   |  UGH  |
[   15.465796] cloud-init[996]: ci-info: |   2   |   192.168.1.0   |   0.0.0.0   |  255.255.255.0  |    ens3   |   U   |
[   15.468643] cloud-init[996]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
[   15.471468] cloud-init[996]: 2016-12-14 10:56:39,954 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [404]
[   16.589183] cloud-init[996]: 2016-12-14 10:56:41,126 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [404]

my ip netns exec qrouter-baa1e7f8-1915-46ec-a46d-f90c4d1bfc1c iptables -t nat -L is as below

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-PREROUTING  all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
neutron-l3-agent-POSTROUTING  all  --  anywhere             anywhere            
neutron-postrouting-bottom  all  --  anywhere             anywhere            

Chain neutron-l3-agent-OUTPUT (1 references)
target     prot opt source               destination         
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-POSTROUTING (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697
DNAT       all  --  anywhere             141.30.74.132        to:192.168.1.5
DNAT       all  --  anywhere             141.30.74.138        to:192.168.1.6

Chain neutron-l3-agent-float-snat (1 references)
target     prot opt source               destination         
SNAT       all  --  192.168.1.5          anywhere             to:141.30.74.132
SNAT       all  --  192.168.1.6          anywhere             to:141.30.74.138

Chain neutron-l3-agent-snat (1 references)
target     prot opt source               destination         
neutron-l3-agent-float-snat  all  --  anywhere             anywhere            
SNAT       all  --  anywhere             anywhere             to:141.30.74.134
SNAT       all  --  anywhere             anywhere             mark match ! 0x2/0xffff ctstate DNAT to:141.30.74.134

Chain neutron-postrouting-bottom (1 references)
target     prot opt source               destination         
neutron-l3-agent-snat  all  --  anywhere             anywhere             /* Perform source NAT on outgoing traffic. */

in nova-api.log it seems like metadata server is properly started..

2016-12-13 18:02:04.804 6920 INFO nova.wsgi [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] metadata listening on 0.0.0.0:8772
2016-12-13 18:02:04.805 6920 INFO oslo_service.service [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Starting 32 workers
2016-12-13 18:02:05.054 6920 WARNING oslo_config.cfg [req-c4785bdf-2b22-4c17-b8a9-c1ebc8cb12a9 - - - - -] Option "rpc_backend" from group "DEFAULT" is deprecated for
2016-12-13 18:02:05.234 7905 INFO nova.metadata.wsgi.server [req-5990f777-630b-429c-b741-fd0b7ee2220f - - - - -] (7905) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.235 7906 INFO nova.metadata.wsgi.server [req-b081eaae-b009-45f4-90a9-191f36e0fecf - - - - -] (7906) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.267 7907 INFO nova.metadata.wsgi.server [req-e939ec30-1de1-4d04-80c8-99dba46aab3d - - - - -] (7907) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.281 7909 INFO nova.metadata.wsgi.server [req-14bffcd5-42a2-4588-a935-7afb7180ab02 - - - - -] (7909) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.312 7911 INFO nova.metadata.wsgi.server [req-9c783d6f-5910-4fa7-86a7-fb4ed4b52faf - - - - -] (7911) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.323 7915 INFO nova.metadata.wsgi.server [req-db8248a1-87d2-4125-bc9d-73e8d7837dab - - - - -] (7915) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.330 7927 INFO nova.metadata.wsgi.server [req-0c32a119-47e7-471a-9ed2-11acac4b214b - - - - -] (7927) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.335 7925 INFO nova.metadata.wsgi.server [req-fa4d0e5e-d7cf-4b1e-bc64-e4901d173847 - - - - -] (7925) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.338 7931 INFO nova.metadata.wsgi.server [req-e4109117-5b71-45a0-9dbb-edaa165bca59 - - - - -] (7931) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.344 7930 INFO nova.metadata.wsgi.server [req-20e06336-9671-4e6e-8d52-a1e664a74c6f - - - - -] (7930) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.348 7916 INFO nova.metadata.wsgi.server [req-6668dc9a-ccc4-47cc-ac01-76bea7540225 - - - - -] (7916) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.351 7919 INFO nova.metadata.wsgi.server [req-27625ad2-54e9-4add-80da-915550c809a5 - - - - -] (7919) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.352 7923 INFO nova.metadata.wsgi.server [req-6fcb8d8d-e166-4403-b071-50f438d5edb8 - - - - -] (7923) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.362 7934 INFO nova.metadata.wsgi.server [req-d101b3e2-89da-48ff-99ef-db8ec8d25482 - - - - -] (7934) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.377 7908 INFO nova.metadata.wsgi.server [req-65bc543d-14de-460b-8320-00b44dadde09 - - - - -] (7908) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.383 7936 INFO nova.metadata.wsgi.server [req-2c9a3635-79ea-4792-a851-3e01f0769cb2 - - - - -] (7936) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.397 7942 INFO nova.metadata.wsgi.server [req-b8912d2c-b175-4b3f-93b1-804bf10b89cb - - - - -] (7942) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.403 7913 INFO nova.metadata.wsgi.server [req-27d0fcaa-514f-4ac1-902b-012732a0333f - - - - -] (7913) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.416 7945 INFO nova.metadata.wsgi.server [req-2c4de1f7-7750-44c7-ba45-3be0c6502cd3 - - - - -] (7945) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.417 7940 INFO nova.metadata.wsgi.server [req-d56dc28b-cf01-4fc6-9fb0-1ad45c3f7d2b - - - - -] (7940) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.418 7921 INFO nova.metadata.wsgi.server [req-0e43662e-a41d-48c8-9ae0-18abe28f335b - - - - -] (7921) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.435 7952 INFO nova.metadata.wsgi.server [req-7b6e53de-4b45-48b5-ac6c-2f2dc882c783 - - - - -] (7952) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7957 INFO nova.metadata.wsgi.server [req-901e8816-e43c-4446-a081-9b5c7a2cabb5 - - - - -] (7957) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.447 7954 INFO nova.metadata.wsgi.server [req-f1269c16-73ff-42f7-91f5-e99ace884c8f - - - - -] (7954) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.463 7956 INFO nova.metadata.wsgi.server [req-24bd6016-6e81-4e99-80df-e833f3620e01 - - - - -] (7956) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.479 7949 INFO nova.metadata.wsgi.server [req-06b7e743-1602-477d-9ad2-97fa356d973d - - - - -] (7949) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.480 7939 INFO nova.metadata.wsgi.server [req-109b03d7-3493-4d08-b6b2-2769f6977917 - - - - -] (7939) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.499 7963 INFO nova.metadata.wsgi.server [req-a5efa08c-1c54-4428-bfac-35d05cf7191c - - - - -] (7963) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.503 7946 INFO nova.metadata.wsgi.server [req-860da4fe-77c5-41f2-bca3-f0f475cfb82b - - - - -] (7946) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.504 7960 INFO nova.metadata.wsgi.server [req-2cc25bfb-31f6-48e1-9a12-93d6b44246a1 - - - - -] (7960) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.535 7948 INFO nova.metadata.wsgi.server [req-4804e557-1004-4b1a-9701-4941c4555324 - - - - -] (7948) wsgi starting up on http://0.0.0.0:8772
2016-12-13 18:02:05.546 7961 INFO nova.metadata.wsgi.server [req-aad8654e-4c4c-4a9a-b000-76be2f1ee562 - - - - -] (7961) wsgi starting up on http://0.0.0.0:8772

Please help .. Any help to solve this issue is really appreciated..