Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

network is unreachable in instance

I have similar problem as discussed in : https://ask.openstack.org/en/question/80462/network-is-unreachable-in-instance/

I didn't get the solution, so asking for some help: We have 3 controller nodes connected to 12 compute nodes. I was working fine, we were facing some issues connecting to floating ips. After some time, complete network went down. Not sure what is the problem. If you need more logs , please let me know. I will provide:

[Thu Jun 14 14:38:57] root@:~# node-ping-check

 Ping check of all service-ip, vm-ip, data-ip and floating-ip, ping
 status result after ip, 1/0 = alive/dead

 command start time 2018-06-14 14:39:00

 node    service-ip        vm-ip          data-ip        floating-ip
---------------------------------------------------------------------
 h002   10.1.12.102 1   10.2.12.102 1   10.3.13.102 1   10.4.13.102 1 (controller 1)
 h003   10.1.12.103 1   10.2.12.103 1   10.3.13.103 1   10.4.13.103 1 (controller 2)
 h004   10.1.12.104 1   10.2.12.104 1   10.3.13.104 1   10.4.13.104 1 (controller 3 + Network node)
 h005   10.1.12.105 1   10.2.12.105 1   10.3.13.105 1   10.4.13.105 0
 h006   10.1.12.106 1   10.2.12.106 1   10.3.13.106 1   10.4.13.106 0
 h007   10.1.12.107 1   10.2.12.107 1   10.3.13.107 1   10.4.13.107 0
 h008   10.1.12.108 1   10.2.12.108 1   10.3.13.108 1   10.4.13.108 0
 h009   10.1.12.109 1   10.2.12.109 1   10.3.13.109 1   10.4.13.109 0
 h010   10.1.12.110 1   10.2.12.110 1   10.3.13.110 1   10.4.13.110 0
 h011   10.1.14.111 1   10.2.14.111 1   10.3.15.111 1   10.4.15.111 0
 h012   10.1.14.112 1   10.2.14.112 1   10.3.15.112 1   10.4.15.112 0
 h013   10.1.14.113 1   10.2.14.113 1   10.3.15.113 1   10.4.15.113 0
 h014   10.1.14.114 1   10.2.14.114 1   10.3.15.114 1   10.4.15.114 0
 h015   10.1.14.115 1   10.2.14.115 1   10.3.15.115 1   10.4.15.115 0
 h016   10.1.14.116 0   10.2.14.116 0   10.3.15.116 0   10.4.15.116 0  ( Down because of hardware issue)
 h017   10.1.14.117 1   10.2.14.117 1   10.3.15.117 1   10.4.15.117 0

[Thu Jun 14 14:39:39] root@:~#

I tired to bring br-ex to fix the problem by associating with eth3 ip . If i don't do this ping to 10.4.13.104 was failing.

~$ifconfig eth3 down
~$ifconfig eth3 0.0.0.0 up
~$ovs-vsctl del-br br-ex
~$ovs-vsctl add-br br-ex
~$ovs-vsctl add-port br-ex eth3
~$ifconfig br-ex 10.4.13.104 netmask 255.255.255.0
~$route add -net 10.4.0.0 netmask 255.255.0.0 gateway 10.4.13.1 dev br-ex

I have several VM's running on the compute nodes.

neutron agent-list   :-)             # ouput is clean with all node  
                                       (except node 16 - which is down)

nova service-list     enabled        # (except node 16 - which is down)

All service are running:

h004:~$ service --status-all | grep nova
     [ + ]  nova-api
     [ + ]  nova-cert
     [ + ]  nova-conductor
     [ + ]  nova-consoleauth
     [ + ]  nova-novncproxy
     [ + ]  nova-scheduler
h004:~$ service --status-all | grep neutron
     [ + ]  neutron-dhcp-agent
     [ + ]  neutron-l3-agent
     [ + ]  neutron-metadata-agent
     [ + ]  neutron-openvswitch-agent
     [ + ]  neutron-ovs-cleanup
     [ + ]  neutron-server

On network node, I have following logs:

On: /var/log/neutron/neutron-openvswitch-agent.log (several errors)

2018-06-15 14:37:28.558 4534 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-67d6d195-9909-4411-9b57-5b6f8a5a8e71 - - - - -] Skipping ARP spoofing rules for port 'sg-34cae0fa-d9' because it has port security disabled
2018-06-15 14:37:29.836 4534 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-67d6d195-9909-4411-9b57-5b6f8a5a8e71 - - - - -] Skipping ARP spoofing rules for port 'qg-0e0ed974-b2' because it has port security disabled
2018-06-15 14:37:33.165 4534 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-67d6d195-9909-4411-9b57-5b6f8a5a8e71 - - - - -] Configuration for devices up [u'3ab3b173-bc30-4e50-a76a-6ed4f14cfd82', u'c8b7dc22-ed0f-4400-a6da-45481c0847f1', u'efa1fed1-7c22-460f-88cf-4d9134c1c646', u'2c068b75-b5
ba-41e7-9429-8c6e21d2b4e9', u'ffd0c852-a767-49be-83ad-9dd39734a070', u'5b3579c3-951c-419b-a41b-69d6e99908a2', u'c83d7c12-74dc-4cbf-8f94-8bd901199c2f', u'4b366aff-7844-49c9-a7a6-3ae9496883d9', u'34cae0fa-d901-4c0d-acb8-b2fe07221dcd', u'0e0ed974-b239-4f1b-8fcf-5a732105c616'] and devices down [] completed.

On: /var/log/neutron/neutron-l3-agent.log

2018-06-15 14:32:18.566 4549 INFO neutron.agent.linux.interface [-] Device sg-2c068b75-b5 already exists
2018-06-15 14:32:34.122 4549 INFO neutron.agent.linux.interface [-] Device sg-34cae0fa-d9 already exists
2018-06-15 14:32:35.415 4549 INFO neutron.agent.linux.interface [-] Device sg-4b366aff-78 already exists