Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

During Kolla deploy, when neutron comes up, networking goes down

Greetings,

I've been trying to complete a Kolla deployment for about a week now, and I'm just plain stuck. I've gotten the boxes set up, and can get through the prechecks, but when I execute deploy, I only get as far as "Starting neutron-server container". At that point, all of the nodes lose network connectivity. Logging into the nodes from the console, I can see that everything appears normal on the box. The interfaces are up, and you can ping from each interface to the other, but no traffic can get into or out of the box. Stopping the docker containers does not fix the problem. Rebooting the box will bring the networking back, but only until you restart docker and the containers, at which point the network stop working again.

I'm assuming I've made some obvious noobie mistake in the configuration, which I hope someone here can spot for me.

All boxes have two NICs. The primary interface is on 192.168.60.0/24, and has internet connectivity. The secondary interface is on 192.168.65.0/24, and only has connectivity between the nodes.

The interface names are different on the control/network nodes, and the compute/storage nodes, and are specified in the multinode inventory file.

The interesting part of my multinode inventory file looks like:

[control]
smss02 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss04 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[network]
smss03 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss05 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[compute]
smst[01:16] neutron_external_interface=ens6 api_interface=ens5 storage_interface=ens5 tunnel_interface=ens5 network_interface=ens5 kolla_external_vip_interface=ens6

The only non-comment lines in my globals.yml are:

kolla_internal_vip_address: 192.168.65.254
kolla_external_vip_address: 192.168.60.254
docker_registry: "192.168.60.50:4000"
keepalived_virtual_router_id: "10"
openstack_logging_debug: "True"
enable_cinder: "yes"

Is there anything blatantly wrong with this setup? I can provide further info if it will help.

Any comments or suggestions will be greatly appreciated.

Many thanks, Brian

During Kolla deploy, when neutron comes up, networking goes down

Greetings,

I've been trying to complete a Kolla deployment for about a week now, and I'm just plain stuck. I've gotten the boxes set up, and can get through the prechecks, but when I execute deploy, I only get as far as "Starting neutron-server container". At that point, all of the nodes lose network connectivity. Logging into the nodes from the console, I can see that everything appears normal on the box. The interfaces are up, and you can ping from each interface to the other, but no traffic can get into or out of the box. Stopping the docker containers does not fix the problem. Rebooting the box will bring the networking back, but only until you restart docker and the containers, at which point the network stop working again.

I'm assuming I've made some obvious noobie mistake in the configuration, which I hope someone here can spot for me.

All boxes have two NICs. The primary interface is on 192.168.60.0/24, and has internet connectivity. The secondary interface is on 192.168.65.0/24, and only has connectivity between the nodes.

The interface names are different on the control/network nodes, and the compute/storage nodes, and are specified in the multinode inventory file.

The interesting part of my multinode inventory file looks like:

[control]
smss02 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss04 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[network]
smss03 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss05 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[compute]
smst[01:16] neutron_external_interface=ens6 api_interface=ens5 storage_interface=ens5 tunnel_interface=ens5 network_interface=ens5 kolla_external_vip_interface=ens6

The only non-comment lines in my globals.yml are:

kolla_internal_vip_address: 192.168.65.254
kolla_external_vip_address: 192.168.60.254
docker_registry: "192.168.60.50:4000"
keepalived_virtual_router_id: "10"
openstack_logging_debug: "True"
enable_cinder: "yes"

Is there anything blatantly wrong with this setup? I can provide further info if it will help.

Any comments or suggestions will be greatly appreciated.

Many thanks, Brian

Brian.

EDIT 1: A few more system details:

Servers are running Centos7:
[root@smss01 ~]# uname -a
Linux smss01 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@smss01 ~]# ansible --version
ansible 1.9.6
  configured module search path = None

[root@smss01 ~]# docker --version
Docker version 1.11.2, build b9f10c9

Kolla up-to-date as of Wed Jun 8 16:57:49 2016 +0000

During Kolla deploy, when neutron comes up, networking goes down

Greetings,

I've been trying to complete a Kolla deployment for about a week now, and I'm just plain stuck. I've gotten the boxes set up, and can get through the prechecks, but when I execute deploy, I only get as far as "Starting neutron-server container". At that point, all of the nodes lose network connectivity. Logging into the nodes from the console, I can see that everything appears normal on the box. The interfaces are up, and you can ping from each interface to the other, but no traffic can get into or out of the box. Stopping the docker containers does not fix the problem. Rebooting the box will bring the networking back, but only until you restart docker and the containers, at which point the network stop working again.

I'm assuming I've made some obvious noobie mistake in the configuration, which I hope someone here can spot for me.

All boxes have two NICs. The primary interface is on 192.168.60.0/24, and has internet connectivity. The secondary interface is on 192.168.65.0/24, and only has connectivity between the nodes.

The interface names are different on the control/network nodes, and the compute/storage nodes, and are specified in the multinode inventory file.

The interesting part of my multinode inventory file looks like:

[control]
smss02 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss04 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[network]
smss03 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss05 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[compute]
smst[01:16] neutron_external_interface=ens6 api_interface=ens5 storage_interface=ens5 tunnel_interface=ens5 network_interface=ens5 kolla_external_vip_interface=ens6

The only non-comment lines in my globals.yml are:

kolla_internal_vip_address: 192.168.65.254
kolla_external_vip_address: 192.168.60.254
docker_registry: "192.168.60.50:4000"
keepalived_virtual_router_id: "10"
openstack_logging_debug: "True"
enable_cinder: "yes"

Is there anything blatantly wrong with this setup? I can provide further info if it will help.

Any comments or suggestions will be greatly appreciated.

Many thanks, Brian.

EDIT 1: A few more system details:

Servers are running Centos7:
[root@smss01 ~]# uname -a
Linux smss01 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@smss01 ~]# ansible --version
ansible 1.9.6
  configured module search path = None

[root@smss01 ~]# docker --version
Docker version 1.11.2, build b9f10c9

Kolla up-to-date as of Wed Jun 8 16:57:49 2016 +0000

EDIT 2:

Here are the interfaces on one of the compute/storage nodes. The control/network nodes are similar, but with different interface names.

[root@smst01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens15f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:04 brd ff:ff:ff:ff:ff:ff
3: ens15f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:05 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c0 brd ff:ff:ff:ff:ff:ff
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c1 brd ff:ff:ff:ff:ff:ff
6: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c2 brd ff:ff:ff:ff:ff:ff
7: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c3 brd ff:ff:ff:ff:ff:ff
8: ens4f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:02 brd ff:ff:ff:ff:ff:ff
**9: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e4:1d:2d:2c:f0:80 brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.53/24 brd 192.168.60.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:fe2c:f080/64 scope link
       valid_lft forever preferred_lft forever**
10: ens6d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e4:1d:2d:2c:f0:81 brd ff:ff:ff:ff:ff:ff
**11: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e4:1d:2d:b1:09:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.53/24 brd 192.168.65.255 scope global ens5
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:feb1:990/64 scope link
       valid_lft forever preferred_lft forever**
12: ens5d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e4:1d:2d:b1:09:91 brd ff:ff:ff:ff:ff:ff
13: ens4f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:03 brd ff:ff:ff:ff:ff:ff
14: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:27:84:79:54 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

During Kolla deploy, when neutron comes up, networking goes down

Greetings,

I've been trying to complete a Kolla deployment for about a week now, and I'm just plain stuck. I've gotten the boxes set up, and can get through the prechecks, but when I execute deploy, I only get as far as "Starting neutron-server container". At that point, all of the nodes lose network connectivity. Logging into the nodes from the console, I can see that everything appears normal on the box. The interfaces are up, and you can ping from each interface to the other, but no traffic can get into or out of the box. Stopping the docker containers does not fix the problem. Rebooting the box will bring the networking back, but only until you restart docker and the containers, at which point the network stop stops working again.

I'm assuming I've made some obvious noobie mistake in the configuration, which I hope someone here can spot for me.

All boxes have two NICs. The primary interface is on 192.168.60.0/24, and has internet connectivity. The secondary interface is on 192.168.65.0/24, and only has connectivity between the nodes.

The interface names are different on the control/network nodes, and the compute/storage nodes, and are specified in the multinode inventory file.

The interesting part of my multinode inventory file looks like:

[control]
smss02 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss04 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[network]
smss03 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11
smss05 neutron_external_interface=ens11 api_interface=ens11d1 storage_interface=ens11d1 tunnel_interface=ens11d1 network_interface=ens11d1 kolla_external_vip_interface=ens11

[compute]
smst[01:16] neutron_external_interface=ens6 api_interface=ens5 storage_interface=ens5 tunnel_interface=ens5 network_interface=ens5 kolla_external_vip_interface=ens6

The only non-comment lines in my globals.yml are:

kolla_internal_vip_address: 192.168.65.254
kolla_external_vip_address: 192.168.60.254
docker_registry: "192.168.60.50:4000"
keepalived_virtual_router_id: "10"
openstack_logging_debug: "True"
enable_cinder: "yes"

Is there anything blatantly wrong with this setup? I can provide further info if it will help.

Any comments or suggestions will be greatly appreciated.

Many thanks, Brian.

EDIT 1: A few more system details:

Servers are running Centos7:
[root@smss01 ~]# uname -a
Linux smss01 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@smss01 ~]# ansible --version
ansible 1.9.6
  configured module search path = None

[root@smss01 ~]# docker --version
Docker version 1.11.2, build b9f10c9

Kolla up-to-date as of Wed Jun 8 16:57:49 2016 +0000

EDIT 2:

Here are the interfaces on one of the compute/storage nodes. The control/network nodes are similar, but with different interface names.

[root@smst01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens15f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:04 brd ff:ff:ff:ff:ff:ff
3: ens15f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:05 brd ff:ff:ff:ff:ff:ff
4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c0 brd ff:ff:ff:ff:ff:ff
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c1 brd ff:ff:ff:ff:ff:ff
6: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c2 brd ff:ff:ff:ff:ff:ff
7: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:9d:c3 brd ff:ff:ff:ff:ff:ff
8: ens4f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:02 brd ff:ff:ff:ff:ff:ff
**9: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e4:1d:2d:2c:f0:80 brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.53/24 brd 192.168.60.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:fe2c:f080/64 scope link
       valid_lft forever preferred_lft forever**
10: ens6d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e4:1d:2d:2c:f0:81 brd ff:ff:ff:ff:ff:ff
**11: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e4:1d:2d:b1:09:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.65.53/24 brd 192.168.65.255 scope global ens5
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:feb1:990/64 scope link
       valid_lft forever preferred_lft forever**
12: ens5d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e4:1d:2d:b1:09:91 brd ff:ff:ff:ff:ff:ff
13: ens4f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0c:c4:7a:bd:61:03 brd ff:ff:ff:ff:ff:ff
14: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:27:84:79:54 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever