Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

Neutron server and RabbitMQ are configured and running.

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

Neutron server and RabbitMQ are configured and running.

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

Pings from network to controller, internet and compute node through 10.0.0.21 and 10.0.3.21 work. Controller can ping network at 10.0.0.21 and 10.0.1.21 but not 10.0.3.21 (br-ex; network's internet connection).

tcpdump of ping to 10.0.3.21 (the br-ex IP of network node) from controller node
ARP, Request who-has 10.0.3.21 tell 10.0.3.11, length 28
"Destination Host Unreachable. 100% packet loss."

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

Neutron server and RabbitMQ are configured and running.

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

Pings from network to controller, internet and compute node through 10.0.0.21 and 10.0.3.21 work. Controller can ping network at 10.0.0.21 and 10.0.1.21 but not 10.0.3.21 (br-ex; network's internet connection).

tcpdump of ping to 10.0.3.21 (the br-ex IP of network node) from controller node
ARP, Request who-has 10.0.3.21 tell 10.0.3.11, length 28
"Destination Host Unreachable. 100% packet loss."

No bridges show up in brctl .

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

Neutron server and RabbitMQ are configured and running.

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

Pings from network to controller, internet and compute node through 10.0.0.21 and 10.0.3.21 work. Controller can ping network at 10.0.0.21 and 10.0.1.21 but not 10.0.3.21 (br-ex; network's internet connection).

tcpdump of ping to 10.0.3.21 (the br-ex IP of network node) from controller node
ARP, Request who-has 10.0.3.21 tell 10.0.3.11, length 28
"Destination Host Unreachable. 100% packet loss."

After entering neutron agent-list, tcpdump on network's eth1 reports:

ARP, Request who-has 10.0.3.11 tell 10.0.3.21, length 28

Pinging 10.0.3.11 from network node results in Destination Host Unreachable.

No bridges show up in brctl .

Is this a problem with the NAT setup on 10.0.3.xx or bridging (br-ex is on eth1 on 10.0.3.21)?

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

Update: I installed nova-compute on the compute1 node. Nova-compute is running (nova-compute start/running) on compute1 node, but nova-manage service list only shows cert, consoleauth, scheduler and conductor. Therefore it seems there is limited connectivity between the controller, network and compute1 nodes. Pings to 10.0.0.xx and 10.0.1.xx work on all nodes, but not pings to 10.0.3.xx. (NAT, bridging)

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

Neutron server and RabbitMQ are configured and running.running. (I'm not sure how to check whether RabbitMQ is connecting)

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

Pings from network to controller, internet and compute node through 10.0.0.21 and 10.0.3.21 work. Controller can ping network at 10.0.0.21 and 10.0.1.21 but not 10.0.3.21 (br-ex; network's internet connection).

tcpdump of ping to 10.0.3.21 (the br-ex IP of network node) from controller node
ARP, Request who-has 10.0.3.21 tell 10.0.3.11, length 28
"Destination Host Unreachable. 100% packet loss."

After entering neutron agent-list, tcpdump on network's eth1 reports:

ARP, Request who-has 10.0.3.11 tell 10.0.3.21, length 28

Pinging 10.0.3.11 from network node results in Destination Host Unreachable.

No bridges show up in brctl .

Is this a problem with the NAT setup on 10.0.3.xx or bridging (br-ex is on eth1 on 10.0.3.21)?

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

"connection to neutron failed" on network node (VirtualBox VM)

neutron agent-list works on the controller node but fails on the network node.

Update: I installed nova-compute Solution:

Despite following the guide on the compute1 https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst there was no connection between the Network and Controller nodes while using "neutron agent-list" on the network node. Nova-compute is running (nova-compute start/running) on compute1 node, Error was simply "connection to neutron failed".

The nodes were run on VirtualBox. The internal network 10.0.0.xx and tunneling 10.0.1.xx were host-only networks and the external network 10.0.3.xx was on NAT for internet access.

The solution to this was to change NAT to Bridged Network in VirtualBox.

Therefore:

Management Network (10.0.0.0/24) is a host-only network.

VM Traffic Network (10.0.1.0/24) is a host-only network.

Public Network (192.168.100.0/24) is a bridged network. (IP begins with 192.168 but nova-manage service list only shows cert, consoleauth, scheduler and conductor. Therefore it seems there is limited connectivity between the controller, network and compute1 nodes. Pings to 10.0.0.xx and 10.0.1.xx work on all nodes, but not pings to 10.0.3.xx. (NAT, bridging)may end differently depending on your setup. The automatically generated IPs and default gateway were manually entered into /etc/network/interfaces/.)

Solved problem:

I am on the Network Node=>Check Neutron agents step of this guide: https://github.com/ChaimaGhribi/OpenStack-Icehouse-Installation/blob/3d9581f92919fedbd984105cb24be623a40c56ec/OpenStack-Icehouse-Installation.rst

ConnectionFailed: Connection to neutron failed:
DEBUG: neutronclient.shell clean_up ListAgent
DEBUG: neutronclient.shell Got an error: Connection to neutron failed:

Nodes are Ubuntu 14.04 VMs on VirtualBox. Eth0 is a host-only network on 10.0.0.xx. Eth1 is a NAT network for internet access with fixed IPs and port forwarding, 10.0.3.xx. Eth2 is tunneling for the network node on 10.0.1.xx.

10.0.1.xx. Neutron server and RabbitMQ are configured and running. (I'm not sure how to check whether RabbitMQ is connecting)

running.

Network node can ping the controller at 10.0.0.11 (host-only) but not at 10.0.3.11 (NAT). 10.0.3.11:9696 is the publicurl of neutron. The same happens with telnet. 10.0.1.21 can be pinged by controller.

Pings from network to controller, internet and compute node through 10.0.0.21 and 10.0.3.21 work. Controller can ping network at 10.0.0.21 and 10.0.1.21 but not 10.0.3.21 (br-ex; network's internet connection).

tcpdump of ping to 10.0.3.21 (the br-ex IP of network node) from controller node
ARP, Request who-has 10.0.3.21 tell 10.0.3.11, length 28
"Destination Host Unreachable. 100% packet loss."

After entering neutron agent-list, tcpdump on network's eth1 reports:

 ARP, Request who-has 10.0.3.11 tell 10.0.3.21, length 28

28

Pinging 10.0.3.11 from network node results in Destination Host Unreachable.

No bridges show up in brctl .

Is this a problem with the NAT setup on 10.0.3.xx or bridging (br-ex is on eth1 on 10.0.3.21)?

/etc/network/interfaces/

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.21
        netmask 255.255.255.0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto eth2
iface eth2 inet static
        address 10.0.1.21
        netmask 255.255.255.0

auto br-ex
iface br-ex inet static
        address 10.0.3.21
        netmask 255.255.255.0
        gateway 10.0.3.2
        dns-nameservers 8.8.8.8

ifconfig on network node

br-ex     Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet addr:10.0.3.21  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::dc0c:aaff:fec7:2057/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0                                                                                          
          TX packets:569 errors:0 dropped:0 overruns:0 carrier:0                                                                                        
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:745976 (745.9 KB)  TX bytes:42084 (42.0 KB)                                                                                          

br-int    Link encap:Ethernet  HWaddr 1e:12:f3:bc:01:47                                                                                                 
          inet6 addr: fe80::2897:4dff:fe6e:19b1/64 Scope:Link                                                                                           
          UP BROADCAST RUNNING  MTU:1500  Metric:1                                                                                                      
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                                                                                            
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0                                                                                          
          collisions:0 txqueuelen:0                                                                                                                     
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)                                                                                                    

br-tun    Link encap:Ethernet  HWaddr f6:41:f7:e6:7d:40                                                                                                 
          inet6 addr: fe80::78c2:f7ff:fe0a:da94/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:37:58:4e  
          inet addr:10.0.0.21  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe37:584e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:351 errors:0 dropped:0 overruns:0 frame:0
          TX packets:345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41976 (41.9 KB)  TX bytes:45210 (45.2 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:0f:fb:eb  
          inet6 addr: fe80::a00:27ff:fe0f:fbeb/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:575 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:745976 (745.9 KB)  TX bytes:42600 (42.6 KB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:b5:37:62  
          inet addr:10.0.1.21  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb5:3762/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12682 (12.6 KB)  TX bytes:12682 (12.6 KB)

ovs-vsctl show

34ffa9a7-8825-4e63-bfcd-c231c34862b3
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.2"

ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=343.021s, table=0, n_packets=6, n_bytes=480, idle_age=337, priority=1 actions=NORMAL
 cookie=0x0, duration=342.939s, table=22, n_packets=0, n_bytes=0, idle_age=342, priority=0 actions=drop

ovs-ofctl dump-flows br-ex

NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=370.773s, table=0, n_packets=1415, n_bytes=791276, idle_age=17, priority=0 actions=NORMAL

ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x2): dpid:00001e12f3bc0147
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-tun): addr:56:01:e1:93:d3:8e
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:1e:12:f3:bc:01:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

ovs-ofctl show br-ex

OFPT_FEATURES_REPLY (xid=0x2): dpid:00000800270ffbeb
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:08:00:27:0f:fb:eb
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

[Included logs were removed due to being unnecessary for solution]