Ask Your Question
0

Packets not leaving compute host

asked 2014-04-11 23:01:50 -0600

arunuke gravatar image

updated 2014-04-14 13:12:30 -0600

larsks gravatar image

I am having an unusual issue with networking behavior in my environment (Fedora 20/Icehouse/RDO). My VMs are booting up, they are being assigned IPs by DHCP (as shown by nova list), but none of the VMs are getting IP addresses assigned. The issue appears to be that no packets are actually leaving the compute host.

Here are some of the steps I took to check.

  1. Login to vm console and manually issue udhcpc (cirros image). I see bootp packets in the tcpdump trace of phy-br-eth1, but I do not see them in the tcpdump of phy-br-eth1 on the network node.

    On local compute node:

    [root@aragorn arunt]# tcpdump -i phy-br-eth1
tcpdump: WARNING: phy-br-eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on phy-br-eth1, link-type EN10MB (Ethernet), capture size 65535 bytes


23:57:25.288979 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:26:fa:5d (oui Unknown), length 280
23:57:28.301814 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:26:fa:5d (oui Unknown), length 280
23:57:31.308918 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:26:fa:5d (oui Unknown), length 280

  2. Manually configure the IP address of the VM on the link. Ping another manually configured VM on the same compute host. Ping works.

  3. Ping another manually configured VM on another compute node, ping does not work. The ICMP packet is seen on phy-br-eth1 of the compute host in question, but not on phy-br-eth1 of the other compute node.

  4. security groups and rules seem to be in place.

    [root@grey utils]# nova secgroup-list-rules default
+-------------+-----------+---------+------------+--------------+
| IP Protocol | From Port | To Port | IP Range   | Source Group |
+-------------+-----------+---------+------------+--------------+
|             |           |         |            | default      |
| icmp        | -1        | -1      | 0.0.0.0/24 |              |
|             |           |         |            | default      |
| tcp         | 22        | 22      | 0.0.0.0/24 |              |
+-------------+-----------+---------+------------+--------------+

I am using openvswitch and not ml2.

Any inputs are greatly appreciated.

edit retag flag offensive close merge delete

Comments

can you check iptables rules that is getting applied to tap interfaces. using: iptables -L and see whether there are any drop rule. Also try disabling neutron security group and create a new vm for testing.

SGPJ gravatar imageSGPJ ( 2014-04-13 07:46:42 -0600 )edit

Thanks Shankar Ganesh. My iptables rule seem to be fine with no drops listed. I went ahead and disabled iptables and neutron security groups while switching all my services to use the Noop driver. Still no luck.

arunuke gravatar imagearunuke ( 2014-04-15 14:03:17 -0600 )edit
add a comment

3 answers

Sort by ยป oldest newest most voted
0

answered 2014-04-15 17:08:18 -0600

arunuke gravatar image

Issue resolved. The physical interface (eth1) on top of which the bridge (br-eth1) was built was DOWN on the compute and networking hosts. 

[test@arwen ~]$ sudo ovs-ofctl show br-eth1
OFPT_FEATURES_REPLY (xid=0x2): dpid:00006ea765782249
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:52:54:00:d9:b1:7e
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 2(phy-br-eth1): addr:d2:53:6e:7f:f9:2b
     config:     0
     state:      0
     current:    10GB-FD COPPER
     speed: 10000 Mbps now, 0 Mbps max
 LOCAL(br-eth1): addr:6e:a7:65:78:22:49
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

After I modified them, things are looking good.

[test@thorin ~]$ sudo ip link set dev eth1 up
[test@thorin ~]$ sudo ovs-ofctl show br-eth1
OFPT_FEATURES_REPLY (xid=0x2): dpid:00008a33604d754d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(eth1): addr:52:54:00:eb:72:6b
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(phy-br-eth1): addr:e2:d9:2b:a1:0d:61
     config:     0
     state:      0
     current:    10GB-FD COPPER
     speed: 10000 Mbps now, 0 Mbps max
 LOCAL(br-eth1): addr:8a:33:60:4d:75:4d
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[test@thorin ~]$ sudo tcpdump -i br-eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:03:40.488784 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:f0:5a:31 (oui Unknown), length 280
15:03:40.509560 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:f0:5a:31 (oui Unknown), length 292

2 packets captured
2 packets received by filter
0 packets dropped by kernel

Thanks to all the folks who assisted.

edit flag offensive delete link more
add a comment
0

answered 2014-04-15 04:04:40 -0600

gklyne gravatar image

I'm not sure if this will help, but I ran into a similar problem with nova networking on Havana, installing on Ubuntu 12.04.

My problem was trying to use the same physical interface for "flat_interface" and "public_interface". That seems to stop the required NAT from happening. When I changed this to use separate interfaces, things started working for me.

My resulting nova.conf network configuration contains the following:

[DEFAULT]
 :
# network specific settings
network_manager=nova.network.manager.FlatDHCPManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
network_size=254
allow_same_net_traffic=False
multi_host=True
send_arp_for_ha=True
share_dhcp_address=True
force_dhcp_release=True
flat_network_bridge=br100
flat_interface=eth1
public_interface=eth0
edit flag offensive delete link more

Comments

Thanks gklyne. My environment does not use the flat networks, but vlans.

arunuke gravatar imagearunuke ( 2014-04-15 14:05:11 -0600 )edit
add a comment
0

answered 2014-04-16 04:54:16 -0600

Pratik gravatar image

Check ur iptables...close ur iptacble.. check ur logfile if it is givng amqp error...check ur rabitmq services running on both the nodes... u can change rabitmq guest password

edit flag offensive delete link more
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2014-04-11 23:01:50 -0600

Seen: 1,137 times

Last updated: Apr 16 '14

Related questions

Rocks Cluster on Openstack

multinode in centos7 - network card configuration

separate DNS domains via dhcp in multi-tenant Juno

sriov binding failure

how does ml2 work specifically?

Instances can't reach external network and Internet

see number of free floating ips?

Instances not getting IP through DHCP though IP is being displayed on dashboard [closed]

No ip from DHCP

What are endpoints?