Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

VM's can't get DHCP

Hi!

I'm running an openstack multinode configuration (3 compute nodes), IceHouse, CentOS 6.5. I use flat networking with VLAN tagging.

When I boot up and instance, it does not get any DHCP from the network node.

On the compute node: tail /var/log/neutron/openvswitch-agent.log

2014-09-05 09:11:15.843 2804 INFO neutron.agent.securitygroups_rpc [req-0617459d-a623-4392-97a4-6336d44ec70a None] Security group member updated [u'ad057153-a11f-4f9b-b7fd-7f1131bb4cc0']
2014-09-05 09:11:21.703 2804 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'f686a831-afa2-4d2b-9059-7104c6a1cf35'])
2014-09-05 09:11:22.225 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port f686a831-afa2-4d2b-9059-7104c6a1cf35 updated. Details: {u'admin_state_up': True, u'network_id': u'bc31c8d0-f892-431b-a3f7-5c2b3a8b2504', u'segmentation_id': 1401, u'physical_network': u'physnet1', u'device': u'f686a831-afa2-4d2b-9059-7104c6a1cf35', u'port_id': u'f686a831-afa2-4d2b-9059-7104c6a1cf35', u'network_type': u'vlan'}
2014-09-05 09:11:22.226 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Assigning 2 as local vlan for net-id=bc31c8d0-f892-431b-a3f7-5c2b3a8b2504
2014-09-05 09:11:22.620 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Configuration for device f686a831-afa2-4d2b-9059-7104c6a1cf35 completed.

On the network node; tail /var/log/neutron/openvswtich-agent.log

2014-09-05 08:55:01.753 5423 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-ofctl', 'dump-flows', 'br-int', 'table=22']
Exit code: 0
Stdout: 'NXST_FLOW reply (xid=0x4):\n cookie=0x0, duration=2793.91s, table=22, n_packets=0, n_bytes=0, idle_age=2793, priority=0 actions=drop\n'
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74
2014-09-05 08:55:01.754 5423 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent rpc_loop - iteration:1395 completed. Processed ports statistics: {'ancillary': {'removed': 0, 'added': 0}, 'regular': {'updated': 0, 'added': 0, 'removed': 0}}. Elapsed:0.061 rpc_loop /usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1386

This log keeps growing with this message popping up every 1 second. Is this packets being discarded by the network node?

When I restart the openvswith service on the network node, the log shows;

2014-09-05T07:21:13Z|00026|fatal_signal|WARN|terminating with signal 15 (Terminated)
2014-09-05T07:21:13Z|00003|worker(worker)|INFO|worker process exiting
2014-09-05T07:21:13Z|00002|daemon(monitor)|INFO|pid 5095 died, killed (Terminated), exiting
2014-09-05T07:21:14Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2014-09-05T07:21:14Z|00002|worker(worker)|INFO|worker process started
2014-09-05T07:21:14Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2014-09-05T07:21:14Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2014-09-05T07:21:14Z|00004|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-2ddc5852-e5 device failed: No such device
2014-09-05T07:21:14Z|00005|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-99334d8e-e4 device failed: No such device
2014-09-05T07:21:14Z|00006|netdev|WARN|failed to set flags for network device qg-2ddc5852-e5: No such device
2014-09-05T07:21:14Z|00007|netdev|WARN|failed to set flags for network device qr-99334d8e-e4: No such device
2014-09-05T07:21:14Z|00008|bridge|INFO|bridge br-ex: using datapath ID 00000026554dbcec
2014-09-05T07:21:14Z|00009|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qg-2ddc5852-e5 device failed: No such device
2014-09-05T07:21:14Z|00010|netdev_linux|WARN|qg-2ddc5852-e5: removing policing failed: No such device
2014-09-05T07:21:14Z|00011|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt"
2014-09-05T07:21:14Z|00012|bridge|INFO|bridge br-eth1: using datapath ID 00000026554dbcea
2014-09-05T07:21:14Z|00013|connmgr|INFO|br-eth1: added service controller "punix:/var/run/openvswitch/br-eth1.mgmt"
2014-09-05T07:21:14Z|00014|bridge|INFO|bridge br-int: using datapath ID 000016215ea0d144
2014-09-05T07:21:14Z|00015|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-99334d8e-e4 device failed: No such device
2014-09-05T07:21:14Z|00016|netdev_linux|WARN|qr-99334d8e-e4: removing policing failed: No such device
2014-09-05T07:21:14Z|00017|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"
2014-09-05T07:21:14Z|00018|bridge|INFO|ovs-vswitchd (Open vSwitch) 1.11.0
2014-09-05T07:21:15Z|00019|bridge|INFO|bridge br-int: added interface int-br-eth1 on port 1
2014-09-05T07:21:15Z|00020|bridge|INFO|bridge br-eth1: added interface phy-br-eth1 on port 2
2014-09-05T07:21:24Z|00021|memory|INFO|8460 kB peak resident set size after 10.0 seconds
2014-09-05T07:21:24Z|00022|memory|INFO|facets:29 ports:9 rules:16 subfacets:35
2014-09-05T07:21:24Z|00023|ofproto|INFO|br-int: 5 flow_mods in the 1 s starting 10 s ago (4 adds, 1 deletes)
2014-09-05T07:21:24Z|00024|ofproto|INFO|br-eth1: 3 flow_mods 10 s ago (2 adds, 1 deletes)

exectuing ip netns shows that my router is created and I can also ping the internal interface and the external interface using ip netns exec ID ping IF. The external interface on the router is also reachable from the outside. It's only the inside network which seems to have some issues.

Some help would be very much appreciated! :)

VM's can't get DHCP

Hi!

I'm running an openstack multinode configuration (3 compute nodes), IceHouse, CentOS 6.5. I use flat networking with VLAN tagging.

When I boot up and instance, it does not get any DHCP from the network node.

On I've set the VLAN range to be from 1401 to 1403, but when I boot up and instance and do this on the compute node: node; cat /var/log/neutron/openvswitch-agent.log | grep "tag=" I can only see networks tagged for VLAN 1 og 2.

When I do; tail /var/log/neutron/openvswitch-agent.log-f /var/log/neutron/openvswitch-agent.log I get this message constantly popping up;

2014-09-05 09:11:15.843 2804 INFO neutron.agent.securitygroups_rpc [req-0617459d-a623-4392-97a4-6336d44ec70a None] Security group member updated [u'ad057153-a11f-4f9b-b7fd-7f1131bb4cc0']
2014-09-05 09:11:21.703 2804 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'f686a831-afa2-4d2b-9059-7104c6a1cf35'])
2014-09-05 09:11:22.225 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port f686a831-afa2-4d2b-9059-7104c6a1cf35 updated. Details: {u'admin_state_up': True, u'network_id': u'bc31c8d0-f892-431b-a3f7-5c2b3a8b2504', u'segmentation_id': 1401, u'physical_network': u'physnet1', u'device': u'f686a831-afa2-4d2b-9059-7104c6a1cf35', u'port_id': u'f686a831-afa2-4d2b-9059-7104c6a1cf35', u'network_type': u'vlan'}
2014-09-05 09:11:22.226 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Assigning 2 as local vlan for net-id=bc31c8d0-f892-431b-a3f7-5c2b3a8b2504
2014-09-05 09:11:22.620 2804 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Configuration for device f686a831-afa2-4d2b-9059-7104c6a1cf35 completed.

On the network node; tail /var/log/neutron/openvswtich-agent.log

2014-09-05 08:55:01.753 5423 014-09-08 09:41:24.035 7085 DEBUG neutron.agent.linux.utils [-] [req-591a544b-4608-4b38-8dc9-b98359f7a5df None] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-ofctl', 'dump-flows', 'br-int', 'table=22'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-09-08 09:41:24.092 7085 DEBUG neutron.agent.linux.utils [req-591a544b-4608-4b38-8dc9-b98359f7a5df None] 
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-ofctl', 'dump-flows', 'br-int', 'table=22']
Exit code: 0
Stdout: 'NXST_FLOW reply (xid=0x4):\n cookie=0x0, duration=2793.91s, duration=1119.772s, table=22, n_packets=0, n_bytes=0, idle_age=2793, idle_age=1119, priority=0 actions=drop\n'
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74
2014-09-05 08:55:01.754 5423 2014-09-08 09:41:24.093 7085 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] [req-591a544b-4608-4b38-8dc9-b98359f7a5df None] Agent rpc_loop - iteration:1395 iteration:559 completed. Processed ports statistics: {'ancillary': {'removed': 0, 'added': 0}, 'regular': {'updated': 0, 'added': 0, 'removed': 0}}. Elapsed:0.061 Elapsed:0.058 rpc_loop /usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:1386

This log keeps growing with this The same message is popping up every 1 second. Is this packets being discarded by on the network node?

When I restart the openvswith service compute nodes hosting virtual machines.

The control-flows on the network node, the log shows;node seems ok as well(?);

2014-09-05T07:21:13Z|00026|fatal_signal|WARN|terminating with signal 15 (Terminated)
2014-09-05T07:21:13Z|00003|worker(worker)|INFO|worker process exiting
2014-09-05T07:21:13Z|00002|daemon(monitor)|INFO|pid 5095 died, killed (Terminated), exiting
2014-09-05T07:21:14Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2014-09-05T07:21:14Z|00002|worker(worker)|INFO|worker process started
2014-09-05T07:21:14Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2014-09-05T07:21:14Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2014-09-05T07:21:14Z|00004|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-2ddc5852-e5 device failed: No such device
2014-09-05T07:21:14Z|00005|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-99334d8e-e4 device failed: No such device
2014-09-05T07:21:14Z|00006|netdev|WARN|failed to set flags for network device qg-2ddc5852-e5: No such device
2014-09-05T07:21:14Z|00007|netdev|WARN|failed to set flags for network device qr-99334d8e-e4: No such device
2014-09-05T07:21:14Z|00008|bridge|INFO|bridge br-ex: using datapath ID 00000026554dbcec
2014-09-05T07:21:14Z|00009|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qg-2ddc5852-e5 device failed: No such device
2014-09-05T07:21:14Z|00010|netdev_linux|WARN|qg-2ddc5852-e5: removing policing failed: No such device
2014-09-05T07:21:14Z|00011|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt"
2014-09-05T07:21:14Z|00012|bridge|INFO|bridge br-eth1: using datapath ID 00000026554dbcea
2014-09-05T07:21:14Z|00013|connmgr|INFO|br-eth1: added service controller "punix:/var/run/openvswitch/br-eth1.mgmt"
2014-09-05T07:21:14Z|00014|bridge|INFO|bridge br-int: using datapath ID 000016215ea0d144
2014-09-05T07:21:14Z|00015|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-99334d8e-e4 device failed: No such device
2014-09-05T07:21:14Z|00016|netdev_linux|WARN|qr-99334d8e-e4: removing policing failed: No such device
2014-09-05T07:21:14Z|00017|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"
2014-09-05T07:21:14Z|00018|bridge|INFO|ovs-vswitchd (Open vSwitch) 1.11.0
2014-09-05T07:21:15Z|00019|bridge|INFO|bridge br-int: added interface int-br-eth1 on port 1
2014-09-05T07:21:15Z|00020|bridge|INFO|bridge br-eth1: added interface phy-br-eth1 on port 2
2014-09-05T07:21:24Z|00021|memory|INFO|8460 kB peak resident set size after 10.0 seconds
2014-09-05T07:21:24Z|00022|memory|INFO|facets:29 ports:9 rules:16 subfacets:35
2014-09-05T07:21:24Z|00023|ofproto|INFO|br-int: 5 flow_mods in the 1 s starting 10 s ago (4 adds, 1 deletes)
2014-09-05T07:21:24Z|00024|ofproto|INFO|br-eth1: 3 flow_mods 10 s ago (2 adds, 1 deletes)
[root@network ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1265.329s, table=0, n_packets=0, n_bytes=0, idle_age=1265, priority=3,in_port=5,dl_vlan=1401 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=1266.638s, table=0, n_packets=6, n_bytes=468, idle_age=1256, priority=2,in_port=5 actions=drop
 cookie=0x0, duration=1267.577s, table=0, n_packets=6, n_bytes=252, idle_age=997, priority=1 actions=NORMAL
 cookie=0x0, duration=1267.519s, table=22, n_packets=0, n_bytes=0, idle_age=1267, priority=0 actions=drop

[root@network ~]# ovs-ofctl dump-flows br-eth1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1358.995s, table=0, n_packets=0, n_bytes=0, idle_age=1358, priority=4,in_port=4,dl_vlan=1 actions=mod_vlan_vid:1401,NORMAL
 cookie=0x0, duration=1360.185s, table=0, n_packets=12, n_bytes=724, idle_age=1091, priority=2,in_port=4 actions=drop
 cookie=0x0, duration=1360.919s, table=0, n_packets=46, n_bytes=8142, idle_age=9, priority=1 actions=NORMAL

exectuing ip netns shows that And on one of the compute nodes hosting a virtual machine;

[root@compute1 ~]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1443.474s, table=0, n_packets=0, n_bytes=0, idle_age=1443, priority=3,in_port=7,dl_vlan=1401 actions=mod_vlan_vid:1,NORMAL
 cookie=0x0, duration=1445.137s, table=0, n_packets=6, n_bytes=468, idle_age=1435, priority=2,in_port=7 actions=drop
 cookie=0x0, duration=1446.381s, table=0, n_packets=0, n_bytes=0, idle_age=1446, priority=1 actions=NORMAL
 cookie=0x0, duration=1446.309s, table=22, n_packets=0, n_bytes=0, idle_age=1446, priority=0 actions=drop

[root@compute1 ~]# ovs-ofctl dump-flows br-eth5
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=1479.978s, table=0, n_packets=0, n_bytes=0, idle_age=1479, priority=4,in_port=4,dl_vlan=1 actions=mod_vlan_vid:1401,NORMAL
 cookie=0x0, duration=1481.495s, table=0, n_packets=6, n_bytes=468, idle_age=1471, priority=2,in_port=4 actions=drop
 cookie=0x0, duration=1482.454s, table=0, n_packets=0, n_bytes=0, idle_age=1482, priority=1 actions=NORMAL

I guess this is some problems with my router is created configuration, but I haven't found any good documented routine on how to configure it. The plugin.ini file for my network node and I can also ping the internal interface and the external interface using ip netns exec ID ping IF. The external interface on the router is also reachable from the outside. It's only the inside network which seems to have some issues.

compute nodes;

[ml2]
type_drivers = flat,vlan
tenant_network_types = vlan,flat
mechanism_drivers = openvswitch
[ml2_type_flat]
flat_networks = physnet1
[ml2_type_vlan]
network_vlan_ranges = physnet1:1401:1403
[ml2_type_gre]
[ml2_type_vxlan]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
[ovs]
bridge_mappings = physnet1:br-ethX

Some help would be very much appreciated! :)