Icehouse ML2 w/ SR-IOV (VLAN): Can't ping VM IP
Hello,
I'm trying OpenStack Icehouse ML2 w/ SR-IOV (VLAN) on CentOS7 and Mellanox ConnectX-3 cards (40gE):
(1) https://wiki.openstack.org/wiki/Mellanox-Neutron-Icehouse-Redhat-Ethernet (https://wiki.openstack.org/wiki/Mella...) (2) https://wiki.openstack.org/wiki/Nova-neutron-sriov (https://wiki.openstack.org/wiki/Nova-...) (3) https://community.mellanox.com/docs/DOC-1317 (https://community.mellanox.com/docs/D...)
After setting SR-IOV w/ ConnectX-3 while following (3), I also verified that vlan is working among systems.
I installed OpenStack Icehouse using packstack on single node (all-in-one). And then, I modified all the configurations for SR-IOV while following (1), except /etc/neutron/dhcp_agent.ini. I fixed the interface_driver from BridgeInterfaceDriver to OVSInterfaceDriver in /etc/neutron/dhcp_agent.ini.
Even though VM can get the DHCP IP, host and VM can't ping each other.
[root@gpu6 ~(keystone_admin)]# cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core port_type_array=2,2 num_vfs=16 probe_vf=0 enable_64b_cqe_eqe=0 log_num_mgm_entry_size=-1
[root@gpu6 ~(keystone_admin)]# neutron net-list
+--------------------------------------+--------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+--------+-----------------------------------------------------+
| 1c555886-f026-4727-a2e6-99913e383bf2 | net40g | afdeec0e-6b9f-421a-9a5b-421a77c283d8 192.168.2.0/24 |
+--------------------------------------+--------+-----------------------------------------------------+
[root@gpu6 ~(keystone_admin)]# neutron subnet-list
+--------------------------------------+-------------+----------------+--------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+-------------+----------------+--------------------------------------------------+
| afdeec0e-6b9f-421a-9a5b-421a77c283d8 | demo-subnet | 192.168.2.0/24 | {"start": "192.168.2.2", "end": "192.168.2.254"} |
+--------------------------------------+-------------+----------------+--------------------------------------------------+
[root@gpu6 ~(keystone_admin)]# neutron port-list
+--------------------------------------+------------+-------------------+------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------------+-------------------+------------------------------------------------------------------------------------+
| 385600c0-fafa-4e15-b0b4-83f780e26daf | | fa:16:3e:ce:2b:5f | {"subnet_id": "afdeec0e-6b9f-421a-9a5b-421a77c283d8", "ip_address": "192.168.2.2"} |
| 9a291386-c020-4cfd-9e11-bc98fa418566 | | fa:16:3e:90:d8:bc | {"subnet_id": "afdeec0e-6b9f-421a-9a5b-421a77c283d8", "ip_address": "192.168.2.1"} |
| e0f81bbb-2da3-4ba0-9bba-3f90a79fd9a7 | sriov_port | fa:16:3e:8b:83:76 | {"subnet_id": "afdeec0e-6b9f-421a-9a5b-421a77c283d8", "ip_address": "192.168.2.7"} |
+--------------------------------------+------------+-------------------+------------------------------------------------------------------------------------+
[root@gpu6 ~(keystone_admin)]# neutron port-show sriov_port
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | gpu6.east.isi.edu |
| binding:profile | {"physical_network": "physnet1"} |
| binding:vif_details | {"port_filter": false} |
| binding:vif_type | hostdev |
| binding:vnic_type | direct |
| device_id | c77e4d18-e9f3-4b35-9f8c-a544412a7ee6 |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "afdeec0e-6b9f-421a-9a5b-421a77c283d8", "ip_address": "192.168.2.7"} |
| id | e0f81bbb-2da3-4ba0-9bba-3f90a79fd9a7 |
| mac_address | fa:16:3e:8b:83:76 |
| name | sriov_port |
| network_id | 1c555886-f026-4727-a2e6-99913e383bf2 |
| security_groups | 70114155-0fb7-4d13-979c-c080e1c4f620 |
| status | ACTIVE |
| tenant_id | 79f18e79626142ada326878a8622d066 |
+-----------------------+------------------------------------------------------------------------------------+
[root@gpu6 ~(keystone_admin)]# ip netns
qdhcp-1c555886-f026-4727-a2e6-99913e383bf2
qrouter-4d297bce-3888-4036-9b63-e61028f9ff8f
[root@gpu6 ~(keystone_admin)]# ip netns exec qdhcp-1c555886-f026-4727-a2e6-99913e383bf2 ping -c1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.027 ms
--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.027/0.027/0.027/0.000 ms
[root@gpu6 ~(keystone_admin)]# ip netns exec qdhcp-1c555886-f026-4727-a2e6-99913e383bf2 ping -c1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.495 ms
--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.495/0.495/0.495/0.000 ms
[root@gpu6 ~(keystone_admin)]# ip netns exec qdhcp-1c555886-f026-4727-a2e6-99913e383bf2 ping -c1 192.168.2.7
PING 192.168.2.7 (192.168.2.7) 56(84) bytes of data.
^C
--- 192.168.2.7 ping statistics ---
1 packets transmitted, 0 received, 100 ...
I'm trying to compare (1) default OpenStack testbed using LibvirtGenericVIFDriver to (2) SR-IOV+VLAN testbed using MlxEthVIFDriver. (1) works fine, but (2) doesn't have qv* device (ip link / ovs-ofctl show br-int) and doesn't have iptables rules that are related to neutron-openvswi-* and VM IP/port.