Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

After a hard reboot of nodes network is not starting

After we restarted all the nodes after some networking issues on the hardware in the DC. The openvswitch seems to be messed up and using the wrong interface? We attempted to start the instances on the nodes but it gets a permission denied error. I notices the following errors on the ovs-vsctl show command. also dpdk does not work either. Please let me know what other logs i can get.

failed services: ovs-vswitchd.service loaded failed failed Open vSwitch Forwarding Unit ● sdkserver.service loaded failed failed zVM SDK API server ● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured

logs: 2020-05-03T16:21:04.152Z|00007|dpdk|INFO|Using DPDK 18.11.2 2020-05-03T16:21:04.152Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2020-05-03T16:21:04.152Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-05-03T16:21:04.152Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-05-03T16:21:04.152Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-05-03T16:21:04.152Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-05-03T16:21:04.152Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x03 --socket-mem 1024,1024 --socket-limit 1024,1024. 2020-05-03T16:21:04.153Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2020-05-03T16:21:04.153Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-05-03T16:21:04.160Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-05-03T16:21:04.178Z|00017|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00018|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00019|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00020|dpdk|WARN|EAL: No free hugepages reported in hugepages-1048576kB 2020-05-03T16:21:04.178Z|00021|dpdk|ERR|EAL: Cannot get hugepage information. 2020-05-03T16:21:04.178Z|00022|dpdk|EMER|Unable to initialize DPDK: Permission denied 2020-05-03T16:21:04.203Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped) 2020-05-03T16:21:04.203Z|00003|daemon_unix|EMER|could not detach from foreground session 2020-05-03T16:21:04.614Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2020-05-03T16:21:04.617Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2020-05-03T16:21:04.617Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2020-05-03T16:21:04.617Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2020-05-03T16:21:04.617Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2020-05-03T16:21:04.617Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2020-05-03T16:21:04.619Z|00007|dpdk|INFO|Using DPDK 18.11.2 2020-05-03T16:21:04.619Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2020-05-03T16:21:04.619Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-05-03T16:21:04.619Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-05-03T16:21:04.619Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-05-03T16:21:04.619Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-05-03T16:21:04.619Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x03 --socket-mem 1024,1024 --socket-limit 1024,1024. 2020-05-03T16:21:04.620Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2020-05-03T16:21:04.620Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-05-03T16:21:04.627Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-05-03T16:21:04.644Z|00017|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00018|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00019|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00020|dpdk|WARN|EAL: No free hugepages reported in hugepages-1048576kB 2020-05-03T16:21:04.644Z|00021|dpdk|ERR|EAL: Cannot get hugepage information. 2020-05-03T16:21:04.644Z|00022|dpdk|EMER|Unable to initialize DPDK: Permission denied 2020-05-03T16:21:04.713Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped) 2020-05-03T16:21:04.713Z|00003|daemon_unix|EMER|could not detach from foreground session

interfaces root@web2:/home/ubuntu# ip a 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq master br-eth0 state UP group default qlen 1000 link/ether 80:18:44:eb:89:0c brd ff:ff:ff:ff:ff:ff 3: eno2: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 80:18:44:eb:89:0d brd ff:ff:ff:ff:ff:ff 4: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 80:18:44:eb:89:0e brd ff:ff:ff:ff:ff:ff inet 172.27.1.52/24 brd 172.27.1.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::8218:44ff:feeb:890e/64 scope link valid_lft forever preferred_lft forever 5: eno4: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 80:18:44:eb:89:0f brd ff:ff:ff:ff:ff:ff 6: br-eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e6:43:b0:58:73:6d brd ff:ff:ff:ff:ff:ff inet 172.27.1.2/24 brd 172.27.1.255 scope global br-eth0 valid_lft forever preferred_lft forever inet6 fe80::e443:b0ff:fe58:736d/64 scope link valid_lft forever preferred_lft forever 7: lxdbr0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 7a:1b:9b:84:13:52 brd ff:ff:ff:ff:ff:ff inet 10.229.170.1/24 scope global lxdbr0 valid_lft forever preferred_lft forever inet6 fe80::781b:9bff:fe84:1352/64 scope link valid_lft forever preferred_lft forever 9: vethDNIYRU@if8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:ca:a5:bf:35:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::fcca:a5ff:febf:35bb/64 scope link valid_lft forever preferred_lft forever 11: vethN3N52G@if10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:6d:32:8e:a6:34 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::fc6d:32ff:fe8e:a634/64 scope link valid_lft forever preferred_lft forever 13: vethOT6VW5@if12: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:50:2c:00:cf:4a brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::fc50:2cff:fe00:cf4a/64 scope link valid_lft forever preferred_lft forever 15: vethCUL9SM@if14: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:d5:d5:da:49:ab brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::fcd5:d5ff:feda:49ab/64 scope link valid_lft forever preferred_lft forever 17: vethFA89DV@if16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:17:98:4f:92:50 brd ff:ff:ff:ff:ff:ff link-netnsid 4 inet6 fe80::fc17:98ff:fe4f:9250/64 scope link valid_lft forever preferred_lft forever 19: veth8UO3NM@if18: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:4b:3a:ff:97:8c brd ff:ff:ff:ff:ff:ff link-netnsid 5 inet6 fe80::fc4b:3aff:feff:978c/64 scope link valid_lft forever preferred_lft forever

ovs-vsctl: root@web2:/home/ubuntu# ovs-vsctl show 5f494674-8b92-453d-83d9-4122ca33962a Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-eth0" Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port "phy-br-eth0" Interface "phy-br-eth0" type: patch options: {peer="int-br-eth0"} Port "br-eth0" Interface "br-eth0" type: internal error: "could not add network device br-eth0 to ofproto (File exists)" Bridge br-data Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port phy-br-data Interface phy-br-data type: patch options: {peer=int-br-data} Port br-data Interface br-data type: internal Port "eth0" Interface "eth0" error: "could not add network device eth0 to ofproto (Device or resource busy)" Bridge br-ex Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Bridge br-tun Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port int-br-data Interface int-br-data type: patch options: {peer=phy-br-data} Port "int-br-eth0" Interface "int-br-eth0" type: patch options: {peer="phy-br-eth0"} Port br-int Interface br-int type: internal ovs_version: "2.12.0"

After a hard reboot of nodes network is not starting

well I tried to fix the formatting so people could understand the problem. but I do not have 10 points to upload anything. Seriously you guy

After we restarted all the nodes after some networking issues on the hardware in the DC. The openvswitch seems to be messed up and using the wrong interface? We attempted to start the instances on the nodes but it gets a permission denied error. I notices the following errors on the ovs-vsctl show command. also dpdk does not work either. Please let me know what other logs i can get.

failed services: ovs-vswitchd.service loaded failed failed Open vSwitch Forwarding Unit ● sdkserver.service loaded failed failed zVM SDK API server ● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured

logs: 2020-05-03T16:21:04.152Z|00007|dpdk|INFO|Using DPDK 18.11.2 2020-05-03T16:21:04.152Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2020-05-03T16:21:04.152Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-05-03T16:21:04.152Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-05-03T16:21:04.152Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-05-03T16:21:04.152Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-05-03T16:21:04.152Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x03 --socket-mem 1024,1024 --socket-limit 1024,1024. 2020-05-03T16:21:04.153Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2020-05-03T16:21:04.153Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-05-03T16:21:04.160Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-05-03T16:21:04.178Z|00017|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00018|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00019|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.178Z|00020|dpdk|WARN|EAL: No free hugepages reported in hugepages-1048576kB 2020-05-03T16:21:04.178Z|00021|dpdk|ERR|EAL: Cannot get hugepage information. 2020-05-03T16:21:04.178Z|00022|dpdk|EMER|Unable to initialize DPDK: Permission denied 2020-05-03T16:21:04.203Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped) 2020-05-03T16:21:04.203Z|00003|daemon_unix|EMER|could not detach from foreground session 2020-05-03T16:21:04.614Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2020-05-03T16:21:04.617Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2020-05-03T16:21:04.617Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2020-05-03T16:21:04.617Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2020-05-03T16:21:04.617Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2020-05-03T16:21:04.617Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2020-05-03T16:21:04.619Z|00007|dpdk|INFO|Using DPDK 18.11.2 2020-05-03T16:21:04.619Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2020-05-03T16:21:04.619Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2020-05-03T16:21:04.619Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2020-05-03T16:21:04.619Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2020-05-03T16:21:04.619Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2020-05-03T16:21:04.619Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x03 --socket-mem 1024,1024 --socket-limit 1024,1024. 2020-05-03T16:21:04.620Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2020-05-03T16:21:04.620Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2020-05-03T16:21:04.627Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/dpdk/rte/mp_socket 2020-05-03T16:21:04.644Z|00017|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00018|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00019|dpdk|WARN|EAL: No free hugepages reported in hugepages-2048kB 2020-05-03T16:21:04.644Z|00020|dpdk|WARN|EAL: No free hugepages reported in hugepages-1048576kB 2020-05-03T16:21:04.644Z|00021|dpdk|ERR|EAL: Cannot get hugepage information. 2020-05-03T16:21:04.644Z|00022|dpdk|EMER|Unable to initialize DPDK: Permission denied 2020-05-03T16:21:04.713Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped) 2020-05-03T16:21:04.713Z|00003|daemon_unix|EMER|could not detach from foreground session

interfaces root@web2:/home/ubuntu# ip a 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq master br-eth0 state UP group default qlen 1000 link/ether 80:18:44:eb:89:0c brd ff:ff:ff:ff:ff:ff 3: eno2: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 80:18:44:eb:89:0d brd ff:ff:ff:ff:ff:ff 4: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 80:18:44:eb:89:0e brd ff:ff:ff:ff:ff:ff inet 172.27.1.52/24 brd 172.27.1.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::8218:44ff:feeb:890e/64 scope link valid_lft forever preferred_lft forever 5: eno4: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 80:18:44:eb:89:0f brd ff:ff:ff:ff:ff:ff 6: br-eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e6:43:b0:58:73:6d brd ff:ff:ff:ff:ff:ff inet 172.27.1.2/24 brd 172.27.1.255 scope global br-eth0 valid_lft forever preferred_lft forever inet6 fe80::e443:b0ff:fe58:736d/64 scope link valid_lft forever preferred_lft forever 7: lxdbr0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 7a:1b:9b:84:13:52 brd ff:ff:ff:ff:ff:ff inet 10.229.170.1/24 scope global lxdbr0 valid_lft forever preferred_lft forever inet6 fe80::781b:9bff:fe84:1352/64 scope link valid_lft forever preferred_lft forever 9: vethDNIYRU@if8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:ca:a5:bf:35:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::fcca:a5ff:febf:35bb/64 scope link valid_lft forever preferred_lft forever 11: vethN3N52G@if10: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:6d:32:8e:a6:34 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::fc6d:32ff:fe8e:a634/64 scope link valid_lft forever preferred_lft forever 13: vethOT6VW5@if12: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:50:2c:00:cf:4a brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::fc50:2cff:fe00:cf4a/64 scope link valid_lft forever preferred_lft forever 15: vethCUL9SM@if14: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:d5:d5:da:49:ab brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::fcd5:d5ff:feda:49ab/64 scope link valid_lft forever preferred_lft forever 17: vethFA89DV@if16: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:17:98:4f:92:50 brd ff:ff:ff:ff:ff:ff link-netnsid 4 inet6 fe80::fc17:98ff:fe4f:9250/64 scope link valid_lft forever preferred_lft forever 19: veth8UO3NM@if18: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master br-eth0 state UP group default qlen 1000 link/ether fe:4b:3a:ff:97:8c brd ff:ff:ff:ff:ff:ff link-netnsid 5 inet6 fe80::fc4b:3aff:feff:978c/64 scope link valid_lft forever preferred_lft forever

ovs-vsctl: root@web2:/home/ubuntu# ovs-vsctl show 5f494674-8b92-453d-83d9-4122ca33962a Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-eth0" Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port "phy-br-eth0" Interface "phy-br-eth0" type: patch options: {peer="int-br-eth0"} Port "br-eth0" Interface "br-eth0" type: internal error: "could not add network device br-eth0 to ofproto (File exists)" Bridge br-data Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port phy-br-data Interface phy-br-data type: patch options: {peer=int-br-data} Port br-data Interface br-data type: internal Port "eth0" Interface "eth0" error: "could not add network device eth0 to ofproto (Device or resource busy)" Bridge br-ex Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Port phy-br-ex Interface phy-br-ex type: patch options: {peer=int-br-ex} Bridge br-tun Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int Controller "tcp:127.0.0.1:6633" fail_mode: secure datapath_type: system Port int-br-ex Interface int-br-ex type: patch options: {peer=phy-br-ex} Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port int-br-data Interface int-br-data type: patch options: {peer=phy-br-data} Port "int-br-eth0" Interface "int-br-eth0" type: patch options: {peer="phy-br-eth0"} Port br-int Interface br-int type: internal ovs_version: "2.12.0"