# lukas.pustina's profile - activity

2014-10-14 13:44:55 -0600 commented answer How to bring up second network node

Hi Sam, first of all thanks a lot for making this clear. Finally, a definitive statement. It makes sense and explains the behavior I observe. I found the script at GitHub for reference.

2014-10-09 08:33:25 -0600 commented answer How to restart Neutron properly?

2014-10-09 08:33:11 -0600 commented answer How to restart Neutron properly?

Okay, just checking: So it is expected that you have to restart the neutron service on the compute nodes after a network node has been restarted? Do you know what? Which information is lost during the reboot?

2014-10-09 08:30:22 -0600 asked a question How to bring up second network node

We followed the OpenStack High Availability Guide. For all stateless services we use HAproxy. The only services that need special attention are Neutron DHCP agent, Neutron L3 agent, Neutron metadata agent for which we are trying to use an active/passive failover. Unfortunately, this does not work at all with the following observations. Hopefully you guys have some insight to explain them.

## Our Setup

We use Icehouse with two controllers also running the network node software stack with OVS. The primary network node is called control01 and the secondary control02. All nodes use Ubuntu 14.04 and ovs-vsctl shows a mesh connecting all network and compute nodes.

## Our Observations

### 1. Stopping Neutron Agents does not work

If we try to stop the Neutron agents on control01, the dnsmasq process remain as well as all dhcp and router namespaces. We check this using ip nets.

After stopping the agents, the network connectivity of running VMs is still okay.

### 2. Starting Neutron Agents on secondary does not work

If we try to start the Neutron agents on control02, only the dhcp namespaces are created in contrast to control01. Since the configuration of both nodes is identical, we cannot understand, why this happens.

I'm grateful for any insight.

2014-10-09 08:14:43 -0600 commented answer How to restart Neutron properly?

So, you're saying, I shall restart all neutron services on the compute nodes after I rebooted the control node?

2014-10-09 07:58:58 -0600 asked a question How to restart Neutron properly?

We have a control node that also acts as the network node and multiple compute nodes running Icehouse.

Sometimes it's necessary to restart the control node due to maintenance. After the control node is back up, the network connections between virtual machines running on the computes nodes as well as their connections to the outside are often broken. Only a restart of the compute node helps.

This is ridiculous. Do you guys have any advice on how to repair this situation without the need to reboot the whole cluster of nodes? I'd be happy for any workflow you successfully use.

2014-09-17 04:35:46 -0600 answered a question Irregular Network Latency to Instances

As commented before, I observed the same problem and solved it by explicitly activating the VHOST_NET driver in Ubuntu 14.04. Please see my answer to question 46303.

2014-09-17 04:26:18 -0600 answered a question Packets to VM are dropped in case of multiple senders

I found the answer. In short, Ubuntu 14.04 has an unintuitive default setting regarding the use of the para-virtual network driver VHOST_NET.

\$ cat /etc/default/qemu-kvm
# To disable qemu-kvm's page merging feature, set KSM_ENABLED=0 and
# sudo restart qemu-kvm
KSM_ENABLED=1
SLEEP_MILLISECS=200
# To load the vhost_net module, which in some cases can speed up
# network performance, set VHOST_NET_ENABLED to 1.
VHOST_NET_ENABLED=0

# Set this to 1 if you want hugepages to be available to kvm under
# /run/hugepages/kvm
KVM_HUGEPAGES=0


To solve the problem, I just had to set VHOST_NET_ENABLED to 1 and restart all qemu-kvm processes, i.e., restart the virtual machines.

I blogged about how I found and fixed the problem in detail here: https://blog.codecentric.de/en/2014/0... https://blog.codecentric.de/en/2014/0... https://blog.codecentric.de/en/2014/0...

2014-08-31 06:04:34 -0600 edited question Packets to VM are dropped in case of multiple senders

Setup

We are running OpenStack Icehouse on Ubuntu 14.04 with a dedicated network controller for Neutron networking and 4 compute nodes using Open vSwitch and GRE tunneling. On all bare metals, GRO is disabled. All VMs are running Ubuntu 14.04 as well.

Problem

A VM called time01 runs an NTP server. It has a floating IP so it can be reached via the external network. The corresponding security group permits ICMP, UDP, TCP ingress connections on all ports. All bare metals and other VM use this server to update their clocks using NTP triggered by cron at the same time. We observed that the time sync using ntpdate does not work correctly.

Reduced Problem

We were able to reduced the problem to the following scenario. If we use http://www.bitwizard.nl/mtr/ (mtr) using ICMP, TCP or UDP packets from any hosts (VM or bare metal) to traceroute time01 all works fine. If we start a second mtr from another host, the first mtr process suddenly starts to show packet loss. Over time, this happens to the second mtr process as well.

If we run tcpdump on time01 we only see the first ICMP, UDP or TCP packet arrive. Subsequent packets never show up in tcpdump.

It seems like the NAT router for time01 gets confused. We appreciate any help and can provide whatever additional information is necessary to narrow down the problem.

Update 1

• baremetal network: 10.102.2.0/24
• os-mgmt network: 10.102.6.0/24
• os-data network: 10.102.7.0/24
• os-ext network: 10.102.8.0/24

iptables chain for security group of time01 tenant: neutron-openvswi-ieb60ba28-b

iptables -L -v on network node

Chain INPUT (policy ACCEPT 176M packets, 97G bytes)
pkts bytes target     prot opt in     out     source               destination
176M   97G neutron-openvswi-INPUT  all  --  any    any     anywhere             anywhere
176M   97G nova-api-INPUT  all  --  any    any     anywhere             anywhere
0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:domain
0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:domain
0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere             udp dpt:bootps
0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
0     0 neutron-filter-top  all  --  any    any     anywhere             anywhere
0     0 neutron-openvswi-FORWARD  all  --  any    any     anywhere             anywhere
0     0 nova-filter-top  all  --  any    any     anywhere             anywhere
0     0 nova-api-FORWARD  all  --  any    any     anywhere             anywhere
0     0 ACCEPT     all  --  any    virbr0  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
0     0 ACCEPT     all  --  virbr0 any     192.168.122.0/24     anywhere
0     0 ACCEPT     all  --  virbr0 virbr0  anywhere             anywhere
0     0 REJECT     all  --  any    virbr0  anywhere             anywhere             reject-with icmp-port-unreachable
0     0 REJECT     all  --  virbr0 any     anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 167M packets, 41G bytes)
pkts bytes target     prot opt in     out     source               destination
167M   41G neutron-filter-top  all  --  any    any     anywhere             anywhere
167M   41G neutron-openvswi-OUTPUT  all  --  any    any     anywhere             anywhere
167M   41G nova-filter-top  all  --  any ...