Ask Your Question
1

Icehouse Neutron ML2 (OVS & GRE) MTU Problem?

asked 2014-08-06 20:10:49 -0500

stacker_filo_23 gravatar image

Hi all,

I have built an OpenStack IceHouse cluster on Ubuntu 14.04 by following the instructions http://docs.openstack.org/icehouse/install-guide/install/apt/content/ (here).

My environment looks like the following:

  1. One (1) Controller node
  2. One (1) Network node
  3. Three (3) Compute nodes

This http://docs.openstack.org/icehouse/install-guide/install/apt/content/figures/1/figures/installguide_arch-neutron.png (diagram) depicts how OpenStack services are distributed on the nodes.

I can launch VMs on a tenant network. They successfully get an IP on the internal network via dhcp. I can successfully attach a floating IP public to the VMs. I can ping both the internal and floating IPs. So far so good.

Now my problems:

  1. I can't ssh into the instances. ssh -v shows connection but just hangs.
  2. During bootup, the VMs can't reach the metadata server at http://169.254.169.254 . No key injection. Explains #1.

But after bootup, I can get into a cirros image through console and curl http://169.254.169.254 successfully. The VM can also reach the outside world (the internet). I can resolve IP's using google's 8.8.8.8. But telnet http://yahoo.com 80 fails.

Searching this forum, I have found two clues:

  1. The known MTU problem. Something like https://ask.openstack.org/en/question/31911/metadata-query-hanging/ (this). But the fix of trying to set VM MTU to 1400 is not working. Cirros doesn't respect mtu via DHCP. On the other Ubuntu images, I can't login to check.
  2. Metadata proxy server misconfiguration. I've gone over this several times and I can't note a misconfiguration. All logs seem to show successful metadata being given out.

Here are some pertinent config files from the environment:

nova.conf from Controller node

root@controller:~# cat /etc/nova/nova.conf
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
#
rpc_backend = rabbit
rabbit_host = controller
rabbit_password = Fusiondc10!
#
my_ip = 10.20.81.101
vncserver_listen = 10.20.81.101
vncserver_proxyclient_address = 10.20.81.101
#
auth_strategy = keystone
#
network_api_class = nova.network.neutronv2.api.API
neutron_url = http://controller:9696
neutron_auth_strategy = keystone
neutron_admin_tenant_name = service
neutron_admin_username = neutron
neutron_admin_password = Fusiondc10!
neutron_admin_auth_url = http://controller:35357/v2.0
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
security_group_api = neutron
service_neutron_metadata_proxy = true
neutron_metadata_proxy_shared_secret = e773b738013e04efd8f1
#
libvirt_images_type=rbd
libvirt_images_rbd_pool=vms
libvirt_images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
rbd_secret_uuid=94b68ab4-a9d7-4a53-8d49-c80fde07a2bc
libvirt_inject_password=false
libvirt_inject_key=false
libvirt_inject_partition=-1
libvirt_live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST"


[database]
connection = mysql://nova:Fusiondc10!@controller/nova

[keystone_authtoken]
auth_uri = http://controller:5000
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = Fusiondc10!
root@controller:~#

neutron.conf on Network Server

root@neutron1:~# egrep -v '^$|^#' /etc/neutron/neutron.conf
[DEFAULT]
verbose = True
state_path = /var/lib/neutron
lock_path = $state_path/lock
bind_host = 10.20.81.101 ...
(more)
edit retag flag offensive close merge delete

Comments

Here is more info I found that points towards an MTU fragmentation issue. Here is what happens in tcpdump when I'm doing curl to 169.254.169.254/2009-04-04/meta-data from my cirros guest. It returns successfully. But it takes about 3 minutes. This causes the timeout during boot up.

I see tons of cksum (incorrect) and [DF] "do not fragment" flags. Changing the mtu of the VM doesn't help.

   root@neutron1:~# ip netns exec qrouter-8c781c2b-44ee-4e85-913d-20499dcce34f tcpdump -vv -i qr-930b8a5f-62 host 192.168.1.26

        192.168.1.26.45007 > 169.254.169.254.http: Flags [P.], cksum 0x1689 (incorrect -> 0x2b46), seq 4294966204:4294966367, ack 1, win 1700, options [nop,nop,TS val 309904 ecr 91698904], length 163
    23:43:50.168433 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        169.254.169.254.http > 192.168.1.26.45007 ...
(more)
stacker_filo_23 gravatar imagestacker_filo_23 ( 2014-08-07 02:02:37 -0500 )edit

have you had any success in solving this issue?

aw00kie gravatar imageaw00kie ( 2014-08-20 18:06:30 -0500 )edit

I finally made it worked last week by clean installing under the updated OpenStack Installation Guide for Ubuntu per Sep 19, 2014. There was an addition to the dhcp_agent.ini on Network node regarding the MTU. All is good.:)

chrone gravatar imagechrone ( 2014-09-24 00:06:12 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-08-07 04:01:41 -0500

foexle gravatar image

Hi,

some hint:

nova.conf

libvirt_inject_partition=-1 should be -2
libvirt_inject_key=false => true
libvirt_nonblocking = True
vif_plugging_timeout = 0

please post your plugin configuration file (ml2) and metadata. In addition check if your metadata secret in all config files are the same.

Ahh not to forget, check your sec rules :) you need to open http/https to the outside otherwise your vm can't communicate with metadata service.

Cheers Heiko

edit flag offensive delete link more

Comments

Thank you for the response, mate.

  1. I had attached ml2_conf.ini and metadata_agent.ini to my originalpost. Hit (more).
  2. I made the nova.conf edits you suggested. It didn't help. That was for ceph (to avoid filesystem injection and to rely on cloud-init instead).
  3. Security rules are in place.
stacker_filo_23 gravatar imagestacker_filo_23 ( 2014-08-07 14:58:43 -0500 )edit

What happens when you set MTU size to 1400 from within the Cirros VM? Login from console and set the MTU. FYI... I have the same issue. On Ubuntu I enabled root login and set a password before uploading it into glance so I could login from the console.

jay-janardhan gravatar imagejay-janardhan ( 2014-08-07 15:36:53 -0500 )edit

Setting the MTU to 1400 within CirrOS doesn't seem to improve things. I wish that trick had worked for me. I have my dhcp_conf.ini set so that it will give all VMs 1400 MTU. Not sure if it's working on Ubuntu images or not. I'm going to try your method of uploading a root'able Ubuntu image to glace. Clever.

stacker_filo_23 gravatar imagestacker_filo_23 ( 2014-08-07 15:48:10 -0500 )edit

Could you paste: /etc/neutron/dnsmasq/dnsmasq-neutron.conf you can try: dhcp-option-force=26,1454

foexle gravatar imagefoexle ( 2014-08-08 02:54:51 -0500 )edit

HI Foexle, Here is my dnsmasq-neutron.conf file. I've tried both 1400 and 1450 to no avail.

root@neutron1:~# cat /etc/neutron/dnsmasq/dnsmasq-neutron.conf
dhcp-option-force=26,1400
stacker_filo_23 gravatar imagestacker_filo_23 ( 2014-08-08 13:22:17 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

4 followers

Stats

Asked: 2014-08-06 20:10:49 -0500

Seen: 1,982 times

Last updated: Aug 07 '14