Ask Your Question
0

VM network interface reordering

asked 2015-08-18 11:58:33 -0500

joshuamiller01 gravatar image

In my environment I have multiple network interfaces on my VMs. I've found that 'nova boot' always honors the order of the NICs, but that oftentimes (roughly 50% out of 100 VM sample) the interfaces are reordered following stop / start / rebuild actions. For the VMs which are affected, I can observe in the xml definitions that the MACs on the PCI buses are swapped. From a networking perspective, everything still works: the interfaces can come up, are mapped to the right VLANs, gets addresses from DHCP, pass traffic, etc. However, I've found that there are a number of things in my environment that expect the interface ordering to stay the same throughout the life of the system.

The most recent problem, and one I haven't been able to think up a solution for, has been the following: routing for the metadata service ( i.e. http://169.254.169.254/latest/meta-da... ) is facilitated by static routes served out by DHCP, but with multiple NICs on the VM, the VM gets multiple static routes for the same destination. It seems the metadata service knows the host by its "first" IP, such that when the routes conflict and the requests go out the second interface, the metadata service fails. I thought I had beat this by modifying the dhcp client config on my VMs to not request static routes from the second interface ("eth1"), which works, but if the interface order isn't dependable, "eth1" doesn't really mean anything. I can cover the stop / start case with udev rules that force the interface names based on MAC address, but in the rebuild case, this isn't an option.

The core problem: how can I force my NICs to stay in the same order as was originally defined by 'nova boot' ?

Some details on my components: juno, kvm, ubuntu Trusty hosts, neutron with VLANs, running ubuntu VMs

edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted
1

answered 2015-08-18 14:23:06 -0500

dbaxps gravatar image

updated 2015-08-19 13:51:43 -0500

Don't create new interfaces via Nova CLI when booting instance.
You may install libguestfs tools package ( http://www.jaddog.org/2013/04/04/moun... )

Mount qcow2 image prior to uploading  to glance and add /etc/sysconfig/network-scripts/ifcfg-eth1 or undertake similar actions on Ubuntu. Actually,you may create 2,3,4,.... VNICs  on qcow2 image and then upload it to glance.  Libguestfs-tools package is available on any RH OS and Ubuntu as well.

Details here http://bderzhavets.blogspot.com/2014/...

[boris@fedora22wks Downloads]$  guestfish --rw -a vivid-server-cloudimg-amd64-disk1.img

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
 100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
><fs>  list-filesystems
/dev/sda1: ext4
><fs>  mount /dev/sda1 /
><fs> ls /etc/network/interfaces.d
eth0.cfg
><fs> cp  /etc/network/interfaces.d/eth0.cfg /etc/network/interfaces.d/eth1.cfg
><fs> edit /etc/network/interfaces.d/eth1.cfg
><fs> cat /etc/network/interfaces.d/eth1.cfg
# The primary network interface
auto eth1
iface eth1 inet dhcp

><fs> ls  /etc/network/interfaces.d/
eth0.cfg
eth1.cfg
><fs> cat /etc/network/interfaces.d/eth0.cfg
# The primary network interface
auto eth0
iface eth0 inet dhcp

><fs> cat /etc/network/interfaces.d/eth1.cfg
# The primary network interface
auto eth1
iface eth1 inet dhcp

><fs> 
[root@fedora22wks ~(keystone_demo)]# neutron subnet-list
+--------------------------------------+-----------------+----------------+----------------------------------------------------+
| id                                   | name            | cidr           | allocation_pools                                   |
+--------------------------------------+-----------------+----------------+----------------------------------------------------+
| 1afe4413-ca33-4f4f-a3db-d4fcd43c6160 | sub_private_net | 30.0.0.0/24    | {"start": "30.0.0.10", "end": "30.0.0.254"}        |
| 7de660f0-09aa-4e4c-9cc3-4e3386304814 | sub_demo_net    | 50.0.0.0/24    | {"start": "50.0.0.10", "end": "50.0.0.254"}        |
| 248eae76-7411-42de-baed-f24b775afdd9 | sub_public      | 192.168.1.0/24 | {"start": "192.168.1.150", "end": "192.168.1.254"} |
+--------------------------------------+-----------------+----------------+----------------------------------------------------+

image description
Picture bellow is pretty stable after install GNOME 3 Desktop on VM :

[   27.522663] cloud-init[1423]: Cloud-init v. 0.7.6 running 'init' at Wed, 19 Aug 2015 18:48:02 +0000. Up 27.51 seconds.
[   27.650869] cloud-init[1423]: ci-info: +++++++++++++++++++++++++Net device info+++++++++++++++++++++++++
[   27.651346] cloud-init[1423]: ci-info: +--------+------+-----------+---------------+-------------------+
[   27.651723] cloud-init[1423]: ci-info: | Device |  Up  |  Address  |      Mask     |     Hw-Address    |
[   27.654344] cloud-init[1423]: ci-info: +--------+------+-----------+---------------+-------------------+
[   27.655433] cloud-init[1423]: ci-info: |  lo:   | True | 127.0.0.1 |   255.0.0.0   |         .         |
[   27.666123] cloud-init[1423]: ci-info: | eth1:  | True | 40.0.0.13 | 255.255.255.0 | fa:16:3e:4c:37:aa |
[   27.668223] cloud-init[1423]: ci-info: | eth0:  | True | 50.0.0.22 | 255.255.255.0 | fa:16:3e:ae:7a:d3 |
[   27.670390] cloud-init[1423]: ci-info: +--------+------+-----------+---------------+-------------------+
[   27.678827] cloud-init[1423]: ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
[   27.680657] cloud-init[1423]: ci-info: +-------+-------------+----------+---------------+-----------+-------+
[   27.715891] cloud-init[1423]: ci-info: | Route | Destination | Gateway  |    Genmask    | Interface | Flags |
[   27.719301] cloud-init[1423]: ci-info: +-------+-------------+----------+---------------+-----------+-------+
[   27.719964] cloud-init[1423]: ci-info: |   0   |   0.0.0.0   | 50.0.0.1 |    0.0.0.0    |    eth0   |   UG  |
[   27.722574] cloud-init[1423]: ci-info: |   1   |   0.0.0.0   | 40.0.0.1 |    0.0.0.0    |    eth1   |   UG  |
[   27.723285] cloud-init[1423]: ci-info: |   2   |   40.0.0.0  | 0.0.0.0  | 255.255.255 ...
(more)
edit flag offensive delete link more

Comments

How does neutron / nova know how many interfaces to set up if it's only specified within the image itself? I'm referring to DHCP entries, bridges, the VM bridge interfaces in the XML definition, etc...

joshuamiller01 gravatar imagejoshuamiller01 ( 2015-08-18 17:31:21 -0500 )edit

When you launch instance via dashboard attach as many private network as many you need ( say 2)

private_net1=> eth0
private_net2=> eth1
dbaxps gravatar imagedbaxps ( 2015-08-19 06:40:14 -0500 )edit

Via CLI:-

nova boot ....  --nic net-id=private_net1-id  --nic net-id=private_net2-id ...
dbaxps gravatar imagedbaxps ( 2015-08-19 06:43:31 -0500 )edit

Right - this is how I spin up my systems. The problem is that the NIC order can change after initial provisioning, which breaks a bunch of things (as described in the OP).

joshuamiller01 gravatar imagejoshuamiller01 ( 2015-08-19 11:37:02 -0500 )edit

Qqow2 image before uploading to glance image should have:-

/etc/network/interfaces.d/eth0.cfg
/etc/network/interfaces.d/eth1.cfg

configured. Did you do that ?

dbaxps gravatar imagedbaxps ( 2015-08-19 11:52:40 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

Stats

Asked: 2015-08-18 11:58:33 -0500

Seen: 2,335 times

Last updated: Aug 19 '15