Ask Your Question
0

Cannot connect to a running instance, even after the euca-authorize stuff

asked 2011-04-22 08:59:12 -0500

brian-amedro gravatar image

Hi !

I experience some difficulties do connect to the instances I've created with nova. I can neither ping nor ssh the instance, even if I perform the euca-authorize stuff.

I use FlatDHCPManager.

Any help would be greatly appreciated !

Here are some details about my config and the problem :

==== /etc/nova/nova.conf ==== --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --verbose --libvirt_type=kvm --network_manager=nova.network.manager.FlatDHCPManager --flat_network_dhcp_start=138.96.126.200 --public_interface=eth0 --flat_injected=False

==== /etc/network/interfaces ====

The loopback network interface

auto lo iface lo inet loopback

Networking for OpenStack Compute

auto br100 iface br100 inet dhcp bridge_ports eth0 bridge_stp off bridge_maxwait 0 bridge_fd 0

==== /sbin/ifconfig ==== br100 Link encap:Ethernet HWaddr 00:22:19:9a:6c:20
inet addr:138.96.126.4 Bcast:138.96.126.255 Mask:255.255.255.0 inet6 addr: fe80::222:19ff:fe9a:6c20/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:73420 errors:0 dropped:0 overruns:0 frame:0 TX packets:23403 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:57123865 (57.1 MB) TX bytes:3052602 (3.0 MB)

eth0 Link encap:Ethernet HWaddr 00:22:19:9a:6c:20
inet6 addr: fe80::222:19ff:fe9a:6c20/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:115824 errors:0 dropped:0 overruns:0 frame:0 TX packets:24616 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:62186577 (62.1 MB) TX bytes:3385628 (3.3 MB) Interrupt:21 Memory:ea000000-ea012800

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:11291121 errors:0 dropped:0 overruns:0 frame:0 TX packets:11291121 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:783143897 (783.1 MB) TX bytes:783143897 (783.1 MB)

virbr0 Link encap:Ethernet HWaddr 26:dd:54:d5:85:df
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:1b:62:26
inet6 addr: fe80::fc16:3eff:fe1b:6226/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:822 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:2948 (2.9 KB) TX bytes:110929 (110.9 KB)

==== # brctl show bridge name bridge id STP enabled interfaces br100 8000.0022199a6c20 no eth0 vnet0 ... (more)

edit retag flag offensive close merge delete

10 answers

Sort by ยป oldest newest most voted
0

answered 2011-10-20 10:31:25 -0500

btw. is it normal that the vnet0 gets an "fe:" mac-address while the vm gets a "02:" mac?

edit flag offensive delete link more
0

answered 2011-04-22 10:58:40 -0500

brian-amedro gravatar image

I already saw https://answers.launchpad.net/nova/+question/145820 (https://answers.launchpad.net/nova/+q...) which looks like to be a similar problem, but I am already with FlatDHCPManager. I also tried to connect to the VM through a serial console, but it fails:

==== $ virsh console instance-00000019 ==== Connected to domain instance-00000019 Escape character is ^] error: internal error character device (null) is not using a PTY

==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ==== <domain type="kvm"> <name>instance-00000019</name> <memory>524288</memory> <os> <type>hvm</type> <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel> <cmdline>root=/dev/vda console=ttyS0</cmdline> </os> <features> <acpi/> </features> <vcpu>1</vcpu> <devices> <disk type="file"> <driver type="qcow2"/> <source file="/var/lib/nova/instances/instance-00000019/disk"/> <target dev="vda" bus="virtio"/> </disk>

    <interface type='bridge'>
        <source bridge='br100'/>
        <mac address='02:16:3e:1b:62:26'/>
        <!--   <model type='virtio'/>  CANT RUN virtio network right now -->
        <filterref filter="nova-instance-instance-00000019-02163e1b6226">
            <parameter name="IP" value="138.96.126.201" />
            <parameter name="DHCPSERVER" value="138.96.126.1" />
        </filterref>
    </interface>
    <!-- The order is significant here.  File must be defined first -->
    <serial type="file">
        <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
        <target port='1'/>
    </serial>

    <console type='pty' tty='/dev/pts/2'>
        <source path='/dev/pts/2'/>
        <target port='0'/>
    </console>

    <serial type='pty'>
        <source path='/dev/pts/2'/>
        <target port='0'/>
    </serial>

    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>
</devices>

</domain>

edit flag offensive delete link more
0

answered 2011-04-22 14:14:06 -0500

vishvananda gravatar image

i would guess problems with dnsmasq. It looks like you are trying to put instances on the same network as your host machine which may be difficult to get working if you have an external router or gateway responding to dnsmasq queries. I would check the process list for dnsmasq and make sure that there are exactly two running and that the command line for them is the same and it looks reasonable.

Vish

On Apr 22, 2011, at 3:58 AM, Brian Amedro wrote:

Question #153827 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Brian Amedro gave more information on the question: I already saw https://answers.launchpad.net/nova/+q... which looks like to be a similar problem, but I am already with FlatDHCPManager. I also tried to connect to the VM through a serial console, but it fails:

==== $ virsh console instance-00000019 ==== Connected to domain instance-00000019 Escape character is ^] error: internal error character device (null) is not using a PTY

==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ==== <domain type="kvm"> <name>instance-00000019</name> <memory>524288</memory> <os> <type>hvm</type> <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel> <cmdline>root=/dev/vda console=ttyS0</cmdline> </os> <features> <acpi/> </features> <vcpu>1</vcpu> <devices> <disk type="file"> <driver type="qcow2"/> <source file="/var/lib/nova/instances/instance-00000019/disk"/> <target dev="vda" bus="virtio"/> </disk>

   <interface type='bridge'>
       <source bridge='br100'/>
       <mac address='02:16:3e:1b:62:26'/>
       <!--   <model type='virtio'/>  CANT RUN virtio network right now -->
       <filterref filter="nova-instance-instance-00000019-02163e1b6226">
           <parameter name="IP" value="138.96.126.201" />
           <parameter name="DHCPSERVER" value="138.96.126.1" />
       </filterref>
   </interface>
   <!-- The order is significant here.  File must be defined first -->
   <serial type="file">
       <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
       <target port='1'/>
   </serial>

   <console type='pty' tty='/dev/pts/2'>
       <source path='/dev/pts/2'/>
       <target port='0'/>
   </console>

   <serial type='pty'>
       <source path='/dev/pts/2'/>
       <target port='0'/>
   </serial>

   <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>

</devices> </domain>

You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-04-22 14:52:37 -0500

brian-amedro gravatar image

Hi vish, thanks for the reply !

You're right, I am trying to put instances on the same network as the host machine. This network already has a DHCP server on 138.96.126.1, which give IP for the range 138.96.126.1-199. Also, I cannot modify the config of this server.

Is there any way so the host machine respond to the VM requests instead of the external DHCP server ? Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1"/>

Also, and to be complete, even if I put --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started instances was assigned with an IP in the wrong range. Thus, I modified by hand the database, in order to exclude some range by setting the 'reserved' column in the fixed_ips table.

As you asked, here is the list of dnsmasq processes.

nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

$ cat /var/lib/nova/networks/nova-br100.conf 02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200

edit flag offensive delete link more
0

answered 2011-04-22 15:15:38 -0500

vishvananda gravatar image

Yes that is your problem. It attempts to give the first address in the subnet to your machine and it is conflicting with your external dns server. You could try using a smaller network, such as: 128.96.126.224/27 although your .1 server may still give out addresses and mess things up. Worth a shot. You'll have to delete networks and fixed_ip tables and recreate the smaller range with nova-network network create you should also change: --fixed_range=128.96.224/27 --flat_network_dhcp_start=138.96.126.225

killall dnsmasq iptables -F iptabbles -t nat -F restart nova-network

try again?

Vish

On Apr 22, 2011, at 7:52 AM, Brian Amedro wrote:

Question #153827 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Status: Answered => Open

Brian Amedro is still having a problem: Hi vish, thanks for the reply !

You're right, I am trying to put instances on the same network as the host machine. This network already has a DHCP server on 138.96.126.1, which give IP for the range 138.96.126.1-199. Also, I cannot modify the config of this server.

Is there any way so the host machine respond to the VM requests instead of the external DHCP server ? Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1"/>

Also, and to be complete, even if I put --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started instances was assigned with an IP in the wrong range. Thus, I modified by hand the database, in order to exclude some range by setting the 'reserved' column in the fixed_ips table.

As you asked, here is the list of dnsmasq processes.

nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --pid- file=/var/run/libvirt/network/default.pid --conf-file= --except- interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid- file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp- hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin /nova-dhcpbridge --leasefile-ro

root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid- file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp- hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin /nova-dhcpbridge --leasefile-ro

$ cat /var/lib/nova/networks/nova-br100.conf 02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200


You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-04-22 15:39:49 -0500

brian-amedro gravatar image

I still have the same problem: 2011-04-22 17:29:42,257 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id (http://169.254.169.254/2009-04-04/met...)

2011-04-22 17:29:42,259 - DataSourceEc2.py[WARNING]: 17:29:42 [ 1/100]: url error [[Errno 101] Network is unreachable]

Just to remind, my config is now: $ cat /etc/nova/nova.conf --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --verbose --libvirt_type=kvm --network_manager=nova.network.manager.FlatDHCPManager --fixed_range=138.96.224/27 --flat_network_dhcp_start=138.96.126.225 --flat_interface=eth0 --flat_injected=False

and iptables-save gives

Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011

*nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [6:360] :POSTROUTING ACCEPT [6:360] :nova-compute-OUTPUT - [0:0] :nova-compute-POSTROUTING - [0:0] :nova-compute-PREROUTING - [0:0] :nova-compute-floating-snat - [0:0] :nova-compute-snat - [0:0] :nova-network-OUTPUT - [0:0] :nova-network-POSTROUTING - [0:0] :nova-network-PREROUTING - [0:0] :nova-network-floating-snat - [0:0] :nova-network-snat - [0:0] :nova-postrouting-bottom - [0:0] -A PREROUTING -j nova-compute-PREROUTING -A PREROUTING -j nova-network-PREROUTING -A OUTPUT -j nova-compute-OUTPUT -A OUTPUT -j nova-network-OUTPUT -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-network-POSTROUTING -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE -A POSTROUTING -j nova-postrouting-bottom -A nova-compute-snat -j nova-compute-floating-snat -A nova-network-POSTROUTING -s 138.96.224.0/27 -d 10.128.0.0/24 -j ACCEPT -A nova-network-POSTROUTING -s 138.96.224.0/27 -d 138.96.224.0/27 -j ACCEPT -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 138.96.126.4:8773 -A nova-network-snat -j nova-network-floating-snat -A nova-network-snat -s 138.96.224.0/27 -j SNAT --to-source 138.96.126.4 -A nova-postrouting-bottom -j nova-compute-snat -A nova-postrouting-bottom -j nova-network-snat COMMIT

Completed on Fri Apr 22 19:31:57 2011

Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011

*mangle :PREROUTING ACCEPT [15392:1070864] :INPUT ACCEPT [15390:1070800] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [15266:1065332] :POSTROUTING ACCEPT [15266:1065332] -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill COMMIT

Completed on Fri Apr 22 19:31:57 2011

Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011

*filter :INPUT ACCEPT [8347:581314] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [8270:577902] :nova-compute-FORWARD - [0:0] :nova-compute-INPUT - [0:0] :nova-compute-OUTPUT - [0:0] :nova-compute-inst-33 - [0:0] :nova-compute-local - [0:0] :nova-compute-sg-fallback - [0:0] :nova-filter-top - [0:0] :nova-network-FORWARD - [0:0] :nova-network-INPUT - [0:0] :nova-network-OUTPUT - [0:0] :nova-network-local - [0:0] -A INPUT -j nova-compute-INPUT -A INPUT -j nova-network-INPUT -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m ... (more)

edit flag offensive delete link more
0

answered 2011-04-22 16:48:12 -0500

brian-amedro gravatar image

I have some evolution on the error. Now it is a "timed out" error, instead of a "Network is unreachable".

==== euca-get-console-output ==== 2011-04-22 18:24:52,350 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id (http://169.254.169.254/2009-04-04/met...) 2011-04-22 18:24:52,353 - DataSourceEc2.py[WARNING]: 18:24:52 [ 1/100]: url error [timed out] 2011-04-22 18:24:55,360 - DataSourceEc2.py[WARNING]: 18:24:55 [ 2/100]: url error [timed out] 2011-04-22 18:24:58,367 - DataSourceEc2.py[WARNING]: 18:24:58 [ 3/100]: url error [timed out] 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out] 2011-04-22 18:25:02,380 - DataSourceEc2.py[WARNING]: 18:25:02 [ 5/100]: url error [[Errno 113] No route to host] 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out] Hundred times...

Also, when I ping the instance, I have a response from an other IP : $ nova list +----+-----------+--------+-----------+----------------+ | ID | Name | Status | Public IP | Private IP | +----+-----------+--------+-----------+----------------+ | 35 | Server 35 | ACTIVE | | 138.96.126.227 | +----+-----------+--------+-----------+----------------+

$ ping 138.96.126.227 PING 138.96.126.227 (138.96.126.227) 56(84) bytes of data. From 138.96.126.225 icmp_seq=1 Destination Host Unreachable From 138.96.126.225 icmp_seq=2 Destination Host Unreachable From 138.96.126.225 icmp_seq=3 Destination Host Unreachable From 138.96.126.225 icmp_seq=5 Destination Host Unreachable

Network was created with:

nova-manage network create 138.96.126.224/27 1 32

edit flag offensive delete link more
0

answered 2011-04-22 19:05:14 -0500

vishvananda gravatar image

Yes so it isn't actually getting the right ip. I'm not sure if there is a way to get a config like this to work. You may have to switch to using private addressing (like the 10.x default range) and assigning floating ips to give your vms ips on the 138 network.

Vish

On Apr 22, 2011, at 9:48 AM, Brian Amedro wrote:

Question #153827 on OpenStack Compute (nova) changed: https://answers.launchpad.net/nova/+q...

Brian Amedro gave more information on the question: I have some evolution on the error. Now it is a "timed out" error, instead of a "Network is unreachable".

==== euca-get-console-output ==== 2011-04-22 18:24:52,350 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/met... 2011-04-22 18:24:52,353 - DataSourceEc2.py[WARNING]: 18:24:52 [ 1/100]: url error [timed out] 2011-04-22 18:24:55,360 - DataSourceEc2.py[WARNING]: 18:24:55 [ 2/100]: url error [timed out] 2011-04-22 18:24:58,367 - DataSourceEc2.py[WARNING]: 18:24:58 [ 3/100]: url error [timed out] 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out] 2011-04-22 18:25:02,380 - DataSourceEc2.py[WARNING]: 18:25:02 [ 5/100]: url error [[Errno 113] No route to host] 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out] Hundred times...

Also, when I ping the instance, I have a response from an other IP : $ nova list +----+-----------+--------+-----------+----------------+ | ID | Name | Status | Public IP | Private IP | +----+-----------+--------+-----------+----------------+ | 35 | Server 35 | ACTIVE | | 138.96.126.227 | +----+-----------+--------+-----------+----------------+

$ ping 138.96.126.227 PING 138.96.126.227 (138.96.126.227) 56(84) bytes of data.

From 138.96.126.225 icmp_seq=1 Destination Host Unreachable From 138.96.126.225 icmp_seq=2 Destination Host Unreachable From 138.96.126.225 icmp_seq=3 Destination Host Unreachable From 138.96.126.225 icmp_seq=5 Destination Host Unreachable

Network was created with:

nova-manage network create 138.96.126.224/27 1 32


You received this question notification because you are a member of Nova Core, which is an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-04-23 06:58:29 -0500

brian-amedro gravatar image

I will turn my config this way. Private addressing + floating ips on the public network will do the trick.

Many thanks for your time Vish.

edit flag offensive delete link more
0

answered 2011-04-23 06:58:52 -0500

brian-amedro gravatar image

Thanks Vish Ishaya, that solved my question.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-04-22 08:59:12 -0500

Seen: 197 times

Last updated: Oct 20 '11