Ask Your Question

Nate_McKay's profile - activity

2016-05-12 11:05:29 -0500 received badge  Student (source)
2015-08-07 16:38:30 -0500 received badge  Famous Question (source)
2015-08-03 16:16:03 -0500 answered a question ssh fails with a public_key issue when create by heat?

I have this same problem. Any instances created via a Heat template can not be logged into, but a manually created instance using the same image and key works as expected. Any ideas? I see that the instance in question is making requests to the metadata service:

2015-08-03 14:09:26.875 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.627 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2012-08-10/meta_data.json HTTP/1.1" 200 725 0.750399
2015-08-03 14:09:27.637 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.643 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2013-10-17 HTTP/1.1" 200 167 0.005155
2015-08-03 14:09:27.653 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.901 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2013-10-17/vendor_data.json HTTP/1.1" 200 124 0.247671
2015-08-03 14:09:27.912 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.919 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2013-10-17/vendor_data.json HTTP/1.1" 200 124 0.006323
2015-08-03 14:09:27.928 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.936 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2013-10-17/user_data HTTP/1.1" 200 7283 0.006698
2015-08-03 14:09:27.945 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:27.953 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:27] "GET /openstack/2013-10-17/user_data HTTP/1.1" 200 7283 0.005900
2015-08-03 14:09:27.961 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:28.279 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:28] "GET /openstack/2013-10-17/meta_data.json HTTP/1.1" 200 1429 0.316215
2015-08-03 14:09:28.290 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:28.299 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:28] "GET /openstack/2013-10-17/meta_data.json HTTP/1.1" 200 1429 0.007537
2015-08-03 14:09:28.310 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:28.319 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:28] "GET /latest/meta-data HTTP/1.1" 200 362 0.006932
2015-08-03 14:09:28.331 3795 INFO eventlet.wsgi.server [-] (3795) accepted ''
2015-08-03 14:09:28.339 3795 INFO eventlet.wsgi.server [-] 192.168.2.34,<local> - - [03/Aug/2015 14:09:28] "GET /latest/meta-data/block-device-mapping/ HTTP/1.1" 200 124 0.006972
2015-08-03 14:09:28.350 3795 INFO eventlet.wsgi.server ...
(more)
2015-07-08 00:20:38 -0500 answered a question Load-balancer not passing traffic correctly when accessed via floating-IP

Sorted this a while back and neglected to post the cause/solution.

What I found was that the qlbaas namespace under which the haproxy load-balancer was running did not have a default route set. Consequently the haproxy process did not know how to route packets back to the client, and thus the client never received any kind of response from the VIP.

Manually setting the default route inside of the qlbaas namespace resolved the problem, but did not persist accross reboots until eventually it kind of just sorted itself out and stopped happening. I had been able to repro this once on a stack cloned from the one where I was experiencing the problem initially, but not reliably or predictably.

For what it's worth...

2015-07-08 00:16:36 -0500 received badge  Famous Question (source)
2015-05-19 03:02:55 -0500 received badge  Notable Question (source)
2015-05-19 03:02:55 -0500 received badge  Popular Question (source)
2015-05-12 10:36:45 -0500 received badge  Notable Question (source)
2015-05-12 10:36:45 -0500 received badge  Popular Question (source)
2015-05-02 00:42:39 -0500 commented question Load-balancer not passing traffic correctly when accessed via floating-IP

I can ping the VIP fixed address from an instance on the same subnet.

I cannot ping the floating address associated with the VIP from a node on the same subnet.

The network node is responding to ARP requests for the floating address however.

2015-05-02 00:39:01 -0500 received badge  Enthusiast
2015-04-30 19:10:19 -0500 asked a question Load-balancer not passing traffic correctly when accessed via floating-IP

Hi all,

I have a multi-node Juno lab on CentOS 7.1 with one controller node, one network node, and two compute nodes.

I am using haproxy for lbaas, and am finding that while I can create a working VIP/pool et al that passes traffic on the internal VIP address, the floating-ip I have associated with it fails to respond or otherwise pass traffic. The floating-ips that I have associated with the individual instances themselves pass traffic no problem however.

I see that the VIP floating-ip is responding to ARP requests, but will not ping or return a SYN-ACK for configured ports despite being allowed by the security group. I am not seeing any log errors that look like they would point to this specifically. I do see some errors in the ovs-vswitchd.log relating to non-existent devices, but I wouldn't think them related (need to sort that separately).

Any ideas where to start looking in order to track this down? Any help would be much appreciated!

Curl against the VIP from a peer instance:

[fedora@web-84ffe0f7-9169-4c29-8be5-d2f4ba443019 ~]$ curl -v http://192.168.1.200/server.txt
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.200...
* Connected to 192.168.1.200 (192.168.1.200) port 80 (#0)
> GET /server.txt HTTP/1.1
> User-Agent: curl/7.37.0
> Host: 192.168.1.200
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx/1.6.3 is not blacklisted
< Server: nginx/1.6.3
< Date: Thu, 30 Apr 2015 23:53:14 GMT
< Content-Type: text/plain
< Content-Length: 41
< Last-Modified: Thu, 30 Apr 2015 18:28:58 GMT
< ETag: "5542746a-29"
< Accept-Ranges: bytes
<
web-6744ff64-8b4c-4c60-9823-ee891d37adc3
* Connection #0 to host 192.168.1.200 left intact
[fedora@web-84ffe0f7-9169-4c29-8be5-d2f4ba443019 ~]$

Curl against the pool member instance's floating-IP:

[nmckay@bistromath ~]$ curl -v http://10.12.21.204/server.txt
* Hostname was NOT found in DNS cache
*   Trying 10.12.21.204...
* Connected to 10.12.21.204 (10.12.21.204) port 80 (#0)
> GET /server.txt HTTP/1.1
> User-Agent: curl/7.36.0
> Host: 10.12.21.204
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx/1.6.3 is not blacklisted
< Server: nginx/1.6.3
< Date: Thu, 30 Apr 2015 23:52:01 GMT
< Content-Type: text/plain
< Content-Length: 41
< Last-Modified: Thu, 30 Apr 2015 18:28:58 GMT
< Connection: keep-alive
< ETag: "5542746a-29"
< Accept-Ranges: bytes
<
web-6744ff64-8b4c-4c60-9823-ee891d37adc3
* Connection #0 to host 10.12.21.204 left intact
[nmckay@bistromath ~]$

Curl against the VIP's floating IP:

[nmckay@bistromath ~]$ curl -v http://10.12.21.205/server.txt
* Hostname was NOT found in DNS cache
*   Trying 10.12.21.205...
* connect to 10.12.21.205 port 80 failed: Operation timed out
* Failed to connect to 10.12.21.205 port 80: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to 10.12.21.205 port 80: Operation timed out
[nmckay@bistromath ~]$

Tcpdump output showing ARP response:

[nmckay@bistromath ~]$ sudo tcpdump -i em2 host 10.12.21.205 ...
(more)
2015-04-27 11:40:52 -0500 commented answer ovs-vswitchd crashing continuously on reboot of network node

Yeah that's sorted it. Too easy, thanks!

2015-04-26 02:44:18 -0500 received badge  Scholar (source)
2015-04-25 22:11:46 -0500 asked a question ovs-vswitchd crashing continuously on reboot of network node

Hi,

I recently finished the first phase of a 4 node OpenStack (Juno) lab on CentOS 7 using the guide found here:

http://docs.openstack.org/juno/install-guide/install/yum/content/ch_preface.html (http://docs.openstack.org/juno/instal...)

On the whole this went pretty smoothly and I was able to launch instances and ping the l3 agent router IPs etc. I'm running into an external issue that necessitates that I reboot my network node and am finding that after it comes back up, my instances are no longer able to leverage any network node services any longer (dhcp and l3 are now inaccessible).

I noticed that ovs-vswitchd seems to be crashing and respawning constantly after the reboot with messages to this effect:

2015-04-22T03:07:31.535Z|00023|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.1.3
2015-04-24T21:47:21.457Z|00024|memory|INFO|43132 kB peak resident set size after 10.0 seconds
2015-04-24T21:47:21.457Z|00025|memory|INFO|dispatchers:1 flow_dumpers:1 handlers:1 ports:13 revalidator keys:1 revalidators:1 rules:11
2015-04-24T21:47:26.187Z|00026|bridge|INFO|bridge br-int: added interface tap9dc19b59-70 on port 1
2015-04-24T21:47:26.855Z|00027|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tap9dc19b59-70 device failed: No such device
2015-04-24T21:47:26.874Z|00028|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap9dc19b59-70 device failed: No such device
2015-04-24T21:47:26.875Z|00029|netdev_linux|WARN|tap9dc19b59-70: removing policing failed: No such device
2015-04-24T21:47:27.013Z|00030|bridge|INFO|bridge br-int: added interface int-br-ex on port 2
2015-04-24T21:47:27.160Z|00031|bridge|INFO|bridge br-ex: added interface phy-br-ex on port 2
2015-04-24T21:47:27.652Z|00002|daemon(monitor)|ERR|1 crashes: pid 666 died, killed (Segmentation fault), restarting
2015-04-24T21:47:27.655Z|00003|memory|INFO|9520 kB peak resident set size after 16.2 seconds
2015-04-24T21:47:27.655Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2015-04-24T21:47:27.655Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2015-04-24T21:47:27.706Z|00006|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tap9dc19b59-70 device failed: No such device
2015-04-24T21:47:27.709Z|00007|bridge|INFO|bridge br-ex: added interface br-ex on port 65534
2015-04-24T21:47:27.709Z|00008|bridge|INFO|bridge br-ex: added interface eno50338560 on port 1
2015-04-24T21:47:27.709Z|00009|bridge|INFO|bridge br-ex: added interface phy-br-ex on port 2
2015-04-24T21:47:27.709Z|00010|bridge|INFO|bridge br-int: added interface br-int on port 65534
2015-04-24T21:47:27.709Z|00011|bridge|INFO|bridge br-int: added interface tap9dc19b59-70 on port 1
2015-04-24T21:47:27.712Z|00012|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap9dc19b59-70 device failed: No such device
2015-04-24T21:47:27.712Z|00013|bridge|INFO|bridge br-int: added interface int-br-ex on port 2
2015-04-24T21:47:27.712Z|00014|bridge|INFO|bridge br-ex: using datapath ID 0000005056996d07
2015-04-24T21:47:27.712Z|00015|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt"
2015-04-24T21:47:27.712Z|00016|bridge|INFO|bridge br-int: using datapath ID 00003a25509d8841
2015-04-24T21:47:27.712Z|00017|netdev_linux ...
(more)