Ask Your Question
0

SSH Keys no longer being injected on instances

asked 2016-11-11 10:43:45 -0600

mvazquezc gravatar image

Hi all, my ssh-keys are no longer being injected in new instances. I've checked openstack-nova-api, openstack-neutron-metadata and both are running. I can't see any errors on instance log.

Old instances already running can be accessed with my already existing keypair, and I can reach the metadata api:

[centos@svn-backup ~]$ curl http://169.254.169.254
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04

Log on new instances shows:

[[32m  OK  [0m] Started Dynamic System Tuning Daemon.
[    9.650087] cloud-init[718]: Cloud-init v. 0.7.5 running 'init' at Fri, 11 Nov 2016 16:05:32 +0000. Up 9.55 seconds.
[    9.719209] cloud-init[718]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    9.720871] cloud-init[718]: ci-info: +--------+------+----------------+---------------+-------------------+
[    9.722402] cloud-init[718]: ci-info: | Device |  Up  |    Address     |      Mask     |     Hw-Address    |
[    9.723930] cloud-init[718]: ci-info: +--------+------+----------------+---------------+-------------------+
[    9.725458] cloud-init[718]: ci-info: |  lo:   | True |   127.0.0.1    |   255.0.0.0   |         .         |
[    9.726903] cloud-init[718]: ci-info: | eth0:  | True | 192.168.122.97 | 255.255.255.0 | fa:16:3e:22:36:ea |
[    9.728342] cloud-init[718]: ci-info: +--------+------+----------------+---------------+-------------------+
[    9.741371] cloud-init[718]: ci-info: +++++++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++++++++
[    9.742913] cloud-init[718]: ci-info: +-------+-----------------+---------------+-----------------+-----------+-------+
[    9.744446] cloud-init[718]: ci-info: | Route |   Destination   |    Gateway    |     Genmask     | Interface | Flags |
[    9.745986] cloud-init[718]: ci-info: +-------+-----------------+---------------+-----------------+-----------+-------+
[    9.747506] cloud-init[718]: ci-info: |   0   |     0.0.0.0     | 192.168.122.1 |     0.0.0.0     |    eth0   |   UG  |
[    9.749046] cloud-init[718]: ci-info: |   1   | 169.254.169.254 | 192.168.122.1 | 255.255.255.255 |    eth0   |  UGH  |
[    9.750555] cloud-init[718]: ci-info: |   2   |  192.168.122.0  |    0.0.0.0    |  255.255.255.0  |    eth0   |   U   |
[    9.754433] cloud-init[718]: ci-info: +-------+-----------------+---------------+-----------------+-----------+-------+
[[32m  OK  [0m] Started Postfix Mail Transport Agent.

CentOS Linux 7 (Core)
Kernel 3.10.0-327.28.3.el7.x86_64 on an x86_64

host-192-168-122-97 login: [  220.164868] cloud-init[718]: 2016-11-11 16:09:03,502 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
[  271.217844] cloud-init[718]: 2016-11-11 16:09:54,556 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
[  289.223614] cloud-init[718]: 2016-11-11 16:10:12,562 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
[  290.228922] cloud-init[718]: 2016-11-11 16:10:13,564 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
[  340.233679] cloud-init[718]: 2016-11-11 16:11:03,572 - url_helper.py[WARNING]: Calling 'http://192.168.122.2//latest/meta-data/instance-id' failed [50/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
[  391.263485] cloud-init[718]: 2016-11-11 16:11:54,602 - url_helper.py[WARNING]: Calling 'http://192.168.122.2//latest/meta-data/instance-id' failed [101/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
[  409.284255] cloud-init[718]: 2016-11-11 16:12:12,623 - url_helper.py[WARNING]: Calling 'http://192.168.122.2//latest/meta-data/instance-id ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2016-11-16 05:44:09 -0600

mvazquezc gravatar image

updated 2016-11-16 05:46:19 -0600

I finally have figured out which was the problem.

The problem was related with security groups, by default there is a security group named "default" that allows egress connectivity to other systems running the same security group.

Users in one of our tenants have created a security group that only allows ingress ssh connections, so when the instance boots up and try to connect to the metadata proxy (169.254.169.254) it can't because of the security group rules.

In order to solve this problem is as easy as allow egress HTTP connectivity to 169.254.169.254 in any of the SG assigned to your instances or assign the "default" SG to your instances.

I recommend you to check out this link in order to understand how metadata proxy works: http://techbackground.blogspot.com.es/2013/06/metadata-via-quantum-router.html (http://techbackground.blogspot.com.es...)

Thank you all!

edit flag offensive delete link more
0

answered 2016-11-11 22:10:08 -0600

diltram gravatar image

You can try to curl exactly the same endpoint which was hit by cloud-init and failed. Plus may you provide logs from neutron-metadata and nova api?

edit flag offensive delete link more

Comments

The endpoint is working from other instances that were launched before the issue arised.

I can't see any errors on neutron-metada or nova-api log. I also tried to re-create neutron port being used as gateway to the internal network... it didn't work

New instances can't ping internal gateway IP.

mvazquezc gravatar imagemvazquezc ( 2016-11-14 05:03:41 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-11-11 10:43:45 -0600

Seen: 1,109 times

Last updated: Nov 16 '16