Ask Your Question
0

can't ssh to new created instances any more

asked 2015-01-29 20:39:16 -0500

Glen.GZ gravatar image

i had a weird case here that i can't ssh to newly created instances anymore (juno, ubuntu instance). same keypair same image, also can't access from console or internet via newly generated keypair. always respones "server refuses your key" or doesnt support the auth method. but the old instance created before with the same keypair is able to sshed.

i also noticed 2 problems here,

  1. instance reboot or bootup takes longer than before and have this error util.py[WARNING]: 'http://169.254.169.254/20090404/metadata/instanceid' failed [50/120s]: url error [timed out].

    1. the key fingerprints in the log of newly created instance is quite different from the fingerprint that loaded at first. see below the newly created instance called "test6" and loading keypair whose fingerprint is 53:4d:9e:ce:2e:2a:28:d8:9b:95:67:db:50:c3:3e:cc

but in the log of test6, fingerprints are different like below * Starting OpenSSH server[74G[ OK ] * Starting regular background program processing daemon[74G[ OK ] * Starting deferred execution scheduler[74G[ OK ] * Stopping save kernel messages[74G[ OK ] * Stopping CPU interrupts balancing daemon[74G[ OK ] * Starting automatic crash report generation[74G[ OK ] open-vm-tools: not starting as this is not a VMware VM landscape-client is not configured, please run landscape-config. * Restoring resolver state... [80G [74G[ OK ] * Stopping System V runlevel compatibility[74G[ OK ]

Ubuntu 14.04.1 LTS ubuntu ttyS0

ubuntu login: Cloud-init v. 0.7.5 running 'modules:config' at Thu, 29 Jan 2015 08:27:42 +0000. Up 419.52 seconds. Generating locales... en_US.UTF-8... up-to-date Generation complete. Cloud-init v. 0.7.5 running 'modules:final' at Thu, 29 Jan 2015 08:27:45 +0000. Up 422.42 seconds. ci-info: no authorized ssh keys fingerprints found for user ubuntu. ci-info: no authorized ssh keys fingerprints found for user ubuntu. ec2: ec2: ############################################################# ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- ec2: 1024 dd:59:7d:7c:1a:36:24:46:c6:e8:c1:74:38:b3:ae:93 root@ubuntu (DSA) ec2: 256 b4:d0:dd:19:d8:19:a5:e3:51:4f:96:2a:a2:b6:2c:d6 root@ubuntu (ECDSA) ec2: 2048 f6:10:fa:fe:88:1a:a9:1c:f4:25:32:fe:1a:ed:b6:1d root@ubuntu (RSA) ec2: -----END SSH HOST KEY FINGERPRINTS----- ec2: ############################################################# -----BEGIN SSH HOST KEY KEYS----- ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNRxzpVW1DcqK/n6HM0Rb2Se+/e7HBQ7YBCqSXp66MnAF3n1ekJQ1kTslIbJv1gS/dG8k+047KYiy0w2fLkFhEs= root@ubuntu ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEFGiKKyryaMNXXhrjUh1ocpdHw2TRmctnIT8eiizmUFP3oVR5YAp3Rn9K/xEdolLFNtTl0Z1FJ6ASDvr/h79KnKNNrWp/J/cva93SgrPM8RK1PskirYHb9+tiwCW6qFuLyZy0dKYi7g/flpT3PA8ZtbXXIL5mXU82jddRElHHhyFqFM04UVIoBtyFM+whkPcYvbG4mb4trU5bIRmfoHpq0QE7BOk0kOm9vtazcUJLzlKZQY2bxrzze589e4UUsFfaiOJFn6eqBRHGwhtEnsbEOTybMwBW5AXcHszfWuEs2mZw/5vV9xqtEmukxHiteT42ixPTjmzIvbo3eenWPGGP root@ubuntu -----END SSH HOST KEY KEYS----- Cloud-init v. 0.7.5 finished at Thu, 29 Jan 2015 08:27:45 +0000. Datasource DataSourceNone. Up 422.60 seconds 2015-01-29 08:27:45,659 - cc_final_message.py[WARNING]: Used fallback datasource

anyone has idea on it?

thanks

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
1

answered 2015-01-30 06:14:39 -0500

I don't have an exact answer as to what the problem is, but I can provide some ideas on how to troubleshoot it.

First, launch an instance with the Cirros image because with that image you can login via the console even if your instances cloud-init does not get any metadata/key injection or network connectivity from outside.

Once you have launch above instance, log into it and do:

ifconfig -a

This will determine if you are getting a DHCP private address assigned to your instance. If you are, that at least confirms your OpenvSwitch plumbing and DHCP agent on the neutron node are working and most likely indicates a problem with the L3-agent.

In your reference above you mention that the console is not accessible either.

In this case, I would check:

nova service-list and make sure everything is happy there neutron agent-list and make sure everything is happy there

If you are seeing issues with nova service output on compute node, restart openvswitch, ovs-cleanup and then openstack-service restart.

If neutron also looks unhappy, restart those services as well but also ensure that before you bring them back up, all the neutron services have been killed off. I have seen dnqmasq and metadata have a tendency to not get stopped. You will also want to do ip netns on neutron node and ensure that all qdhcp and qrouter namespaces have been removed.

Then start up all your neutron services.

edit flag offensive delete link more
0

answered 2016-09-08 13:45:28 -0500

In my case (new install using Ubuntu Juju) the problem was that neutron was misconfigured with the wrong ip address for the metadata server.

This is good: http://techbackground.blogspot.ie/2013/06/metadata-via-dhcp-namespace.html (http://techbackground.blogspot.ie/201...)

That pointed me at: root@neutron:/etc/neutron# cat metadata_agent.ini

    ...
  nova_metadata_ip = 192.168.100.151
  nova_metadata_port = 8775
    ...

I fixed the nova_metadata_ip adn restarted the neutron metadata service and that solved the issue.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-01-29 20:39:16 -0500

Seen: 7,550 times

Last updated: Sep 08 '16