Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Live migration doesn't work after upgrade controllers from Newton to Ocata

For updates I deployed new controllers on VMs and install all packets from Ocata repos. I used new hostnames for new controllers, but I didn't changed IP-addresses. Afther all db sync, I checked all services, but live migration doesn't work. Nova list works fine, glance, cinder, neutron too. I can launch instances, deleted their, associate IPs and more, but only live and normal migrate doesn't work.

openstack server migrate --live compute01 instance01 --debug:

clean_up MigrateServer: Unable to establish connection to http://controller:8774/v2.1/20d6b9dc5e264779a40dfb334197ae67/servers/f6b517e0-b3cf-4a77-8e17-3ec315476830/action: ('Connection aborted.', BadStatusLine("''",))
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 1084, in take_action
    disk_over_commit=parsed_args.disk_overcommit,
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 459, in live_migrate
    disk_over_commit)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1698, in live_migrate
    'disk_over_commit': disk_over_commit})
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1908, in _action
    info=info, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1919, in _action_return_resp_and_body
    return self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 690, in _send_request
    raise exceptions.ConnectFailure(msg)
ConnectFailure: Unable to establish connection to http://controller:8774/v2.1/20d6b9dc5e264779a40dfb334197ae67/servers/f6b517e0-b3cf-4a77-8e17-3ec315476830/action: ('Connection aborted.', BadStatusLine("''",))

END return value: 1

In log nova-api I didn't see any erros. In nova.conf I wrote 'upgrade_levels.compute=newton'. I don't updated compute nodes because I can't live migrate instances.

nova-status upgrade check:

+-------------------------------------------------------------------+
| Upgrade Check Results                                             |
+-------------------------------------------------------------------+
| Check: Cells v2                                                   |
| Result: Success                                                   |
| Details: None                                                     |
+-------------------------------------------------------------------+
| Check: Placement API                                              |
| Result: Success                                                   |
| Details: None                                                     |
+-------------------------------------------------------------------+
| Check: Resource Providers                                         |
| Result: Warning                                                   |
| Details: There are no compute resource providers in the Placement |
|   service but there are 29 compute nodes in the deployment.       |
|   This means no compute nodes are reporting into the              |
|   Placement service and need to be upgraded and/or fixed.         |
|   See                                                             |
|   http://docs.openstack.org/developer/nova/placement.html         |
|   for more details.                                               |
+-------------------------------------------------------------------+

I didn't add the placement section in nova.conf to the compute nodes, because I need to restart openstack-nova-compute after and I think this process may not start because it not updated yet.

This is production OpenStack. Please, help!

When I previously tested updates on the OpenStack testing platform, everything was fine. But there I updated one controller without deploying a new server with new packages. I cannot update old controllers, because I need to switch from RHEL to CentOS. Our company migrate all our services to CentOS because we didn't renew subscription.

For production platform of OpenStack I deployed 3 controllers with HAProxy:

listen nova_compute_api_cluster
  bind 10.10.0.3:8774
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server osx001 10.10.0.11:8774 check inter 2000 rise 2 fall 5
  server osx002 10.10.0.12:8774 check inter 2000 rise 2 fall 5
  server osx003 10.10.0.13:8774 check inter 2000 rise 2 fall 5

Iptables disabled in this servers.