Migration failure, having problems with compute nodes
I'm doing my job in OpenStack Icehouse on Ubuntu 14.04 LTS with an NFS shared storage. I have one Controller node and two compute nodes that are connected by nova network. I want to migrate an instance that is in compute node named 'compute1' to compute node 'compute2'. I wanted to show the instances that are in 'compute1' but got messages like below.
controller@controller:~$ nova-manage vm list
instance node type state launched image kernel ramdisk project user
zone index
Command failed, please check log for more info
2016-05-20 22:18:41.514 10461 CRITICAL nova [req-d1df37b6-e57e-474e-858c-70e1df5b19da None None]
OperationalError: (OperationalError) unable to open database file None None
Meanwhile, the instances on 'compute1' are running without problems.
controller@controller:~$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| e1c0a2fc-28aa-4ba9-bc8c-3f0982499cb4 | test1 | ACTIVE | - | Running | vm-net=10.0.0.3, 172.26.17.166 |
| 2fa3958e-4bc1-4467-a16a-b576330fa27d | test2 | ACTIVE | - | Running | vm-net=10.0.0.4, 172.26.17.167 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
After executing migration instruction i got an error message.
controller@controller:~$ nova live-migration e1c0a2fc-28aa-4ba9-bc8c-3f0982499cb4 compute2
ERROR: Compute service of compute1 is unavailable at this time. (HTTP 400) (Request-ID: req-38ceef0a-c460-43f0-bba4-9abd1ef73ec9)
So I checked nova service-list.
controller@controller:~$ nova service-list
+------------------+------------+----------+----------+-------+----------------------------+-------------------------------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+------------+----------+----------+-------+----------------------------+-------------------------------------+
| nova-cert | controller | internal | enabled | up | 2016-05-20T13:20:46.000000 | - |
| nova-consoleauth | controller | internal | enabled | up | 2016-05-20T13:20:46.000000 | - |
| nova-scheduler | controller | internal | enabled | up | 2016-05-20T13:20:44.000000 | - |
| nova-conductor | controller | internal | enabled | up | 2016-05-20T13:20:42.000000 | - |
| nova-compute | compute1 | nova | enabled | down | 2016-05-17T11:40:08.000000 | None |
| nova-network | compute1 | internal | enabled | up | 2016-05-20T13:20:44.000000 | - |
| nova-compute | compute2 | nova | disabled | down | 2016-05-20T12:40:49.000000 | AUTO: Connection to libvirt lost: 1 |
| nova-network | compute2 | internal | enabled | down | 2016-05-20T12:40:42.000000 | - |
+------------------+------------+----------+----------+-------+----------------------------+-------------------------------------+
To solve the problem in compute nodes I restarted nova-compute, nova-network, nova-api-metadata but it didn't worked. On compute node2 I found those message on libvirtd.log
2016-05-20 12:31:37.744+0000: 4368: info : libvirt version: 1.2.2
2016-05-20 12:31:37.744+0000: 4368: error : netcfStateCleanup:109 : internal error: Attempt to close netcf state drive with open connections
What should I do? This is my nova.conf file on compute node.
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
auth_strategy = keystone
rpc_backend = rabbit
rabbit_host = 192.168.0.40
rabbit_password = rabbitpass
my_ip = 192.168.0.5
vnc_enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 192.168.0.5
novncproxy_base_url = http://192.168.0.40:6080/vnc_auto.html
glance_host = 192.168.0.40
network_api_class = nova.network.api.API
security_group_api = nova
firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
network_manager = nova.network.manager.FlatDHCPManager
network_size = 254
allow_same_net_traffic = False
multi_host = True
send_arp_for_ha = True
share_dhcp_address = True
force_dhcp_release = True
flat_network_bridge = br100
flat_interface = eth1
public_interface = eth1
[database]
connection = mysql://nova:novadbpass@192.168.0.40/nova
[keystone_authtoken ...
Whats your
disk_cachemodes
configuration in/etc/nova/nova.conf
of compute nodes ?I've uploaded one of my nova.conf file on compute node above.