Revision history [back]

click to hide/show revision 1
initial version

Error status after creating VM instance

Hi. I have a controller and a compute node for test purposes. Yesterday, I successfully was able to create instances. They got IP address, etc, everything was fine. Today, I am not able to create VMs, and the status of the instance is ERROR!

root@controller:/# openstack server list
+--------------------------------------+---------------------+--------+----------+------------+
| ID                                   | Name                | Status | Networks | Image Name |
+--------------------------------------+---------------------+--------+----------+------------+
| c59b9cda-3155-4fe8-9b6d-eca702d44431 | provider-instance11 | ERROR  |          | cirros     |
+--------------------------------------+---------------------+--------+----------+------------+

This is the log from nova-scheduler on contoller node:

2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     dbapi_connection = rec.get_connection()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connection = self.__connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     connection = self.__pool._invoke_creator(self)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     raise exc
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller' ([Errno 111] ECONNREFUSED)")
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db 
2016-12-29 09:50:46.001 4260 INFO nova.scheduler.host_manager [req-d2c13055-0041-4a6e-baca-351416376d83 - - - - -] Successfully synced instances from host 'compute1'.
2016-12-29 09:50:47.776 4260 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

This is the nova-compute log from the compute node:

2016-12-29 10:33:17.512 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Total usable vcpus: 8, total allocated vcpus: 0
2016-12-29 10:33:17.512 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Final resource view: name=compute1 phys_ram=15731MB used_ram=512MB phys_disk=1817GB used_disk=0GB total_vcpus=8 used_vcpus=0 pci_stats=[]
2016-12-29 10:33:17.578 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Compute_service record updated for compute1:compute1
2016-12-29 10:33:37.780 3527 ERROR nova.service [-] Service error occurred during cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service Traceback (most recent call last):
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 244, in stop
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.manager.cleanup_host()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1158, in cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.instance_events.cancel_all_events()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in cancel_all_events
2016-12-29 10:33:37.780 3527 ERROR nova.service     tag=tag, data={})
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 307, in __init__
2016-12-29 10:33:37.780 3527 ERROR nova.service     setattr(self, key, kwargs[key])
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 72, in setter
2016-12-29 10:33:37.780 3527 ERROR nova.service     field_value = field.coerce(self, name, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     return self._type.coerce(obj, attr, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 317, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     raise ValueError(msg)
2016-12-29 10:33:37.780 3527 ERROR nova.service ValueError: Field value network-vif-plugged-a7c8ec01-a3f5-4b05-b19f is invalid
2016-12-29 10:33:37.780 3527 ERROR nova.service 
2016-12-29 10:33:39.520 3814 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

Error status after creating VM instance

Hi. I have a controller and a compute node for test purposes. Yesterday, I successfully was able to create instances. They got IP address, etc, everything was fine. Today, I am not able to create VMs, and the status of the instance is ERROR!

root@controller:/# openstack server list
+--------------------------------------+---------------------+--------+----------+------------+
| ID                                   | Name                | Status | Networks | Image Name |
+--------------------------------------+---------------------+--------+----------+------------+
| c59b9cda-3155-4fe8-9b6d-eca702d44431 | provider-instance11 | ERROR  |          | cirros     |
+--------------------------------------+---------------------+--------+----------+------------+

This is the log from nova-scheduler on contoller node:

2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     dbapi_connection = rec.get_connection()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connection = self.__connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     connection = self.__pool._invoke_creator(self)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     raise exc
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller' ([Errno 111] ECONNREFUSED)")
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db 
2016-12-29 09:50:46.001 4260 INFO nova.scheduler.host_manager [req-d2c13055-0041-4a6e-baca-351416376d83 - - - - -] Successfully synced instances from host 'compute1'.
2016-12-29 09:50:47.776 4260 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

This is the nova-compute log from the compute node:

2016-12-29 10:33:17.512 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Total usable vcpus: 8, total allocated vcpus: 0
2016-12-29 10:33:17.512 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Final resource view: name=compute1 phys_ram=15731MB used_ram=512MB phys_disk=1817GB used_disk=0GB total_vcpus=8 used_vcpus=0 pci_stats=[]
2016-12-29 10:33:17.578 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Compute_service record updated for compute1:compute1
2016-12-29 10:33:37.780 3527 ERROR nova.service [-] Service error occurred during cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service Traceback (most recent call last):
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 244, in stop
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.manager.cleanup_host()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1158, in cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.instance_events.cancel_all_events()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in cancel_all_events
2016-12-29 10:33:37.780 3527 ERROR nova.service     tag=tag, data={})
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 307, in __init__
2016-12-29 10:33:37.780 3527 ERROR nova.service     setattr(self, key, kwargs[key])
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 72, in setter
2016-12-29 10:33:37.780 3527 ERROR nova.service     field_value = field.coerce(self, name, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     return self._type.coerce(obj, attr, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 317, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     raise ValueError(msg)
2016-12-29 10:33:37.780 3527 ERROR nova.service ValueError: Field value network-vif-plugged-a7c8ec01-a3f5-4b05-b19f is invalid
2016-12-29 10:33:37.780 3527 ERROR nova.service 
2016-12-29 10:33:39.520 3814 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

Error status after creating VM instance

Hi. I have a controller and a compute node for test purposes. Yesterday, I successfully was able to create instances. They got IP address, etc, everything was fine. Today, I am not able to create VMs, and the status of the instance is ERROR!

root@controller:/# openstack server list
+--------------------------------------+---------------------+--------+----------+------------+
| ID                                   | Name                | Status | Networks | Image Name |
+--------------------------------------+---------------------+--------+----------+------------+
| c59b9cda-3155-4fe8-9b6d-eca702d44431 | provider-instance11 | ERROR  |          | cirros     |
+--------------------------------------+---------------------+--------+----------+------------+

This is the log from nova-scheduler on contoller node:

2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     dbapi_connection = rec.get_connection()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 563, in get_connection
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connection = self.__connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     connection = self.__pool._invoke_creator(self)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     self.connect()
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 922, in connect
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db     raise exc
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller' ([Errno 111] ECONNREFUSED)")
2016-12-29 09:50:37.672 4260 ERROR nova.servicegroup.drivers.db 
2016-12-29 09:50:46.001 4260 INFO nova.scheduler.host_manager [req-d2c13055-0041-4a6e-baca-351416376d83 - - - - -] Successfully synced instances from host 'compute1'.
2016-12-29 09:50:47.776 4260 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

This is the nova-compute log from the compute node:

2016-12-29 10:33:17.578 3527 INFO nova.compute.resource_tracker [req-22738202-edbf-4306-8744-df2257cea54a - - - - -] Compute_service record updated for compute1:compute1
2016-12-29 10:33:37.780 3527 ERROR nova.service [-] Service error occurred during cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service Traceback (most recent call last):
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 244, in stop
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.manager.cleanup_host()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1158, in cleanup_host
2016-12-29 10:33:37.780 3527 ERROR nova.service     self.instance_events.cancel_all_events()
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in cancel_all_events
2016-12-29 10:33:37.780 3527 ERROR nova.service     tag=tag, data={})
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 307, in __init__
2016-12-29 10:33:37.780 3527 ERROR nova.service     setattr(self, key, kwargs[key])
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 72, in setter
2016-12-29 10:33:37.780 3527 ERROR nova.service     field_value = field.coerce(self, name, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     return self._type.coerce(obj, attr, value)
2016-12-29 10:33:37.780 3527 ERROR nova.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 317, in coerce
2016-12-29 10:33:37.780 3527 ERROR nova.service     raise ValueError(msg)
2016-12-29 10:33:37.780 3527 ERROR nova.service ValueError: Field value network-vif-plugged-a7c8ec01-a3f5-4b05-b19f is invalid
2016-12-29 10:33:37.780 3527 ERROR nova.service 
2016-12-29 10:33:39.520 3814 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.