Ask Your Question
0

Nova completely stuck while giving some commands

asked 2013-01-26 17:25:05 -0500

colofabrix gravatar image

Hello! I'm setting up a system with openstack for the frist time and i'm becoming really crazy with this problem. After the installation everything was fine, I did my test and when i wanted to delete an instance suddenly the problem appeared. I gave the following command with no result, meaning it was waiting to complete. Ctrl+C stopped nova with this log:

fabrizio@uranus:/etc/nova$ nova delete ef61492b-14b6-4b43-9f49-51e6f152e26d

^CTraceback (most recent call last): File "/usr/bin/nova", line 9, in <module> load_entry_point('python-novaclient==2.9.0', 'console_scripts', 'nova')() File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 531, in main OpenStackComputeShell().main(sys.argv[1:]) File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 467, in main args.func(self.cs, args) File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1008, in do_delete _find_server(cs, args.server).delete() File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 41, in delete self.manager.delete(self) File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 528, in delete self._delete("/servers/%s" % base.getid(server)) File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 157, in _delete _resp, _body = self.api.client.delete(url) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 248, in delete return self._cs_request(url, 'DELETE', *kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 226, in _cs_request *kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 208, in _time_request resp, body = self.request(url, method, *kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 182, in request resp, body = super(HTTPClient, self).request(args, **kwargs) File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1543, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1293, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1263, in _conn_request response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1034, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 407, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status line = self.fp.readline() File "/usr/lib/python2.7/socket.py", line 430, in readline data = recv(1) KeyboardInterrupt

I obtained the same result with other commands like nova boot or nova stop. I analyzed log files and looked inside socket.py with no result os useful informations. Playing with services I noticed that nova-api is involved in this problem but I don't know what exactly is. I think there could be something misconfigured or some locked file, but I really don't know. I also tried do uninstall all the nova components and their dependencies and the first time it worked, then the problem appeared again and again I uninstalled everything but this time ... (more)

edit retag flag offensive close merge delete

9 answers

Sort by ยป oldest newest most voted
0

answered 2013-01-27 21:03:34 -0500

keith-tobin gravatar image

Is their an more of the log you posted? If so can you post here. What sort of a install have you?

edit flag offensive delete link more
0

answered 2013-01-27 21:08:35 -0500

keith-tobin gravatar image

It may be a good idea to just post the log if that is possible, use a paste bin like http://pastie.org

edit flag offensive delete link more
0

answered 2013-01-26 17:32:42 -0500

colofabrix gravatar image

This is the log of nova-api after the call

fabrizio@uranus:~/nova-log$ sudo nova delete ef61492b-14b6-4b43-9f49-51e6f152e26d

18:30:07 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/%3Cproject_id%3E/servers/detail (http://localhost:8774/v1.1/ 18:30:07 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request 18:30:07 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u''} 18:30:08 INFO nova.api.openstack.wsgi -)http://localhost:8774/v1.1/%3Cproject_id%3E/servers/detail (http://localhost:8774/v1.1/ returned with HTTP 200 18:30:08 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1//servers/detail HTTP/1.1" 200 1631 1.194018)

18:30:08 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/%3Cproject_id%3E/servers/detail (http://localhost:8774/v1.1/ 18:30:08 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request 18:30:08 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u''} 18:30:09 INFO nova.api.openstack.wsgi -)http://localhost:8774/v1.1/%3Cproject_id%3E/servers/detail (http://localhost:8774/v1.1/ returned with HTTP 200 18:30:09 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1//servers/detail HTTP/1.1" 200 1631 1.179668)

18:30:10 INFO nova.api.openstack.wsgi - DELETE http://localhost:8774/v1.1/%3Cproject_id%3E/servers/%3Cinstance-id%3E (http://localhost:8774/v1.1/ 18:30:10 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request 18:30:10 DEBUG nova.compute.api - - Going to try to terminate instance 18:30:11 DEBUG nova.openstack.common.rpc.amqp - Making asynchronous cast on compute.hemera... 18:30:11 DEBUG nova.openstack.common.rpc.amqp - Pool creating new connection 18:30:11 INFO nova.openstack.common.rpc.common - Connected to AMQP server on 192.168.3.1:5672)

edit flag offensive delete link more
0

answered 2013-01-27 23:26:51 -0500

colofabrix gravatar image

All my machines are old, 32bit pentium 4. The cluster controller is running with Ubuntu 12.04, no virtualization, with keystone, glance and all the nova services except nova-compute. nova-compute is running on the other 2 machines over XCP 1.6, but the problem appear in the cluster controller. In this file there are all the logs and che configuration files for nova: http://wikisend.com/download/966144/nova-all.tar.gz (http://wikisend.com/download/966144/n...)

edit flag offensive delete link more
0

answered 2013-01-28 17:17:05 -0500

colofabrix gravatar image

I uninstalled all the nova components and dependencies and then I installed again them. Everything seemed fine but when I nova boot the problem appeared again. After that command also nova network-list got stuck Here there is all the configuration and logs of the cloud controller: http://wikisend.com/download/610934/node-controller.tar.gz (http://wikisend.com/download/610934/n...) and here everything aboud the compute node: http://wikisend.com/download/120592/node-compute.tar.gz (http://wikisend.com/download/120592/n...)

edit flag offensive delete link more
0

answered 2013-01-28 19:35:47 -0500

keith-tobin gravatar image

Hold tight I am looking at files thanks.

edit flag offensive delete link more
0

answered 2013-01-29 22:04:36 -0500

keith-tobin gravatar image

Sorry for delay in getting back to you, in you log their is a a error/exception that occurs when you start nova-compute. This error occurs because the database is missing a table called 'instances'. If this table is missing from the database, something has gone wrong with your install of the database. I would log in in to MySql and verify if this table is missing and then figure why you did not add it, or did something go wrong with your install in some way. Here is a snip of the error. Feel free to ask any other questions and I will try to help. If you get thing working please close this bug.

Keith

2013-01-28 17:19:29 AUDIT nova.service [-] Starting compute node (version 2012.2-LOCALBRANCH:LOCALREVISION) 2013-01-28 17:19:30 DEBUG nova.utils [req-5a439208-87d2-4545-a35f-08607fdb3f49 None None] backend <module 'nova.db.sqlalchemy.api'="" from="" '="" usr="" lib="" python2.7="" dist-packages="" nova="" db="" sqlalchemy="" api.pyc'=""> from (pid=945) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494 2013-01-28 17:19:31 CRITICAL nova [-] (ProgrammingError) (1146, "Table 'nova.instances' doesn't exist") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.key AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight ... (more)

edit flag offensive delete link more
0

answered 2013-01-29 22:26:12 -0500

keith-tobin gravatar image

Adding to my above/last post, you have a second error in you config file, I would need to have a further look at what may cause it but, I would try fix the database problem I pointed out in last post and then restart all the services and see if you have problems still, if so post the new logs and I will help.

2013-01-28 17:15:48 CRITICAL nova [-] [Errno 13] Permission denied 2013-01-28 17:15:48 TRACE nova Traceback (most recent call last): 2013-01-28 17:15:48 TRACE nova File "/usr/bin/nova-compute", line 48, in <module> 2013-01-28 17:15:48 TRACE nova service.wait() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait 2013-01-28 17:15:48 TRACE nova _launcher.wait() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait 2013-01-28 17:15:48 TRACE nova super(ServiceLauncher, self).wait() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait 2013-01-28 17:15:48 TRACE nova service.wait() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait 2013-01-28 17:15:48 TRACE nova return self._exit_event.wait() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait 2013-01-28 17:15:48 TRACE nova return hubs.get_hub().switch() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch 2013-01-28 17:15:48 TRACE nova return self.greenlet.switch() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main 2013-01-28 17:15:48 TRACE nova result = function(args, *kwargs) 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server 2013-01-28 17:15:48 TRACE nova server.start() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start 2013-01-28 17:15:48 TRACE nova self.manager.init_host() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 272, in init_host 2013-01-28 17:15:48 TRACE nova self.driver.init_host(host=self.host) 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 164, in init_host 2013-01-28 17:15:48 TRACE nova vm_utils.ensure_correct_host(self._session) 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in ensure_correct_host 2013-01-28 17:15:48 TRACE nova this_vm_uuid = get_this_vm_uuid() 2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1803, in get_this_vm_uuid 2013-01-28 17:15:48 TRACE nova return f.readline().strip() 2013-01-28 17:15:48 TRACE nova IOError: [Errno 13] Permission denied 2013-01-28 17:15:48 TRACE nova

edit flag offensive delete link more
0

answered 2013-01-30 15:05:04 -0500

colofabrix gravatar image

The table 'instances' is present in the database and the hypervisor is fully accessible. Those where installation issues that I already solved before testing. For example after uninstalling and then installing again the command "nova network-list" worked without problem. Only after I used "nova boot" the first time then "nova network-list" stopped to work. Something locked after nova boot.

While waiting for your kind response I set up a clone of the cluster controller where I managed to get everything working. This means that the compute node is ok as the config files on the controller. I think I will erase the controller to get things done faster with the hope of never see this problem again.

Thanks for your help Keith ;)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-01-26 17:25:05 -0500

Seen: 180 times

Last updated: Jan 30 '13