Ask Your Question
0

Red Dwarf DevStack int-tests Hangs on WaitForGuestInstallationToFinish

asked 2013-04-11 00:14:53 -0500

suntr gravatar image

After following https://wiki.openstack.org/wiki/Reddwarf-installation (https://wiki.openstack.org/wiki/Reddw...)

and executing:

$ ~/reddwarf-integration/scripts/redstack post-devstack mysql $ sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

$ ~/reddwarf-integration/scripts/redstack int-tests

Hangs forever until timeout @:

WaitForGuestInstallationToFinish test_instance_created

It seems this is occurring because the poller is convinced that the state is still BUILD.

$ ~/reddwarf-integration/scripts/redstack nova-client list

+--------------------------------------+---------------------------------+--------+------------------+ | ID | Name | Status | Networks | +--------------------------------------+---------------------------------+--------+------------------+ | 9f879821-566d-4ad7-a44f-3018b22d1314 | TEST_2013-04-10 17:00:50.986100 | ACTIVE | private=10.0.0.2 | +--------------------------------------+---------------------------------+--------+------------------+

$ ~/reddwarf-integration/scripts/redstack nova-client show

+-------------------------------------+---------------------------------------------------------------+ | Property | Value | +-------------------------------------+---------------------------------------------------------------+ | status | ACTIVE | | updated | 2013-04-11T00:02:31Z | | OS-EXT-STS:task_state | None | | OS-EXT-SRV-ATTR:host | suntr-vb | | key_name | None | | image | precise_mysql_image (2d1b5fa5-caa4-4bbc-a00c-9447f0ca05cf) | | private network | 10.0.0.2 | | hostId | a819f57851bc97a4c47dae336478b3bfdd7aee769d6175c76002ecf6 | | OS-EXT-STS:vm_state | active | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-SRV-ATTR:hypervisor_hostname | suntr-vb | | flavor | m1.tiny (1) | | id | 9f879821-566d-4ad7-a44f-3018b22d1314 | | security_groups | [{u'name': u'SecGroup_0354c20c-feff-4fed-822d-a8a0e651e983'}] | | user_id | 558e6b73ae754a08aeea5d5ec40d6f2d | | name | TEST_2013-04-10 17:00:50.986100 | | created | 2013-04-11T00:00:54Z | | tenant_id | 1e709b78937447ad971cc286e196752a | | OS-DCF:diskConfig | MANUAL | | metadata | {} | | accessIPv4 | | | accessIPv6 | | | progress | 0 | | OS-EXT-STS:power_state | 1 | | OS-EXT-AZ:availability_zone | nova | | config_drive | | +-------------------------------------+---------------------------------------------------------------+

$ ~/reddwarf-integration/scripts/redstack rd-client instance list

{ "flavor": { "id": "1", "links": [ { "href": "https://10.0.2.15:8779/v1.0/1e709b78937447ad971cc286e196752a/flavors/1", "rel": "self" }, { "href": "https://10.0.2.15:8779/flavors/1", "rel": "bookmark" } ] }, "id": "0354c20c-feff-4fed-822d-a8a0e651e983", "links": [ { "href": "https://10.0.2.15:8779/v1.0/1e709b78937447ad971cc286e196752a/instances/0354c20c-feff-4fed-822d-a8a0e651e983", "rel": "self" }, { "href": "https://10.0.2.15:8779/instances/0354c20c-feff-4fed-822d-a8a0e651e983", "rel": "bookmark" } ], "name": "TEST_2013-04-10 17:00:50.986100", "status": "BUILD", "volume": { "size": 1 } }

Trying to do a real mysql deployment stays hung in BUILD as well (so it's not isolated to int-tests)

Any ideas?

Specs:

VirtualBox 4.2.10 Host => Mac OS X 10.6.8 (Core i7) w/ 8 GB Mem Guest => Ubuntu 12.04.2 LTS (64-bit)

Red Dwarf DB Data:

mysql> select * from instances \G; ******** 1. row ******** id: 0354c20c-feff-4fed-822d-a8a0e651e983 created: 2013-04-11 00:00:51 updated: 2013-04-11 00:00:54 name: TEST_2013-04-10 17:00:50.986100 hostname: NULL compute_instance_id: 9f879821-566d-4ad7-a44f-3018b22d1314 task_id: 1 task_description: No tasks for the instance. task_start_time: NULL volume_id: 984cdeb0-2bbc-482b-89fe-46030ca7db62 flavor_id: 1 volume_size: 1 tenant_id: 1e709b78937447ad971cc286e196752a server_status: NULL deleted: 0 deleted_at: NULL 1 row in set (0.00 sec)

ERROR: No query specified

mysql> select * from service_statuses \G; ******** 1. row ******** id: 04d0d7a2-d021-44d4-a272-a6c68cab6227 instance_id: 0354c20c-feff-4fed-822d-a8a0e651e983 status_id: 23 status_description: new updated_at: NULL 1 row in set (0.00 sec)

Thanks!

edit retag flag offensive close merge delete

6 answers

Sort by ยป oldest newest most voted
0

answered 2013-04-12 03:57:04 -0500

hubcap gravatar image

awesome. iptables rules generally disappear upon host reboots so you need to make sure you 1) save that rule on the host, or 2) rerun it when u reboot.

edit flag offensive delete link more
0

answered 2013-04-11 17:43:22 -0500

hubcap gravatar image

U might also try ssh'ing in to the vm (assuming 10.0.0.2 is the vm ip, just ssh 10.0.0.2). Try an apt-get update. Also check /tmp/logfile.txt to see if the guest ever got up and running. Then give us your findings.

edit flag offensive delete link more
0

answered 2013-04-11 00:46:08 -0500

It could be that your instance can't get a line out and can't finish its apt-get upgrade/install jobs.

try sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE from the host (not the instance)

edit flag offensive delete link more
0

answered 2013-04-12 01:02:50 -0500

suntr gravatar image

After starting from scratch (you'll see that the users have changed below), I was finally able to SSH into the instance, but the status remains BUILD via rd-client.

From /var/log/upstart/reddwarf-guest.log (on the reddwarf instance):

++++++++++++ stack@ubuntu:/var/log/upstart$ sudo cat reddwarf-guest.log bash: /home/stack/reddwarf/bin/reddwarf-guestagent: No such file or directory Traceback (most recent call last): File "/home/stack/reddwarf/bin/reddwarf-guestagent", line 40, in <module> from reddwarf.common import rpc File "/home/stack/reddwarf/reddwarf/common/rpc.py", line 25, in <module> from reddwarf.openstack.common.rpc import service as rpc_service File "/home/stack/reddwarf/reddwarf/openstack/common/rpc/service.py", line 24, in <module> from reddwarf.openstack.common import service File "/home/stack/reddwarf/reddwarf/openstack/common/service.py", line 30, in <module> import extras ImportError: No module named extras ++++++++++++

It looks like the guest failed to init properly, causing the message to never be peeled off the queue.

Any ideas on why this is occurring? I'll be investigating in parallel....

Thanks!

edit flag offensive delete link more
0

answered 2013-04-12 01:28:01 -0500

suntr gravatar image

Is this related to https://review.openstack.org/#/c/20145/ ?

/reddwarf/tools/test-requires has "testtools>=0.9.22" /reddwarf/tools/pip-requires has "extras"

Perhaps changing the testtools value to "testtools>=0.9.22,<=0.9.24" will fix it?

edit flag offensive delete link more
0

answered 2013-04-12 02:01:25 -0500

suntr gravatar image

Disregard, I needed to re-exec "sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE".

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-04-11 00:14:53 -0500

Seen: 487 times

Last updated: Apr 12 '13