Ask Your Question
0

heartbeats for compute nodes?

asked 2011-06-09 20:18:58 -0600

fred-yang gravatar image

nova-compute creates "nova-compute" into CC DB.service table and new entry into DB.services table. nova-compute then periodically update DB.services.updated_at and report_count fields.

Any way for CC to know a compute node is restarted or the nova-compute service is restarted? any event notification method than continually polling nova DB in monitoring update_at & report_count?

Thanks -Fred

edit retag flag offensive close merge delete

14 answers

Sort by ยป oldest newest most voted
0

answered 2011-07-12 16:10:46 -0600

fred-yang gravatar image

GreenThread is a cooperative yield model according to Eventlet doc, so it should be safe. Thanks for sharing the light.

edit flag offensive delete link more
0

answered 2011-07-11 22:31:06 -0600

Hey Fred,

Correct, it's my understanding that since we're using Eventlet (a Reactor pattern), we shouldn't run into concurrency issues since the service is essentially single threaded. GreenPool is an Eventlet-aware mechanism, so it should be safe too.

Downside is it won't take advantage of multi-core/processors, but we can fire up more than one service.

-S

edit flag offensive delete link more
0

answered 2011-07-11 21:25:10 -0600

fred-yang gravatar image

Sandy,

Continue ZoneManager.service_states[] locking question on this same thread.

ZoneManager.update_service_capabilities() is updating service_states per compute_node periodic_tasks, where JsonFilter.filter_hosts() is also looping through service_states[] to filter each hosts. There is no data locking while accessing service_states[] from both controls, is it implicitly serialized through AMQP by nova-scheduler executing zoneManger and filter_hosts. Am I read it correct?!

nova-scheduler also schedules SchedulerManager.ping() periodically through greenthread. If we derive ping() from SchedulerManager() to update service_states[] periodically on the same scheduler node, any data locking needed or what will be the better locking method?

Thanks, -Fred

edit flag offensive delete link more
0

answered 2011-06-09 23:47:33 -0600

fred-yang gravatar image

Ok, it seems the polling is the way.. But I don't see service create time got reset when a compute node rebooted - I guess the logic is only to do compute_node_update() if it finds its service entry already. Any possibility to identify a compute node has been rebooted since a predefined time? the "watch" mechanism can only identify a new online node which doesn't have service entry yet I would like to build host trustiness state and would like to rebuild the information when a node got rebooted or got on-lined Thanks, -Fred

edit flag offensive delete link more
0

answered 2011-06-09 22:23:47 -0600

Hmm, that's interesting. I should add a hook in manage to dump the ZoneManager info. That would really be useful.

edit flag offensive delete link more
0

answered 2011-06-09 21:37:57 -0600

everett-toews gravatar image

I use

nova-manage service list | sort

and when I want to monitor it

watch -n 2 "nova-manage service list | sort"

Everett

On Thu, Jun 9, 2011 at 2:21 PM, fred yang < question160895@answers.launchpad.net > wrote:

New question #160895 on OpenStack Compute (nova): https://answers.launchpad.net/nova/+question/160895 (https://answers.launchpad.net/nova/+q...)

nova-compute creates "nova-compute" into CC DB.service table and new entry into DB.services table. nova-compute then periodically update DB.services.updated_at and report_count fields.

Any way for CC to know a compute node is restarted or the nova-compute service is restarted? any event notification method than continually polling nova DB in monitoring update_at & report_count?

Thanks -Fred


You received this question notification because you are an answer contact for OpenStack Compute (nova).

edit flag offensive delete link more
0

answered 2011-06-09 21:13:33 -0600

Look at nova.manager.SchedulerDependentManager

Any service that is alive (and has Capabilities defined) will report to the ZoneManager in the Schedulers when they come up. If you look at nova.scheduler.api, you'll see how you can query the schedulers to get status.

This is all done via AMQP and does not require DB updates/changes.

edit flag offensive delete link more
0

answered 2011-06-14 11:00:43 -0600

np Fred, happy to help ... don't hesitate to ping me if you run into issues.

edit flag offensive delete link more
0

answered 2011-06-14 00:56:28 -0600

fred-yang gravatar image

Thanks Sandy Walsh, that solved my question.

edit flag offensive delete link more
0

answered 2011-06-13 16:53:51 -0600

  1. Yup, sounds correct.

  2. I would make a new nova.scheduler.api method for this call rather than adjusting get_zone_capabilities, but you have the right idea.

Look forward to seeing the ML RFC!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-06-09 20:18:58 -0600

Seen: 115 times

Last updated: Jul 12 '11