Ask Your Question

AMQP not returning nova notifications

asked 2016-06-29 02:23:49 -0500

Yaron Yogev gravatar image

updated 2017-06-06 06:05:28 -0500

I'm trying to fetch Nova notifications from OpenStack for a monitoring application. When I try launch and instance/rename/delete, I only get the port.create/update/delete notifications. The environment is RDO Mitaka. In another environment, with Mirantis and Liberty, I do get reasonable notifications. What am I doing wrong here?

My Python code follows:

from kombu.mixins import ConsumerMixin
from kombu.log import get_logger
from kombu import Queue, Exchange

class EventHandler:

def instance_add(self, notification): print("instance_add")

def instance_delete(self, notification): print("instance_delete")

def instance_update(self, notification): pass

def instance_down(self, notification): pass

def instance_up(self, notification): pass

def region_add(self, notification): pass

def region_delete(self, notification): pass

def region_update(self, notification): pass

logger = get_logger(__name__)

class Worker(ConsumerMixin):
  event_queues = [
      Exchange('nova', 'topic', durable=False),
      durable=False, routing_key='#')

  def __init__(self, connection):
    self.connection = connection
    self.handler = EventHandler()
    self.notification_responses = {
      "compute.instance.create.end": self.handler.instance_add,
      "compute.instance.delete.end": self.handler.instance_delete,
      "compute.instance.rebuild.end": self.handler.instance_update,
      "compute.instance.update": self.handler.instance_update,
      "servergroup.create": self.handler.region_add,
      "servergroup.delete": self.handler.region_delete,
      "servergroup.update": self.handler.region_update,
      "servergroup.addmember": self.handler.region_update,
      "compute.instance.shutdown.start": self.handler.instance_down,
      "compute.instance.power_off.start": self.handler.instance_down,
      "compute.instance.power_on.end": self.handler.instance_up,
      "compute.instance.suspend.start": self.handler.instance_down,
      "compute.instance.suspend.end": self.handler.instance_up

  def get_consumers(self, Consumer, channel):
    return [Consumer(queues=self.event_queues,

  def process_task(self, body, message):
    if "event_type" in body:
      self.handle_event(body["event_type"], body)

  def handle_event(self, type, notification):
    if type not in self.notification_responses:
      return ""
    return self.notification_responses[type](notification)

if __name__ == '__main__':
  from kombu import Connection
  from kombu.utils.debug import setup_logging
  # setup root logger
  setup_logging(loglevel='DEBUG', loggers=[''])

  with Connection('amqp://guest:guest@') as conn:
      worker = Worker(conn)
    except KeyboardInterrupt:
edit retag flag offensive close merge delete


Are notifications enabled for Nova? If you're using Neutron then the port updates are probably from Neutron itself, but may be placed on the queue to alert nova. See if the notification_driver has been configured for Nova.

mpetason gravatar imagempetason ( 2016-06-29 15:53:18 -0500 )edit

In /etc/nova/nova.conf I had notify_on_state_change=<none> In Mirantis Liberty env. I had notify_on_state_change=vm_state

Changed this to match in the Devstack Mitaka env. but still no good :-( Left it for now. Will try again after I have Mirantis Mitaka in place. Will report back then.

Yaron Yogev gravatar imageYaron Yogev ( 2016-06-30 00:39:40 -0500 )edit

Check the notification_driver in the RDO environment and compare it to the others. Also see if there are messages getting stuck in the queue:

rabbitmqctl list_queues name consumers messages|grep -i notification

on another note, do want services NOT to consume so your script can pull data?

mpetason gravatar imagempetason ( 2016-06-30 09:38:54 -0500 )edit

RDO: notification_driver=nova.openstack.common.notifier.rabbit_notifier,ceilometer.compute.nova_notifier Mirantis-Liberty, which works, doesn't have this setting at all.

I'm not sure what you mean about services not consuming. Do you mean that if my script consumes other services can't?

Yaron Yogev gravatar imageYaron Yogev ( 2016-06-30 10:10:35 -0500 )edit

Yes, this is also possible. If the exchange U r connecting is not fanout than rabbit will dispatch messages in round robin fashion between your consumer and default one.

yprokule gravatar imageyprokule ( 2016-07-06 09:03:38 -0500 )edit

3 answers

Sort by ยป oldest newest most voted

answered 2016-07-01 02:44:41 -0500

yprokule gravatar image

@yaron-yogev - I'd recommend to check if notifications are enabled and configured


If U have dedicated controllers/computes check config there and restart nova-(conductor|scheduler) services.

Also U can enable store_events=True in ceilometer.conf and restart ceilometer services.

Then, if all is correct U'll get events processed by ceilometer and available with next commands: ceilometer event-type-list and ceilometer-event-list

edit flag offensive delete link more


Thanks for the answer. Re. Ceilometer: I'm actually trying to write code that will serve any OpenStack administrator, not just in my site or company. It should work without having to install Ceilometer.

Yaron Yogev gravatar imageYaron Yogev ( 2016-07-03 02:13:56 -0500 )edit

Sounds interesting, though U need carefully investigate messages flow, not to become the only message consumer. So, did U succeed with notifications?

yprokule gravatar imageyprokule ( 2016-07-06 09:05:20 -0500 )edit

No. I reverted to using Mirantis Liberty for now. This is only in POC stage at the moment, so that was enough.

Yaron Yogev gravatar imageYaron Yogev ( 2016-07-06 09:37:24 -0500 )edit

answered 2016-12-04 09:16:19 -0500

Yaron Yogev gravatar image

updated 2016-12-04 09:18:12 -0500

Finally found the reason for this: The format of message in process_task() is now different, and event_type does not appear in body. Instead, body has JSON message in string "oslo.message". I loaded this string and it gives me event_type and payload like body used to have.

edit flag offensive delete link more

answered 2017-08-29 20:19:20 -0500

kbhonagiri gravatar image

Enable ceilometer in devstack local.conf file and after installation disable ceilometer notification service by "service devstack@ceilometer-anotification stop". This enables notifications for all services

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2016-06-29 02:23:49 -0500

Seen: 862 times

Last updated: Aug 29 '17