Ask Your Question

hannuvisti's profile - activity

2016-05-31 04:05:44 -0500 received badge  Famous Question (source)
2016-03-31 09:18:38 -0500 received badge  Notable Question (source)
2016-03-31 09:18:38 -0500 received badge  Popular Question (source)
2016-03-09 03:00:16 -0500 received badge  Enthusiast
2016-03-08 05:53:57 -0500 asked a question Rabbitmq errors: no exchange 'reply_bf8ddab13a254547974842e6a27f9099' in vhost '/'"

Hi all,

I am running openstack Juno. Recently I have had severe problems with it, as rabbitmq fills the log with error messages like:

=ERROR REPORT==== 8-Mar-2016::11:17:16 ===
connection <0.8110.9>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_bf8ddab13a254547974842e6a27f9099' in vhost '/'",
            'exchange.declare'}

=ERROR REPORT==== 8-Mar-2016::11:17:16 ===
connection <0.8122.9>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_bf8ddab13a254547974842e6a27f9099' in vhost '/'",
            'exchange.declare'}

=ERROR REPORT==== 8-Mar-2016::11:17:16 ===
connection <0.8131.9>, channel 1 - soft error:
{amqp_error,not_found,
            "no exchange 'reply_bf8ddab13a254547974842e6a27f9099' in vhost '/'",
            'exchange.declare'}

and so forth. About 50MB of these get generated in about 5 minutes, then Rabbitmq just hangs and needs to be restarted. When I do the restart, my Openstack cluster actually works fine. Apparently something is wrong somewhere, but how do I find the broken component? Openstack logs do not reveal anything specific, just dropped MQ connections.

nova-manage service list

shows everything up and running up to the point when rabbitmq dies.

Any ideas where to look? What is this "vhost '/'"?

Hannu

2015-08-17 09:40:27 -0500 received badge  Nice Question (source)
2014-11-17 21:12:55 -0500 received badge  Good Answer (source)
2014-11-13 01:08:27 -0500 received badge  Famous Question (source)
2014-11-13 01:08:27 -0500 received badge  Notable Question (source)
2014-11-10 07:53:08 -0500 received badge  Nice Answer (source)
2014-11-10 07:37:16 -0500 received badge  Student (source)
2014-09-02 07:05:27 -0500 received badge  Teacher (source)
2014-09-02 07:05:27 -0500 received badge  Self-Learner (source)
2014-08-29 03:26:51 -0500 received badge  Famous Question (source)
2014-08-26 21:07:28 -0500 received badge  Famous Question (source)
2014-08-11 12:44:15 -0500 received badge  Notable Question (source)
2014-08-06 16:11:43 -0500 received badge  Popular Question (source)
2014-08-06 11:09:17 -0500 answered a question Neutron metadata agent "unexpected error"

... and the reason was regionOne - RegionOne mismatch in metadata-agent.ini.

Thanks dbaxps.

2014-08-06 11:08:36 -0500 commented question Neutron metadata agent "unexpected error"

Thanks. Of course it was regionOne - RegionOne confusion.

2014-08-06 09:30:13 -0500 commented question Neutron metadata agent "unexpected error"

Looks fine to me (added to the end of my initial post)

2014-08-06 09:29:06 -0500 received badge  Editor (source)
2014-08-06 09:12:25 -0500 asked a question Neutron metadata agent "unexpected error"

I am still struggling with the not-so-co-operative neutron. This is now a two-node test installation using virtualbox instances, Ubuntu 14.04, Icehouse. Control node (.101) runs everything except computing, which is done by compute (.103).
After lengthy trial and error session I now manage to launch instances but they are unreachable as the tenant network does not appear to get set up properly. The only real error in logs is in neutron metadata log:

2014-08-06 14:53:01.576 2048 ERROR neutron.agent.metadata.agent [req-ca7f07b7-671f-4ee6-99c0-56ae53db9ef0 None] Unexpected error.
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 110, in __call__
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
....
....
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     endpoint_type=self.endpoint_type)
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 81, in url_for
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     raise exceptions.EndpointNotFound()
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent EndpointNotFound: Could not find Service or Region in Service Catalog.

This appears to be a simple keystone database problem but I fail to see where it is. Neutron is configured to use the service tenant.

visti@control2:~$ keystone service-list
+----------------------------------+----------+----------+------------------------------+
|                id                |   name   |   type   |         description          |
+----------------------------------+----------+----------+------------------------------+
| 6c25e11229194961aac8f9b8e0a78746 |  cinder  |  volume  |   OpenStack Volume Service   |
| 9c325003ddb74c6c8720ff62092a9709 |  glance  |  image   |   OpenStack Image Service    |
| 69f3f9a792bc4d4abffdecb8d27d8175 | keystone | identity |      OpenStack Identity      |
| e8d4d4f3482949f7bf7245fb555136f3 | neutron  | network  | OpenStack Networking service |
| 79b28afb127041099dcb36017bbf84cb |   nova   | compute  |  OpenStack Compute Service   |
+----------------------------------+----------+----------+------------------------------+
visti@control2:~$ keystone endpoint-list
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
|                id                |   region  |                 publicurl                 |                internalurl                |                  adminurl                 |            service_id            |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
| 77357fbc77154140b19898179cc30dc9 | RegionOne |        http://172.16.1.101:9292/v2        |        http://10.10.10.101:9292/v2        |        http://10.10.10.101:9292/v2        | 9c325003ddb74c6c8720ff62092a9709 |
| 8d7f8cb0eee143dda00eea78dce7179e | RegionOne |         http://172.16.1.101:9696/         |         http://10.10.10.101:9696/         |         http://10.10.10.101:9696/         | e8d4d4f3482949f7bf7245fb555136f3 |
| 9b99110cf7d4457b8dda19bbfd00f326 | RegionOne | http://172.16.1.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | 6c25e11229194961aac8f9b8e0a78746 |
| a1a1655f16ad4c90a1f6bd318c1e0f82 | RegionOne |       http://172.16.1.101:5000/v2.0       |       http://10.10.10.101:5000/v2.0       |       http://10.10.10.101:35357/v2.0      | 69f3f9a792bc4d4abffdecb8d27d8175 |
| a307e4fa6f5e4ff386ad0d2664127f2f | RegionOne | http://172.16.1.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | 79b28afb127041099dcb36017bbf84cb |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
visti@control2:~$ keystone tenant-list
+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| 6faa4fd099c345b79dc0496b5a29119e |  admin  |   True  |
| 566c2813ebe64b0093e6bb7bd24d95e6 |   demo  |   True  |
| e75489fa55fd4c85a3c7da7fc4ac0716 | service |   True  |
+----------------------------------+---------+---------+

Everything else works. Neutron commands to create ext-net, demo-net and router between these work absolutely fine. It is just when launching an instance, these errors appear. The instance launches and it gets a private ip address. From the instance console the router is even reachable, but there is no connectivity to the router from the compute host or any other ... (more)

2014-08-06 09:04:00 -0500 received badge  Notable Question (source)
2014-08-05 05:43:00 -0500 commented answer Instance creation fails - neutron problem

Openvswitch logs shows the following:

2014-08-05T10:24:54.965Z|00027|bridge|INFO|bridge br-int: added interface tap0bd5655a-01 on port 2 2014-08-05T10:24:55.292Z|00028|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tap0bd5655a-01 device failed: No such device

2014-08-05 05:19:54 -0500 commented question Instance creation fails - neutron problem

I assume it is. http://docs.openstack.org/training-guides/content/lab000-openstack-training-labs.html (http://docs.openstack.org/training-gu...) is the guide I am following - more or less. I have gre tunnels defined as well as ovs installed. I have edited ml2 configuration file, too

2014-08-04 23:53:42 -0500 received badge  Popular Question (source)
2014-07-03 15:04:32 -0500 received badge  Popular Question (source)
2014-07-03 07:08:22 -0500 asked a question keystone user-list returns nothing

Hi,

I am trying to install Openstack (keystone version 0.7.1) on an Ubuntu 14.04 server. I am new to this so I decided to follow the Openstack installation guide. I managed to do well until this stage: http://docs.openstack.org/icehouse/in...

As long as I am using service tokens, everything is fine.  When I unset OS_SERVICE_TOKEN and OS_SERVICE_ENDPOINT, my luck stops there. 

I am able to run keystone token-get with either command line options (username/passwd/os-auth-url etc) or environment variables.  However, user-list works with neither  If I revert back to OS_SERVICE_TOKEN and OS_SERVICE_ENDPOINT it does work.  I do not get any error messages except the "no handlers" -warning, which appears to be normal. It just does not report back anything, nor do other commands like endpoint-get, tenant-list etc). 

visti@ubuntu:~$ keystone --debug --os-username=admin --os-password=xxx --os-tenant-name=admin --os-auth-url=http://172.241.0.101:35357/v2.0 user-list
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://172.241.0.101:35357/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "foobar"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.241.0.101
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 2520
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Thu, 03 Jul 2014 12:05:38 GMT', 'vary': 'X-Auth-Token', 'content-length': '2520', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-07-03T12:05:38.828937", "expires": "2014-07-03T13:05:38Z", "id": "MIIE-AYJKoZIhvcNAQcCoIIE7TCCBOkCAQExDTALBglghkgBZQMEAgEwggNKBgkqhkiG9w0BBwGgggM7BIIDN3siYWNjZXNzIjogeyJ0b2tlbiI6IHsiaXNzdWVkX2F0IjogIjIwMTQtMDctMDNUMTI6MDU6MzguODI4OTM3IiwgImV4cGlyZXMiOiAiMjAxNC0wNy0wM1QxMzowNTozOFoiLCAiaWQiOiAicGxhY2Vob2xkZXIiLCAidGVuYW50IjogeyJkZXNjcmlwdGlvbiI6ICJhZG1pbiB0ZW5hbnQiLCAiZW5hYmxlZCI6IHRydWUsICJpZCI6ICI3ODFmYjkzMGMxMDc0N2IxYjU1NGQ1NmI2NTY2M2VkNCIsICJuYW1lIjogImFkbWluIn19LCAic2VydmljZUNhdGFsb2ciOiBbeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTcyLjI0MS4wLjEwMTozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4yNDEuMC4xMDE6NTAwMC92Mi4wIiwgImlkIjogIjA1ZDI1MjNmZjhmNjRhYjViMGFkMjJiMWRjYjA1YWNiIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjI0MS4wLjEwMTo1MDAwL3YyLjAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaWRlbml0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAiYWRtaW4iLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjQyN2Q2MTZmMjQ5YjQ5ZjA4Njc0ODgzNGQwZGY0MDhjIiwgInJvbGVzIjogW3sibmFtZSI6ICJfbWVtYmVyXyJ9LCB7Im5hbWUiOiAiYWRtaW4ifV0sICJuYW1lIjogImFkbWluIn0sICJtZXRhZGF0YSI6IHsiaXNfYWRtaW4iOiAwLCAicm9sZXMiOiBbIjlmZTJmZjllZTQzODRiMTg5NGE5MDg3OGQzZTkyYmFiIiwgIjdlNjZhODhiZWZhNTQ2OGY4MDhkZTg1ZDVmMTI5YjA0Il19fX0xggGFMIIBgQIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVVbnNldDEOMAwGA1UEBwwFVW5zZXQxDjAMBgNVBAoMBVVuc2V0MRgwFgYDVQQDDA93d3cuZXhhbXBsZS5jb20CAQEwCwYJYIZIAWUDBAIBMA0GCSqGSIb3DQEBAQUABIIBABDnoll-Gwc1gBOXAFf5l+Wqgem4eccKWmZx5g+jWQ6lHqbuc8qpKTg9xzMErh+eJE2j90qFTBzZBB5xk5LKT-7vuztDtDRstzIKrKcDaXB3+3zfgDHPYprqq3UxesN7ciUWEOdGnsnRm4KmDPOU9Glhfb0DXKFlc3ai1+xciAph+KPW+qmSnWIMZ3vd-A57c8NFcEkdBe0q3JKFr6BFJHRF1XlK+oEg+bRNnJbf7MeCmt2jrks1mDlERe6WD6zn+5lLTK6u1vZypGJMAYm2FZycuH65bM1ucYAVbehcXCi-JbkcmF7V7ydBSfm54KLD-vzlLthuG28+1xafbj+Mz00=", "tenant": {"description": "admin tenant", "enabled": true, "id": "781fb930c10747b1b554d56b65663ed4", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://172.241.0.101:35357/v2.0", "region": "regionOne", "internalURL": "http://172.241.0.101:5000/v2.0", "id": "05d2523ff8f64ab5b0ad22b1dcb05acb", "publicURL": "http://172.241.0.101:5000/v2.0"}], "endpoints_links": [], "type": "idenity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "427d616f249b49f086748834d0df408c", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "7e66a88befa5468f808de85d5f129b04"]}}}

WARNING:keystoneclient.httpclient:Failed to retrieve management_url from token
DEBUG:iso8601.iso8601:Parsed 2014-07-03T13:05:38Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'13', 'daydash': u'03', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'38', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'07', 'day': None, 'minute': u'05'} with default timezone <iso8601.iso8601.Utc object at 0x7fcfd3c932d0>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'07' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 7 for 'month' with default 7
DEBUG:iso8601.iso8601:Got u'03' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 3 for 'day' with default 3
DEBUG:iso8601.iso8601:Got u'13' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'05' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'38' for 'second' with default None


If I enter a wrong username/password combination, I do get an error message ...
(more)