Ask Your Question

Attila Szlovencsak's profile - activity

2018-11-09 04:30:03 -0500 received badge  Necromancer (source)
2016-04-12 07:52:50 -0500 received badge  Student (source)
2015-11-04 02:50:29 -0500 received badge  Famous Question (source)
2015-10-27 09:36:26 -0500 received badge  Notable Question (source)
2015-10-24 03:06:54 -0500 received badge  Self-Learner (source)
2015-10-22 00:54:00 -0500 received badge  Popular Question (source)
2015-10-20 09:43:24 -0500 answered a question Cinder RabbitMQ timeout problem with multiple rabbit hosts

Hi!

Setting kombu_reconnect_delay=3.0 seems to solve this problem.

https://ask.openstack.org/en/question...

2015-10-20 09:25:02 -0500 answered a question RabbitMQ failover issue Kilo

To answer my own question.. :)

It seems to be a timing problem inside kombu.... Browsing the code, I found the following lines in oslo_messaging/_drivers/impl_rabbit.py

# XXX(nic): when reconnecting to a RabbitMQ cluster
# with mirrored queues in use, the attempt to release the
# connection can hang "indefinitely" somewhere deep down
# in Kombu.  Blocking the thread for a bit prior to
# release seems to kludge around the problem where it is
# otherwise reproduceable.
if self.driver_conf.kombu_reconnect_delay > 0:
            time.sleep(self.driver_conf.kombu_reconnect_delay)

So increasing the default value (1.0) seems to solve the problem.

nova.conf

[oslo_messaging_rabbit]
...
kombu_reconnect_delay=3.0

And now in nova-api.log

2015-10-20 15:55:41.260 8909 ERROR oslo_messaging._drivers.impl_rabbit [req-b5dee03a-0fd7-420b-a962-c222362d8a7f 27e5ecf3a76c4db494f424bdb9188de1 78620f45f5e740c7b302e63717201a66 - - -] AMQP server on controller1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2015-10-20 15:55:44.266 8909 ERROR oslo_messaging._drivers.impl_rabbit [req-b5dee03a-0fd7-420b-a962-c222362d8a7f 27e5ecf3a76c4db494f424bdb9188de1 78620f45f5e740c7b302e63717201a66 - - -] AMQP server on controller1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2015-10-20 15:55:47.272 8909 DEBUG amqp [req-b5dee03a-0fd7-420b-a962-c222362d8a7f 27e5ecf3a76c4db494f424bdb9188de1 78620f45f5e740c7b302e63717201a66 - - -] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] _start /usr/lib/python2.7/dist-packages/amqp/connection.py:754
2015-10-20 15:55:47.275 8909 INFO oslo_messaging._drivers.impl_rabbit [req-b5dee03a-0fd7-420b-a962-c222362d8a7f 27e5ecf3a76c4db494f424bdb9188de1 78620f45f5e740c7b302e63717201a66 - - -] Reconnected to AMQP server on controller3:5672

If you have many nova-api instances (one master, and several child processes), you will see lots of Trying to reconnect messages, until all child processes are forced to fail over. During that period, you would see longer delays (like 30s) in compute REST API calls that needs AMQ (like nova pause <vm_id>).

2015-10-19 10:23:05 -0500 asked a question RabbitMQ failover issue Kilo

Hi!

I have an issue with HA RabbitMQ queue under Kilo (2015.1.1).

If a rabbit node fails, some of the services (like nova-scheduler) manage to fail over to an other node.

2015-10-19 16:47:41.741 28727 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on controller1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-19 16:47:42.763 28727 DEBUG amqp [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] _start /usr/lib/python2.7/dist-packages/amqp/connection.py:754
2015-10-19 16:47:42.780 28727 DEBUG amqp [-] Open OK! _open_ok /usr/lib/python2.7/dist-packages/amqp/connection.py:640
2015-10-19 16:47:42.782 28727 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:80
2015-10-19 16:47:42.790 28727 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:438
2015-10-19 16:47:43.053 28727 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on controller2:5672

But on the same node, nova-api fails with the following log

2015-10-19 16:47:42.014 11763 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on controller1:5672 is unreachable: [Errno 111
] ECONNREFUSED. Trying again in 1 seconds.
2015-10-19 16:47:42.271 11763 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] 
ECONNREFUSED
2015-10-19 16:47:42.899 11764 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on controller2:5672
2015-10-19 16:47:42.906 11795 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
2015-10-19 16:47:43.052 11764 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
2015-10-19 16:47:43.151 11796 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
2015-10-19 16:47:43.541 11796 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on controller3:5672
2015-10-19 16:47:43.543 11763 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
2015-10-19 16:47:43.569 11795 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on controller3:5672
2015-10-19 16:47:43.607 11763 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on controller3:5672
2015-10-19 16:47:44.163 11795 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
2015-10-19 16:47:44.326 11764 INFO oslo_messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111]

I wonder why it happens, since these two services use the same nova.conf ... (more)

2015-09-29 03:32:42 -0500 answered a question Console instance not accepting ":" char

I had the same problem. I figured out that the locale presented by the browser was different to the locale of my keyboard layout. So when my locale settings was

$ env |grep LC
LC_PAPER=hu_HU.utf8
LC_MONETARY=hu_HU.utf8
LC_NUMERIC=hu_HU.utf8
LC_MEASUREMENT=hu_HU.utf8
LC_TIME=hu_HU.utf8

, the en_US keyboard layout was producing strange letters, however the layout hu_HU was working correctly.

2015-01-20 07:57:04 -0500 received badge  Enthusiast
2015-01-16 18:01:35 -0500 answered a question use swift stat to vertify but get http return 401 unauthorized

This usually happens if swift-proxy fails to validate your token against keystone. So either you have wrong settings for admin_tenant_name ,admin_user or admin_password, or you admin_user (usually swift) does not have and admin role on tenant services.

As a way to debug, of course you could turn off token validation as suggested by ilya1725 (delete keystone_auth from pipeline:main in /etc/swift/proxy-server.conf). But this renders your swift "world-readable", so that anyone knowing your tenant-id could download your files, as simple as using

$> curl   http://<proxy>:8080/v1/AUTH_<tenant_id>/<container>/<path>

It is better to fix the parameters admin_tenant_name ,admin_user or admin_password. Last time I had this problem, as I was using service instead of services for admin_tenant_name.

2014-10-14 04:02:36 -0500 commented question New to Openstack, Adding a compute node

What a new compute node needs is a keystone,mysql,AMQ So check the related parameters in nova.conf, and api-paste.ini, like

  • nova.conf: sql_connection for mysql
  • nova.conf: qpid_xxx params for amq
  • api-paste.ini: auth_xxx and admin_xxx params for keystone

Copy values from existing node

2014-09-15 07:53:31 -0500 commented question ceilometer sample-list is not going up on Havanna

Might be a stupid question, but do you have instances running?

2014-08-14 11:50:12 -0500 commented answer how to specify architecture for a instance to run in compute node ?

See the edited post. Hope that helps

2014-08-14 11:49:38 -0500 received badge  Editor (source)
2014-08-14 06:35:42 -0500 answered a question how to specify architecture for a instance to run in compute node ?

It is rather works in an indirect way. Based on different properties of the instance, image, flavor the list of available compute nodes are filtered, and one is picked for starting your instance, but you cannot start directly where to/how to start your instance (like when you start qemu directly).

Openstack keeps track of the supported instance types for all compute nodes (this is set in nova.conf)

[root@controller ~]# mysql -e 'select hypervisor_hostname,supported_instances from nova.compute_nodes'
+--------------------------+--------------------------------------------------------------------------------------------------------+
 | hypervisor_hostname      | supported_instances                                                                                    |
+--------------------------+--------------------------------------------------------------------------------------------------------+
 | compute1.openstack.local | [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"]]                                                   |
 | compute2.openstack.local | [["i686", "qemu", "hvm"], ["i686", "kvm", "hvm"], ["x86_64", "qemu", "hvm"],  ["x86_64",   "kvm", "hvm"]] |
+--------------------------+--------------------------------------------------------------------------------------------------------+

So in case setting an image property "architecture" , the ImagePropertiesFilter at nova-scheduler will choose an appropriate compute node for you.

You could also use host aggregates, to suggest your preferred compute node. This case you create a flavor with a specific metadata (like arch=arm), and a matching host-aggregate (having the same key). If you start an instance with a that specific flavor, it will be started on the specific compute node(s) in the aggregate. See more: http://docs.openstack.org/havana/conf...

To start arm images on x86 host, you should 1. Install qemu (ensure you have qemu-system-arm) 2. Check qemu capabilities

[root@compute1 ~]# virsh capabilities |grep -C 2 arm
 <guest>
 <os_type>hvm</os_type>
     <arch name='arm'>
     <wordsize>64</wordsize>
     <emulator>/usr/bin/qemu-system-arm</emulator>
     <machine>integratorcp</machine>
      <machine>collie</machine>

3. If you installed qemu recently, probably you should restart nova-compute 4. Check the nova_compute mysql table again.

 [root@controller ~]# mysql -e 'select hypervisor_hostname,supported_instances from nova.compute_nodes'

 | hypervisor_hostname      | supported_instances                                                                                                                 
+--------------------------+--------------------------------------------------------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| compute1.openstack.local | [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"], ["arm", "qemu", "hvm"], ["microblaze", "qemu", "hvm"], ["microblazeel", "qemu", "hvm"], ["mips", "qemu", "hvm"], ["mipsel", "qemu", "hvm"], ["sparc", "qemu", "hvm"], ["ppc", "qemu", "hvm"], ["ppc64", "qemu", "hvm"], ["s390x", "qemu", "hvm"]]
2014-07-21 10:24:14 -0500 received badge  Teacher (source)