Ask Your Question
1

Services start up priority at boot on controller

asked 2014-11-18 10:45:58 -0500

caddo gravatar image

Hi,

i have one cloud controller for testing with all management services installed on it and nvoa-compute, cinder-volume and neutron installed on separate boxes.

When i reboot the controller many nova services don't start complaining that they can't connect to the mysql instance that is hosted on the same box; if i restart those services manually after boot i get a positive result because the mysql is now up and happily running.

How can i tell nova services to wait to try to boot until mysql is up or at least retry after a while if it is not responding right away? Feels stupid to have to start those manually.

I'm running everything on ubuntu 14.04.1.

Thank you.

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted
0

answered 2014-11-18 11:43:36 -0500

yatin gravatar image

You should make mysql start on startup, this will let nova services to start without fail. You can do this using update-rc.d, sudo update-rc.d mysql defaults check this for other services as well, like rabbitmq-server, etc. You can use parameters described in the link for setting number of tries to connect to database, maximum reties, retry interval, etc: http://docs.openstack.org/juno/config...

edit flag offensive delete link more

Comments

They are already all set for automatic startup, the problem is that if they start and the mysql is not active yet the process dies and never retries.

caddo gravatar imagecaddo ( 2014-11-19 12:56:08 -0500 )edit

So you can manage the number of tries, time interval of tries using nova configuration parameters that are provided in link shared by me. Have you tried them??

yatin gravatar imageyatin ( 2014-11-19 22:17:03 -0500 )edit

Sorry for the delay.

I tried to configure retries but it doesn't seem to work in any way.

If they don't connect the first time then i have to start services manually.

caddo gravatar imagecaddo ( 2014-11-23 16:28:33 -0500 )edit

What configuration you have used for database connection retries from the link i shared.

yatin gravatar imageyatin ( 2014-11-23 22:45:09 -0500 )edit

db_inc_retry_interval = True db_max_retries = -1 db_max_retry_interval = 10 db_retry_interval = 1 max_retries = -1

caddo gravatar imagecaddo ( 2014-11-25 03:32:09 -0500 )edit
0

answered 2014-12-30 05:28:54 -0500

zhdan gravatar image

It seems that this is problem of OpenStack itself - it has no retry on opennning connection pool to db on startup after reboot. You can fix it manually by modifiying /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py file:

  • add the following lines to the beginning of the file

        import logging
        LOG = logging.getLogger(__name__)
        import time
    
  • and replace the following

        return self._connection_cls(self,
                                self.pool.connect(),
                                close_with_result=close_with_result,
                                **kwargs)
    

    by

        ret = None
        try:
            ret= self._connection_cls(self,
                                self.pool.connect(),
                                close_with_result=close_with_result,
                                **kwargs)
        except:
            LOG.warn('Retrying to connect to the database, due to unavailable database! Sleeping 10 seconds before retry...')
            time.sleep( 10 )
            ret= self._connection_cls(self,
                                self.pool.connect(),
                                close_with_result=close_with_result,
                                **kwargs)
        return ret
    

Literally this will add 10 seconds delay to initial pool opening in cae of error and then reconnect.

In my case it was used for fixing (OperationalError) (2003, "Can't connect to MySQL server on 'jcloud' (111)") None None in logs of nova-cert and nova-scheduler after reboot.

edit flag offensive delete link more
0

answered 2015-05-13 12:54:55 -0500

isabyr gravatar image

updated 2015-05-13 23:13:28 -0500

I also had this problem. This solution http://stackblog.net/?p=14 worked fine for me.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-11-18 10:45:58 -0500

Seen: 1,063 times

Last updated: May 13 '15