Revision history [back]

click to hide/show revision 1
initial version

Controller backup and recovery rabbitmq error

Hello world,

hope you doing fine while reading this issue...me????...im losing my hair when trying to setup fail recovery on openstack(legacy) controller node...Please...i need your suggestions and recommendations to solve this issue.

I follow the link below:

http://docs.openstack.org/openstack-ops/content/backup_and_recovery.html)http://docs.openstack.org/openstack-ops/content/backup_and_recovery.html

i tried to setup two controller node on ubuntu12.04 using chef,everything is working well,except for fail recovery.

172.50.0.206 controller master

172.50.0.207 controller standby

Please see my steps below.

im using database replication technique.

1. Controller Master

mysql> SHOW MASTER STATUS; +------------------+----------+-------------------------------------------------------------------+------------------+ | File | Position | Binlog_Do_DB
| Binlog_Ignore_DB | +------------------+----------+-------------------------------------------------------------------+------------------+ | mysql-bin.000007 | 1380647 | ceilometer,cinder,glance,heat,horizon,keystone,mysql,neutron,nova | | +------------------+----------+-------------------------------------------------------------------+------------------+ 1 row in set (0.00 sec)

2. Controller-standby

> mysql> SHOW SLAVE STATUS\G
> *************************** 1. row ***************************
>                Slave_IO_State: Waiting for master to send event
>                   Master_Host: 172.50.0.206
>                   Master_User: controller-standby
>                   Master_Port: 3306
>                 Connect_Retry: 60
>               Master_Log_File: mysql-bin.000007
>           Read_Master_Log_Pos: 1394002
>                Relay_Log_File: mysql-relay-bin.000007
>                 Relay_Log_Pos: 1394148
>         Relay_Master_Log_File: mysql-bin.000007
>              Slave_IO_Running: Yes
>             Slave_SQL_Running: Yes

3.Rsync all files and database backup from controller-master to controller-standby

> nova1="/etc/nova/" 
> nova2="/var/lib/nova/" 
> glance1="/etc/glance/" 
> glance2="/var/lib/glance/" 
> glance3="/var/lib/glance/images/" 
> keystone1="/etc/keystone/" 
> keystone2="/var/lib/keystone/" 
> cinder1="/etc/cinder/" 
> cinder2="/var/lib/cinder/" 
> mysql="/etc/mysql" 
> dashboard="/etc/openstack-dashboard/" 
> rabbit="/etc/rabbitmq/"

And controller-standby recovery

@controller-standby node

gunzip < /root/dbbackup/all-databases-controller-master.20150219.sql.gz | mysql -u root -psecrete

4.shutdown controller-master

5.change controller-standby configuration to become controller-master

controller-standby ip address change from 172.50.0.207 to 172.50.0.206
controller-standby database:change from slave to master
controller-standby hostname from controller-standby to controller-master

After i restart all the services,error occured on the new controller-master.

Error 1:rabbitmq

root@controller-integ:/var/log/rabbitmq# tail -f rabbit@controller-master.log
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

=ERROR REPORT==== 9-Mar-2015::11:08:44 ===
closing AMQP connection <0.15268.77> (172.50.0.206:45275 -> 172.50.0.206:5672):
{handshake_error,opening,0,
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

Error 2 all are triple X

root@controller-integ:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-conductor   controller-master                     internal         enabled    XXX   2015-02-19 08:54:59
nova-scheduler   controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-cert        controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-consoleauth controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-network     controller-master                     internal         enabled    XXX   2015-02-19 08:54:54

Please i need your help..Thank you in advance.

Controller backup and recovery rabbitmq error

Hello world,

hope you doing fine while reading this issue...me????...im losing my hair when trying to setup fail recovery on openstack(legacy) controller node...Please...i need your suggestions and recommendations to solve this issue.

I follow the link below:

http://docs.openstack.org/openstack-ops/content/backup_and_recovery.html)http://docs.openstack.org/openstack-ops/content/backup_and_recovery.html

i tried to setup two controller node on ubuntu12.04 using chef,everything is working well,except for fail recovery.

172.50.0.206 controller master

172.50.0.207 controller standby

Please see my steps below.

im using database replication technique.

1. Controller Master

mysql> SHOW MASTER STATUS; +------------------+----------+-------------------------------------------------------------------+------------------+ | File | Position | Binlog_Do_DB
| Binlog_Ignore_DB | +------------------+----------+-------------------------------------------------------------------+------------------+ | mysql-bin.000007 | 1380647 | ceilometer,cinder,glance,heat,horizon,keystone,mysql,neutron,nova | | +------------------+----------+-------------------------------------------------------------------+------------------+ 1 row in set (0.00 sec)

2. Controller-standby

> mysql> SHOW SLAVE STATUS\G
> *************************** 1. row ***************************
>                Slave_IO_State: Waiting for master to send event
>                   Master_Host: 172.50.0.206
>                   Master_User: controller-standby
>                   Master_Port: 3306
>                 Connect_Retry: 60
>               Master_Log_File: mysql-bin.000007
>           Read_Master_Log_Pos: 1394002
>                Relay_Log_File: mysql-relay-bin.000007
>                 Relay_Log_Pos: 1394148
>         Relay_Master_Log_File: mysql-bin.000007
>              Slave_IO_Running: Yes
>             Slave_SQL_Running: Yes

3.Rsync all files and database backup from controller-master to controller-standby

> nova1="/etc/nova/" 
> nova2="/var/lib/nova/" 
> glance1="/etc/glance/" 
> glance2="/var/lib/glance/" 
> glance3="/var/lib/glance/images/" 
> keystone1="/etc/keystone/" 
> keystone2="/var/lib/keystone/" 
> cinder1="/etc/cinder/" 
> cinder2="/var/lib/cinder/" 
> mysql="/etc/mysql" 
> dashboard="/etc/openstack-dashboard/" 
> rabbit="/etc/rabbitmq/"

And controller-standby recovery

@controller-standby node

gunzip < /root/dbbackup/all-databases-controller-master.20150219.sql.gz | mysql -u root -psecrete

4.shutdown controller-master

5.change controller-standby configuration to become controller-master

controller-standby ip address change from 172.50.0.207 to 172.50.0.206
controller-standby database:change from slave to master
controller-standby hostname from controller-standby to controller-master

After i restart all the services,error occured on the new controller-master.

Error 1:rabbitmq

root@controller-integ:/var/log/rabbitmq# root@controller-master:/var/log/rabbitmq# tail -f rabbit@controller-master.log
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

=ERROR REPORT==== 9-Mar-2015::11:08:44 ===
closing AMQP connection <0.15268.77> (172.50.0.206:45275 -> 172.50.0.206:5672):
{handshake_error,opening,0,
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

Error 2 all are triple X

root@controller-integ:~# root@controller-master:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-conductor   controller-master                     internal         enabled    XXX   2015-02-19 08:54:59
nova-scheduler   controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-cert        controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-consoleauth controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-network     controller-master                     internal         enabled    XXX   2015-02-19 08:54:54

Please i need your help..Thank you in advance.

Controller backup and recovery rabbitmq error

Hello world,

hope you doing fine while reading this issue...me????...im losing my hair when trying to setup fail recovery on openstack(legacy) controller node...Please...i need your suggestions and recommendations to solve this issue.

I follow the link below:

http://docs.openstack.org/openstack-ops/content/backup_and_recovery.html)http://docs.openstack.org/openstack-ops/content/backup_and_recovery.htmlhttp://docs.openstack.org/openstack-ops/content/backup_and_recovery.html

i tried to setup two controller node on ubuntu12.04 using chef,everything is working well,except for fail recovery.

172.50.0.206 controller master

172.50.0.207 controller standby

Please see my steps below.

im using database replication technique.

1. Controller Master

mysql> SHOW MASTER STATUS; +------------------+----------+-------------------------------------------------------------------+------------------+ | File | Position | Binlog_Do_DB
| Binlog_Ignore_DB | +------------------+----------+-------------------------------------------------------------------+------------------+ | mysql-bin.000007 | 1380647 | ceilometer,cinder,glance,heat,horizon,keystone,mysql,neutron,nova | | +------------------+----------+-------------------------------------------------------------------+------------------+ 1 row in set (0.00 sec)

2. Controller-standby

> mysql> SHOW SLAVE STATUS\G
> *************************** 1. row ***************************
>                Slave_IO_State: Waiting for master to send event
>                   Master_Host: 172.50.0.206
>                   Master_User: controller-standby
>                   Master_Port: 3306
>                 Connect_Retry: 60
>               Master_Log_File: mysql-bin.000007
>           Read_Master_Log_Pos: 1394002
>                Relay_Log_File: mysql-relay-bin.000007
>                 Relay_Log_Pos: 1394148
>         Relay_Master_Log_File: mysql-bin.000007
>              Slave_IO_Running: Yes
>             Slave_SQL_Running: Yes

3.Rsync all files and database backup from controller-master to controller-standby

> nova1="/etc/nova/" 
> nova2="/var/lib/nova/" 
> glance1="/etc/glance/" 
> glance2="/var/lib/glance/" 
> glance3="/var/lib/glance/images/" 
> keystone1="/etc/keystone/" 
> keystone2="/var/lib/keystone/" 
> cinder1="/etc/cinder/" 
> cinder2="/var/lib/cinder/" 
> mysql="/etc/mysql" 
> dashboard="/etc/openstack-dashboard/" 
> rabbit="/etc/rabbitmq/"

And controller-standby recovery

@controller-standby node

gunzip < /root/dbbackup/all-databases-controller-master.20150219.sql.gz | mysql -u root -psecrete

4.shutdown controller-master

5.change controller-standby configuration to become controller-master

controller-standby ip address change from 172.50.0.207 to 172.50.0.206
controller-standby database:change from slave to master
controller-standby hostname from controller-standby to controller-master

After i restart all the services,error occured on the new controller-master.

Error 1:rabbitmq

root@controller-master:/var/log/rabbitmq# tail -f rabbit@controller-master.log
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

=ERROR REPORT==== 9-Mar-2015::11:08:44 ===
closing AMQP connection <0.15268.77> (172.50.0.206:45275 -> 172.50.0.206:5672):
{handshake_error,opening,0,
                 {amqp_error,access_refused,
                             "access to vhost '/nova' refused for user 'guest'",
                             'connection.open'}}

Error 2 all are triple X

root@controller-master:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-conductor   controller-master                     internal         enabled    XXX   2015-02-19 08:54:59
nova-scheduler   controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-cert        controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-consoleauth controller-master                     internal         enabled    XXX   2015-02-19 08:54:57
nova-network     controller-master                     internal         enabled    XXX   2015-02-19 08:54:54

Please i need your help..Thank you in advance.