Ask Your Question
0

(Ocata) OpenStack High Availability Setup (glance issue)

asked 2017-06-20 11:18:28 -0500

Elangovan Anganann gravatar image

What is causing "Unauthorized (HTTP 401)" error?

I am setting up OpenStack HA environment using three controllers, two compute nodes, three ceph nodes, and two HAproxy nodes. I am doing manual installation (not using tripleO / Ansible). I successfully deployed ceph, mariadb-galera-cluster, rabbitmq-cluster, haproxy, and keystone). I have provided output of "openstack token issue" below to show keystone is working fine.

[root@controller1 ~]# cat keystonerc_admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin123
export OS_AUTH_URL=http://10.0.0.11:35357/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2
export PS1='[\u@\h \W (admin)]# '

[root@controller1 ~]# source keystonerc_admin
[root@controller1 ~ (admin)]# openstack token issue
+------------+---------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                     |
+------------+---------------------------------------------------------------------------------------------------------------------------+
| expires    | 2017-06-20T17:04:26+0000                                                                                                  |
| id         | gAAAAABZSUeKZLEtrDl54hxjOxgc4HAkj2Hgo-bXk4yerjt6cL4e-                                                                     |
|            | QCsUBM_aQWtyZLfoEWHID8SgXoBGd6eCmguG8MsJ9_9Ci2Oy4btBtchr0ul8YcGHASRTB9rTnGxwF0lsoS7nEcBwF9cYKQoJ-SISCQ-                   |
|            | nbD_W6atbeN4R3gfAN8C-KBmKgI                                                                                               |
| project_id | 96c9b397c67047269b9a59470cf35622                                                                                          |
| user_id    | 35e74dfcd02946d1ae95467f14c3bfb8                                                                                          |
+------------+---------------------------------------------------------------------------------------------------------------------------+

But when I try to list images in glance, I get unauthorized error.

[root@controller1 ~ (admin)]# openstack image list
Unauthorized (HTTP 401)
[root@controller1 ~ (admin)]# glance image-list
Invalid OpenStack Identity credentials.

I don't see any issue with glance user credentials in /etc/glance/glance-api.conf and /etc/glance/glance-registry.conf.

[keystone_authtoken]
auth_uri = http://10.0.0.11:5000
auth_url = http://10.0.0.11:35357
memcached_servers = controller1:11211,controller2:11211,controller3:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = glance
password = GLANCE_PASS

OpenStack Endpoint Details:

[root@controller1 ~ (admin)]# openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                        |
+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------+
| 50577ac48e7446beabcd4336ddd3ad2f | RegionOne | glance       | image        | True    | internal  | http://10.0.0.11:9292      |
| 56a6b82afcd3441fbd71499d62b9d3c1 | RegionOne | glance       | image        | True    | admin     | http://10.0.0.11:9292      |
| 7616f05adf54471e85a944e89786e5e4 | RegionOne | keystone     | identity     | True    | public    | http://10.0.0.11:5000/v3/  |
| 8f4702a609ff4d19a28a4a8e49691586 | RegionOne | glance       | image        | True    | public    | http://10.0.0.11:9292      |
| a8198d52bb674832b39ea5868021c90f | RegionOne | keystone     | identity     | True    | admin     | http://10.0.0.11:35357/v3/ |
| f1ecf62bf9aa42b7863e550f3a676826 | RegionOne | keystone     | identity     | True    | internal  | http://10.0.0.11:5000/v3/  |
+----------------------------------+-----------+--------------+--------------+---------+-----------+----------------------------+

HAProxy Configuration File (/etc/haproxy/haproxy.cfg)

global
    log         127.0.0.1 local0
    log         127.0.0.1 local1 notice
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4096
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    log                     global
    mode                    http
    option                  tcplog
    option                  dontlognull
    retries                 3
    option                  redispatch
    maxconn                 4096
    timeout connect         50000ms
    timeout client          50000ms
    timeout server          50000ms

listen stats 0.0.0.0:80
  mode http
  stats enable
  stats uri /stats
  stats realm HAProxy\ Statistics
  stats auth admin:openstack


listen galera 0.0.0.0:3306
        mode tcp
        balance roundrobin
        option tcpka
        option mysql-check user haproxy
        server controller1 10.0.0.12:3306 check weight 1
        server controller2 10.0.0.13:3306 check weight 1
        server controller3 10.0.0.14:3306 check weight 1

listen keystone_admin_cluster
  bind 0.0.0.0:35357
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server controller1 10.0.0.12:35357 check inter 2000 rise 2 fall 5
  server controller2 10.0.0.13:35357 check inter ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2017-06-20 11:40:57 -0500

Elangovan Anganann gravatar image

updated 2017-06-23 09:52:20 -0500

I solved the issue. Root cause is fernet-keys and credential-keys. The keys in three controllers were different. Once I copied the fernet-keys and credential-keys from controller1 to controller2 and controller3, glance worked well.

edit flag offensive delete link more
0

answered 2017-06-21 02:29:15 -0500

But how to add all the controllers IPs (10.0.0.12, 10.0.0.13, 10.0.0.14) to endpoint catalog?

You don't. You have an HA-Proxy load balancer in front of the three controllers and access them through the load balancer's VIP. So, only the VIP goes into the catalog.

edit flag offensive delete link more

Comments

I added Load Balancer's VIP (10.0.0.11) to the endpoint catalog. Now, when I do "openstack image list", I get "Unauthorized (HTTP 401)" error. I don't see any errors reported log files (keystone.log, keystone_access.log, httpd/error_log).

Elangovan Anganann gravatar imageElangovan Anganann ( 2017-06-21 20:57:21 -0500 )edit

I have also provided the output of "openstack --debug image list" for your review. Thanks.

Elangovan Anganann gravatar imageElangovan Anganann ( 2017-06-21 23:28:30 -0500 )edit

401 Unauthorized This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

This was in the HTTP packet

Elangovan Anganann gravatar imageElangovan Anganann ( 2017-06-21 23:34:34 -0500 )edit

I solved the issue. Root cause is fernet-keys and credential-keys. The keys in three controllers were different. Once I copied the fernet-keys and credential-keys from controller1 to controller2 and controller3, glance worked well.

Elangovan Anganann gravatar imageElangovan Anganann ( 2017-06-23 09:52:54 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2017-06-20 11:18:28 -0500

Seen: 569 times

Last updated: Jun 23 '17