Ask Your Question
1

cinder volume quotas broken for RDO install in centOs?

asked 2016-01-05 15:28:47 -0500

theque42 gravatar image

updated 2016-01-06 03:21:50 -0500

Just installed openstack according to rdo concept. Both times packstack claims success. Even tried a --allinone setup, same failure. openstack quota show <id> fails.

But when I run my scripts for creating projects, users and setting quotas, it starts to fail.

Seems like all quota commands fail when it comes to volume quotas, regardless of if I use openstack or cinder cli, or horizon for that matter. System log complains with things like "Provide either auth_url or endpoint". (nova|neutron quota-show <id> works fine)

Attempting to use horizon to set the default quotas, opens a window with the existing default quota values, but the three volume related parameters, are simply empty.

Or am I just stupid trying to use RDO and standard centOs? (fedora didnt work better). It seems like starting packstack with an identical answer-file, will give different results, every day. The success rate of a packstack install, for me, is about 1 in 10. The actual CLI command just returns:

ERROR: The server has either erred or is incapable of performing the requested operation.

where the system log says:

ERROR cinder.api.middleware.fault [req-70f24ab7-4f8e-4f5a-90cb-54f2234893bb 113aa8326a7d4b1382881440f47b5e6a 8c4c9b18fa9346238e172af5a537a20c - - -] Caught error: Not enough information to determine URL. Provide either auth_url or endpoint

Sniffing on the Volume API call returns:

GET /v2/UUID/os-quota-sets/UUID?usage=False HTTP/1.1.
Host: 10.10.11.100:8776.
Connection: keep-alive.
User-Agent: python-cinderclient.
Accept-Encoding: gzip, deflate.
Accept: application/json.
X-Auth-Token: UUID.


HTTP/1.1 500 Internal Server Error.
Content-Length: 128.
Content-Type: application/json; charset=UTF-8.
X-Compute-Request-Id: req-UUID.
X-Openstack-Request-Id: req-UUID.
Date: Wed, 06 Jan 2016 09:19:13 GMT.
Connection: keep-alive.
.
{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

Sigh

edit retag flag offensive close merge delete

Comments

Note that the UUIDs are reg-exped above, to make it easier to read.

theque42 gravatar imagetheque42 ( 2016-01-06 03:17:28 -0500 )edit

cinder-api service must be down for some reason. Check cinder-api.log for possible reasons. Reasons could be similar to case here

Bharath gravatar imageBharath ( 2016-01-06 03:40:15 -0500 )edit

Even this could be of help.

Bharath gravatar imageBharath ( 2016-01-06 03:41:22 -0500 )edit
2

I dont see how those links help, since they concern nova, and my nova works fine. If my problem was somehow releated to python eventlets, or similar, I would still classify this as a bug in rdo/packstack, since my system is fully updated with yum update.

theque42 gravatar imagetheque42 ( 2016-01-06 04:03:21 -0500 )edit

I'm having exactly the same problem, and it is new somehow. I'm trying to rebuild from known good kickstart file (CentOS 7) and answers file. Simple "packstack --allinone" has the same problems. Even tried installing on different hardware & VMware. All have the same errors, exactly as stated above.

pmbailey gravatar imagepmbailey ( 2016-01-06 15:57:03 -0500 )edit

4 answers

Sort by ยป oldest newest most voted
6

answered 2016-01-08 03:44:48 -0500

Jeffrey Chou gravatar image

Hi

I just encountered the same issue today. Base on the cinder log, I tried to look for something wrong in "cinder.conf" and found there is no key assigned in [keystone_authtoken].

So I added must information just like the guide from ttp://docs.openstack.org/liberty/install-guide-rdo/cinder-controller-install.html

In /etc/cinder/cinder.conf

auth_uri = http://[keystone IP]:5000
auth_url = http://[keystone IP]:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = services
username = cinder
password = [cinder's password]

Hope this can help you...

edit flag offensive delete link more

Comments

This worked perfect! Thanks a million! What happened to packstack to cause it to forget cinder?

pmbailey gravatar imagepmbailey ( 2016-01-08 10:52:05 -0500 )edit

Awesome! Thanks .. Note that the project_name is indeed "services" (plural) not just "service" as in the liberty guide you linked to

kim0 gravatar imagekim0 ( 2016-01-08 13:25:34 -0500 )edit

Are you ffing telling me this solves it?!?!?! I did notice that mismatch, but since I got some other cinder-stuff to work (and now I cant remember what...sigh) I ruled it out as not related..... Maybe I should start using my own advice from my troubleshooting courses: "Verify your assumptions"...

theque42 gravatar imagetheque42 ( 2016-01-08 15:28:04 -0500 )edit

Halleluja! I am to relieved that this solved it so I cant be bothered with the fact that I was starting at the answer two bloody days ago! I cant thank you enough! But it seems like this should definitely be bugreported!

theque42 gravatar imagetheque42 ( 2016-01-08 16:13:57 -0500 )edit

That did it. Thanks a lot!!!

bcollins gravatar imagebcollins ( 2016-01-09 07:21:02 -0500 )edit
2

answered 2016-01-08 23:14:55 -0500

BentTerp gravatar image

I got it working by only adding - and after fixing this, then I could launch instances from new volumes.

auth_uri = http://localhost:5000/

This is probably the minimal change needed. But obviously, I would never have come up with it on my own, so credit goes to the previous posts.

edit flag offensive delete link more

Comments

that worked for me too..

omerfsen gravatar imageomerfsen ( 2016-01-22 15:07:19 -0500 )edit
0

answered 2016-01-07 00:54:30 -0500

meappy gravatar image

updated 2016-01-07 00:55:28 -0500

Hi

As you will know I am experiencing the same issue as OP - this is not a solution yet - just steps I am taking to troubleshoot, and hopefully get to the bottom of this.

Tools used

  1. tcpdump

    tcpdump -s 1024 -i any -tttt -vvvv -A port 8776

  2. strace

    for i in $(ps aux | grep 'cinder-api' | grep -v 'grep' | awk '{print $2}' | tail -n +2);do strace -f -p $i -o /tmp/$i.log & done

Syntax and sample output to test "cinder quota-show" via curl (output may vary - however syntax should auto-retrieve/populate endpoint and auth token)

HTTP 200 Response

[root@controller-A ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 200 OK
X-Compute-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Content-Type: application/json
Content-Length: 216
X-Openstack-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Date: Thu, 07 Jan 2016 06:39:57 GMT

{"quota_set": {"snapshots_iscsi": -1, "gigabytes": 1000, "backup_gigabytes": 1000, "volumes_iscsi": -1, "snapshots": 10, "volumes": 10, "backups": 10, "gigabytes_iscsi": -1, "id": "a15e917f3df64fca9784ec2819f1b0cb"}}

HTTP 50x Response

[root@controller-B ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 500 Internal Server Error
Content-Length: 128
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
X-Openstack-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
Date: Thu, 07 Jan 2016 06:46:49 GMT

{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

I've compared output between 2 hosts

- controller-A - Kilo * working *
vs. 
- controller-B - Liberty * non-working *

Both installed via RDO and running CentOS 7.x

strace has some interesting output - but what stood out was the many "(No such file or directory)" messages

Syntax to scrub output

grep -H -a -i 'no such'  /tmp/*.log  | perl -pe 's/.*\(\"(.*)\"\,.*/$1/g' | sort | uniq

Here's the output for comparison

Controller-A http://pastebin.com/vpCQ1gq5

Controller-B http://pastebin.com/zBvYWGKs

Notice the missing "chardet" "netrc" packages in Controller-B

edit flag offensive delete link more

Comments

Really good catch! Have you tried manually installing the netrc stuff? I am not familiar with it myself....but I just checked. In my installation at least, is seems I already have python netrc modules.

theque42 gravatar imagetheque42 ( 2016-01-07 02:34:35 -0500 )edit

@theque42 No I haven't installed "netrc" - I can't find specifically the package for it. I can't also confirm if cinder-api needs it. "python-chardet" is already installed though. My steps were to carry out 'process of elimination' - hoping we can all find an answer soon. Did you follow what I did?

meappy gravatar imagemeappy ( 2016-01-07 17:29:23 -0500 )edit

I actually now have a reason to think that the missing "chardet" and "netrc" libraries are fine.. apparently the messages are present in a working host running Kilo as well

meappy gravatar imagemeappy ( 2016-01-07 23:05:54 -0500 )edit

This is the full strace. From "GET" to response ""HTTP/1.1 500 Internal" - http://pastebin.com/h4F06YSs

meappy gravatar imagemeappy ( 2016-01-08 00:03:58 -0500 )edit
0

answered 2016-01-06 00:28:12 -0500

Bharath gravatar image

If you are trying to perform openstack operations through CLI did you source the openrc file which contains the auth_url and the credentials required to authenticate against keystone? If you don't have those environment variables set you will get this issue. Try souring an openrc file in the below format.

Create and source the OpenStack RC file

Alternatively, you can create the PROJECT-openrc.sh file from scratch, if for some reason you cannot download the file from the dashboard.

In a text editor, create a file named PROJECT-openrc.sh file and add the following authentication information:

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
# The following lines can be omitted
export OS_TENANT_ID=tenantIDString
export OS_REGION_NAME=regionName
export OS_CACERT=/path/to/cacertFile

On any shell from which you want to run OpenStack commands, source the PROJECT-openrc.sh file for the respective project. In this example, you source the admin-openrc.sh file for the admin project:

$ source admin-openrc.sh

For more details, see http://docs.openstack.org/cli-referen...

edit flag offensive delete link more

Comments

Yes the file was sourced

unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=xxx
export OS_AUTH_URL=http://10.1.1.11:5000/v2.0
meappy gravatar imagemeappy ( 2016-01-06 00:30:48 -0500 )edit
1

I am using the standard keystone_admin-rc file that is generated by packstack, that I have used for the last 2 months when working with the openstack installation attempts. Additionally, Please note that it ALSO fails when using Horizon Dashboard. quota-show works FINE for nova and neutron cmd.

theque42 gravatar imagetheque42 ( 2016-01-06 02:53:25 -0500 )edit
1

can you run any nova/cinder command using "--debug" and paste the output?

Bharath gravatar imageBharath ( 2016-01-06 03:13:46 -0500 )edit

http://pastebin.com/KZHFw2J0

theque42 gravatar imagetheque42 ( 2016-01-06 03:36:24 -0500 )edit

cinder-api must be failed to start for a reason that could be observed from cinder-api.log. Please try to find ERROR in the log which could give possible reasons for failure.

Bharath gravatar imageBharath ( 2016-01-06 03:45:08 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

5 followers

Stats

Asked: 2016-01-05 15:28:47 -0500

Seen: 2,860 times

Last updated: Jan 08 '16