meappy's profile - activity

I have only 2 NICs. Do you have a sample of how your eth0 and eth1 looks like? As well as br-ex and br-eth1 ?

Hi

I am in the process of setting up RDO

Is there a reference architecture set up for a server with 2 NICs? eth0 + eth1

eth0 == External network (1 public IP) eth1 == Internal network for API calls etc (10.1.1.x)

Appreciate assistance

Confirmed works for me too :) This is interesting, isn't this information already provided in api-paste.init? http://pastebin.com/tJDEqVZa Can someone please explain?

This is the full strace. From "GET" to response ""HTTP/1.1 500 Internal" - http://pastebin.com/h4F06YSs

I actually now have a reason to think that the missing "chardet" and "netrc" libraries are fine.. apparently the messages are present in a working host running Kilo as well

@theque42 No I haven't installed "netrc" - I can't find specifically the package for it. I can't also confirm if cinder-api needs it. "python-chardet" is already installed though. My steps were to carry out 'process of elimination' - hoping we can all find an answer soon. Did you follow what I did?

No this is a known bug/issue - check other thread on steps to reproduce https://ask.openstack.org/en/question/87002/cinder-volume-quotas-broken-for-rdo-install-in-centos/ (https://ask.openstack.org/en/question...)

Hi

As you will know I am experiencing the same issue as OP - this is not a solution yet - just steps I am taking to troubleshoot, and hopefully get to the bottom of this.

Tools used

1. tcpdump

   tcpdump -s 1024 -i any -tttt -vvvv -A port 8776

2. strace

   for i in $(ps aux | grep 'cinder-api' | grep -v 'grep' | awk '{print $2}' | tail -n +2);do strace -f -p $i -o /tmp/$i.log & done

Syntax and sample output to test "cinder quota-show" via curl (output may vary - however syntax should auto-retrieve/populate endpoint and auth token)

HTTP 200 Response

[root@controller-A ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 200 OK
X-Compute-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Content-Type: application/json
Content-Length: 216
X-Openstack-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Date: Thu, 07 Jan 2016 06:39:57 GMT

{"quota_set": {"snapshots_iscsi": -1, "gigabytes": 1000, "backup_gigabytes": 1000, "volumes_iscsi": -1, "snapshots": 10, "volumes": 10, "backups": 10, "gigabytes_iscsi": -1, "id": "a15e917f3df64fca9784ec2819f1b0cb"}}

HTTP 50x Response

[root@controller-B ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 500 Internal Server Error
Content-Length: 128
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
X-Openstack-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
Date: Thu, 07 Jan 2016 06:46:49 GMT

{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

I've compared output between 2 hosts

- controller-A - Kilo * working *
vs. 
- controller-B - Liberty * non-working *

Both installed via RDO and running CentOS 7.x

strace has some interesting output - but what stood out was the many "(No such file or directory)" messages

Syntax to scrub output

grep -H -a -i 'no such'  /tmp/*.log  | perl -pe 's/.*\(\"(.*)\"\,.*/$1/g' | sort | uniq

Here's the output for comparison

Controller-A http://pastebin.com/vpCQ1gq5

Controller-B http://pastebin.com/zBvYWGKs

Notice the missing "chardet" "netrc" packages in Controller-B

Same issue here guys

You can reproduce this with curl - see my command http://pastebin.com/Eh9YyvEq

Same here https://ask.openstack.org/en/question/87002/cinder-volume-quotas-broken-for-rdo-install-in-centos/ (https://ask.openstack.org/en/question...)

Definitely no problems with cinder api starting up

# openstack-service status cinder
MainPID=1478 Id=openstack-cinder-api.service ActiveState=active
MainPID=1476 Id=openstack-cinder-backup.service ActiveState=active
MainPID=1475 Id=openstack-cinder-scheduler.service ActiveState=active
MainPID=1474 Id=openstack-cinder-volume.service ActiveState=active

And as OP has pointed out Caught error: Not enough information to determine URL. Provide either auth_url or endpoint is present in the api.log

# pwd
/var/log/cinder
# tail -n4 api.log
2016-01-07 01:57:07.182 1531 INFO cinder.api.openstack.wsgi [req-10cbabae-a141-4b35-bf1b-9cfc1fc01e66 f49ca938aab64524af5894f4bdabee49 f6aa37e5f3d847feabc7bc4c912680b3 - - -] GET http://10.1.1.11:8776/v2/f6aa37e5f3d847feabc7bc4c912680b3/os-quota-sets/2049fac20c884257b9091101c000ea5f?usage=False
2016-01-07 01:57:07.185 1531 ERROR cinder.api.middleware.fault [req-10cbabae-a141-4b35-bf1b-9cfc1fc01e66 f49ca938aab64524af5894f4bdabee49 f6aa37e5f3d847feabc7bc4c912680b3 - - -] Caught error: Not enough information to determine URL. Provide either auth_url or endpoint
2016-01-07 01:57:07.186 1531 INFO cinder.api.middleware.fault [req-10cbabae-a141-4b35-bf1b-9cfc1fc01e66 f49ca938aab64524af5894f4bdabee49 f6aa37e5f3d847feabc7bc4c912680b3 - - -] http://10.1.1.11:8776/v2/f6aa37e5f3d847feabc7bc4c912680b3/os-quota-sets/2049fac20c884257b9091101c000ea5f?usage=False returned with HTTP 500
2016-01-07 01:57:07.187 1531 INFO eventlet.wsgi.server [req-10cbabae-a141-4b35-bf1b-9cfc1fc01e66 f49ca938aab64524af5894f4bdabee49 f6aa37e5f3d847feabc7bc4c912680b3 - - -] 10.1.1.11 - - [07/Jan/2016 01:57:07] "GET /v2/f6aa37e5f3d847feabc7bc4c912680b3/os-quota-sets/2049fac20c884257b9091101c000ea5f?usage=False HTTP/1.1" 500 425

With debug mode turned on is pretty much useless...

# cinder -d  quota-show  2049fac20c884257b9091101c000ea5f
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://10.1.1.11:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] content-length: 223 content-encoding: gzip vary: X-Auth-Token,Accept-Encoding server: Apache/2.4.6 (CentOS) connection: close date: Wed, 06 Jan 2016 15:00:22 GMT content-type: application/json x-openstack-request-id: req-22563ed2-8c01-4410-8309-3908fc5e202f 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.1.1.11:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.1.1.11:5000/v2.0/tokens
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://10.1.1.11:8776/v2/f6aa37e5f3d847feabc7bc4c912680b3/os-quota-sets/2049fac20c884257b9091101c000ea5f?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}5a06955238ec80da0c5117ef19b58d162fff8bce"
DEBUG:keystoneclient.session:RESP: [500] content-length: 128 x-compute-request-id: req-086830d6-c193-4302-9590-f1630141a43a connection: keep-alive date: Wed, 06 Jan 2016 15:00:23 GMT content-type: application/json; charset=UTF-8 x-openstack-request-id: req-086830d6-c193-4302-9590-f1630141a43a 
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-086830d6-c193-4302-9590-f1630141a43a)

0.3312410.335274

Yes the file was sourced

unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=xxx
export OS_AUTH_URL=http://10.1.1.11:5000/v2.0

Same error here