Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Hi

As you will know I am experiencing the same issue as OP - this is not a solution yet - just steps I am taking to troubleshoot, and hopefully get to the bottom of this.

Tools used

  1. tcpdump

    tcpdump -s 1024 -i any -tttt -vvvv -A port 8776

  2. strace

    for i in $(ps aux | grep 'cinder-api' | grep -v 'grep' | awk '{print $2}' | tail -n +2);do strace -f -p $i -o /tmp/$i.log & done

Syntax and sample output to test "cinder quota-show" via curl (output may - however syntax should auto-retrieve/populate endpoint and auth token)

HTTP 200 Response

[root@controller-A ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 200 OK
X-Compute-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Content-Type: application/json
Content-Length: 216
X-Openstack-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Date: Thu, 07 Jan 2016 06:39:57 GMT

{"quota_set": {"snapshots_iscsi": -1, "gigabytes": 1000, "backup_gigabytes": 1000, "volumes_iscsi": -1, "snapshots": 10, "volumes": 10, "backups": 10, "gigabytes_iscsi": -1, "id": "a15e917f3df64fca9784ec2819f1b0cb"}}

HTTP 50x Response

[root@controller-B ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 500 Internal Server Error
Content-Length: 128
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
X-Openstack-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
Date: Thu, 07 Jan 2016 06:46:49 GMT

{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

I've compared output between 2 hosts

- controller-A - Kilo * working *
vs. 
- controller-B - Liberty * non-working *

Both installed via RDO and running CentOS 7.x

strace has some interesting output - but what stood out was the many "(No such file or directory)" messages

Syntax to scrub output

grep -H -a -i 'no such'  /tmp/*.log  | perl -pe 's/.*\(\"(.*)\"\,.*/$1/g' | sort | uniq

Here's the output for comparison

Controller-A http://pastebin.com/vpCQ1gq5

Controller-B http://pastebin.com/zBvYWGKs

Notice the missing "chardet" "netrc" packages in Controller-B

Hi

As you will know I am experiencing the same issue as OP - this is not a solution yet - just steps I am taking to troubleshoot, and hopefully get to the bottom of this.

Tools used

  1. tcpdump

    tcpdump -s 1024 -i any -tttt -vvvv -A port 8776

  2. strace

    for i in $(ps aux | grep 'cinder-api' | grep -v 'grep' | awk '{print $2}' | tail -n +2);do strace -f -p $i -o /tmp/$i.log & done

Syntax and sample output to test "cinder quota-show" via curl (output may vary - however syntax should auto-retrieve/populate endpoint and auth token)

HTTP 200 Response

[root@controller-A ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 200 OK
X-Compute-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Content-Type: application/json
Content-Length: 216
X-Openstack-Request-Id: req-0754fd43-d455-46bc-a027-6854e5c8bb46
Date: Thu, 07 Jan 2016 06:39:57 GMT

{"quota_set": {"snapshots_iscsi": -1, "gigabytes": 1000, "backup_gigabytes": 1000, "volumes_iscsi": -1, "snapshots": 10, "volumes": 10, "backups": 10, "gigabytes_iscsi": -1, "id": "a15e917f3df64fca9784ec2819f1b0cb"}}

HTTP 50x Response

[root@controller-B ~(keystone_admin)]# curl -g -i -X GET $(cinder endpoints | grep -A2 -w cinder | grep adminURL | awk '{print $4}')/os-quota-sets/$(openstack project list | grep admin | awk '{print $2}')?usage=False -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $(openstack token issue | grep '\ id\ ' | awk '{print $4}')";echo
HTTP/1.1 500 Internal Server Error
Content-Length: 128
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
X-Openstack-Request-Id: req-ccf41cdf-d14f-4e20-84ac-b69c31441017
Date: Thu, 07 Jan 2016 06:46:49 GMT

{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

I've compared output between 2 hosts

- controller-A - Kilo * working *
vs. 
- controller-B - Liberty * non-working *

Both installed via RDO and running CentOS 7.x

strace has some interesting output - but what stood out was the many "(No such file or directory)" messages

Syntax to scrub output

grep -H -a -i 'no such'  /tmp/*.log  | perl -pe 's/.*\(\"(.*)\"\,.*/$1/g' | sort | uniq

Here's the output for comparison

Controller-A http://pastebin.com/vpCQ1gq5

Controller-B http://pastebin.com/zBvYWGKs

Notice the missing "chardet" "netrc" packages in Controller-B