snapshot upload fails

asked 2017-02-17 05:45:47 -0600

rl_person gravatar image

I am seeing this in Liberty, I believe - https://bugs.launchpad.net/glance/+bu...

I am taking a snapshot of a shutdown instance.

I have raised the [token] expiration=18000 to 5hrs in keystone.conf but alas, the glance registry raises errors, in this example the snapshot started 17:02, so is failing in much less time.

The image appears in the dashboard, it's ~165GB. Then it fails to upload to glance, wsgi deletes it from the compute node, so I can't even manually use it to upload to glance.

Key extracts from the logs.

2017-02-16 18:10:29.149 20830 INFO eventlet.wsgi.server [req-c574edbb-2bae-4e63-a3d8-58ffab18ccca aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] <IPCONTROLLER> - - [16/Feb/2017 18:10:29] "HEAD /v1/images/338006fe-d7ec-4cc8-925b-3eaf8f3669f4 HTTP/1.1" 200 1568 0.018436
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils [req-18170cce-3806-4d8e-8499-f339a13c1541 aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] Received HTTP error while uploading image 338006fe-d7ec-4cc8-925b-3eaf8f3669f4
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils Traceback (most recent call last):
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils   File "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line 180, in upload_data_to_store
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils     raise webob.exc.HTTPUnauthorized(explanation=e.msg, request=req)
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils HTTPUnauthorized: Authentication required
2017-02-16 18:10:31.974 20837 ERROR glance.api.v1.upload_utils
2017-02-16 18:10:32.005 20837 ERROR glance.registry.client.v1.client [req-18170cce-3806-4d8e-8499-f339a13c1541 aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] Registry client request PUT /images/338006fe-d7ec-4cc8-925b-3eaf8f3669f4 raised NotAuthenticated
...
2017-02-16 18:10:32.006 20837 ERROR glance.api.v1.upload_utils [req-18170cce-3806-4d8e-8499-f339a13c1541 aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] Unable to kill image 338006fe-d7ec-4cc8-925b-3eaf8f3669f4:
...
2017-02-16 18:10:32.013 20837 INFO eventlet.wsgi.server [req-18170cce-3806-4d8e-8499-f339a13c1541 aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] <IPCOMPUTE> - - [16/Feb/2017 18:10:32] "PUT /v1/images/338006fe-d7ec-4cc8-925b-3eaf8f3669f4 HTTP/1.1" 401 425 1357.920412
2017-02-16 18:10:34.199 20836 INFO eventlet.wsgi.server [req-ccb50a36-9d72-400e-a76b-38ac34989ae3 aeb79410ffcf4a37a036336c28d23b62 f0bfca4e0a054f56853a1f45566ffdd3 - - -] <IPCONTROLLER> - - [16/Feb/2017 18:10:34] "HEAD /v1/images/338006fe-d7ec-4cc8-925b-3eaf8f3669f4 HTTP/1.1" 200 1568 0.032151
2017-02-16 18:10:36.570 20830 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2017-02-16 18:10:36.570 20830 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: 3fd6d9837ed24d9daae840c37b57a56a", "code": 404, "title": "Not Found"}}
2017-02-16 18:10:36.571 20830 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2017-02-16 18:10:36.574 20830 INFO eventlet.wsgi.server [-] <IPCOMPUTE> - - [16/Feb/2017 18:10:36] "DELETE /v1/images/338006fe-d7ec-4cc8-925b-3eaf8f3669f4 HTTP/1.1" 401 566 0.030480

keystone.conf

[DEFAULT]
admin_token = ADMIN_TOKEN
debug = false
verbose = false
log_dir = /var/log/keystone
[database]
connection = mysql+pymysql://keystone:KEYSTONEDB_PASS@CONTROLLER/keystone
[memcache]
servers = localhost:11211
[revoke]
driver = sql
[token]
provider = uuid
driver = memcache
expiration = 18000
[extra_headers]
Distribution = Ubuntu

Any help appreciated. Thank you.

edit retag flag offensive close merge delete

Comments

No real answer to this - despite restarting nova glance and keystone services it never worked - restarted the controller node and that took affect.

Thinking about - maybe restart rabbitmq

Who knows?

rl_person gravatar imagerl_person ( 2017-02-17 10:42:58 -0600 )edit