Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Any known nova keypair deletion bugs?

I am running icehouse openstack and using a heat stack to create resources, including a keypair. I have seen several times now where the nova keypair still exists after the heat stack is deleted.

I am wondering if this is a known bug, or if I should create a bug to track this?

I have looked in the nova-api.log file and see the following from my attempt to delete the keypair:

2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Matched DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100 2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs/:(id)', defaults: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102 2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Match dict: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff', 'id': u'isc01_InternalKey'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103 2014-08-15 07:35:46.559 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method="" keypaircontroller.delete="" of="" <nova.api.openstack.compute.contrib.keypairs.keypaircontroller="" object="" at="" 0x7f45d80e6250="">>' (Content-type='None', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945 2014-08-15 07:35:46.563 3115 INFO nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] HTTP exception thrown: The resource could not be found. 2014-08-15 07:35:46.564 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215 2014-08-15 07:35:46.564 3115 INFO nova.osapi_compute.wsgi.server [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] 135.1.194.1 "DELETE /v2/e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey HTTP/1.1" status: 404 len: 272 time: 0.0090990 2014-08-15 07:35:49.657 3103 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

yet...when I turn around and try to create the same keypair, it fails..

2014-08-15 07:41:57.781 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs', defaults: {'action': u'create', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102 2014-08-15 07:41:57.782 3115 DEBUG routes.middleware [-] Match dict: {'action': u'create', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103 2014-08-15 07:41:57.782 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Action: 'create', body: {"keypair": {"name": "isc01_InternalKey"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:940 2014-08-15 07:41:57.783 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method="" keypaircontroller.create="" of="" <nova.api.openstack.compute.contrib.keypairs.keypaircontroller="" object="" at="" 0x7f45d80e6250="">>' (Content-type='application/json', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945 2014-08-15 07:41:57.800 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -N -t rsa -f /tmp/tmpXNkb6L/temp -C Generated by Nova execute /usr/lib/python2.7/dist-packages/no va/openstack/common/processutils.py:154 2014-08-15 07:41:57.985 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187 2014-08-15 07:41:57.986 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -l -f /tmp/tmpXNkb6L/temp.pub execute /usr/lib/python2.7/dist-packages/nova/openstack/common/proce ssutils.py:154 2014-08-15 07:41:57.998 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187 2014-08-15 07:41:58.002 3115 INFO nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] HTTP exception thrown: Key pair 'isc01_InternalKey' already exists. 2014-08-15 07:41:58.003 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Returning 409 to user: Key pair 'isc01_InternalKey' already exists. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215

click to hide/show revision 2
No.2 Revision

Any known nova keypair deletion bugs?

I am running icehouse openstack and using a heat stack to create resources, including a keypair. I have seen several times now where the nova keypair still exists after the heat stack is deleted.

I am wondering if this is a known bug, or if I should create a bug to track this?

I have looked in the nova-api.log file and see the following from my attempt to delete the keypair:

2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Matched DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey **DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey** __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs/:(id)', defaults: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">} <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Match dict: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">, <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff', 'id': u'isc01_InternalKey'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:35:46.559 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method="" keypaircontroller.delete="" of="" <nova.api.openstack.compute.contrib.keypairs.keypaircontroller="" object="" at="" 0x7f45d80e6250="">>' method KeypairController.delete of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>' 
(Content-type='None', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:35:46.563 3115 INFO nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] HTTP **HTTP exception thrown: The resource could not be found. found.**
2014-08-15 07:35:46.564 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215
2014-08-15 07:35:46.564 3115 INFO nova.osapi_compute.wsgi.server [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] 135.1.194.1 "DELETE /v2/e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey HTTP/1.1" status: 404 len: 272 time: 0.0090990
2014-08-15 07:35:49.657 3103 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

yet...when I turn around and try to create the same keypair, it fails..

2014-08-15 07:41:57.781 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs', defaults: {'action': u'create', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">} <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:41:57.782 3115 DEBUG routes.middleware [-] Match dict: {'action': u'create', 'controller': <nova.api.openstack.wsgi.resource object="" at="" 0x7f45d7f67790="">, <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:41:57.782 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Action: **Action: 'create', body: {"keypair": {"name": "isc01_InternalKey"}} "isc01_InternalKey"}}** _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:940
2014-08-15 07:41:57.783 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 **99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method="" keypaircontroller.create="" of="" <nova.api.openstack.compute.contrib.keypairs.keypaircontroller="" object="" at="" 0x7f45d80e6250="">>' method KeypairController.create of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>'** 
(Content-type='application/json', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:41:57.800 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -N  -t rsa -f /tmp/tmpXNkb6L/temp -C Generated by Nova execute /usr/lib/python2.7/dist-packages/no
va/openstack/common/processutils.py:154
2014-08-15 07:41:57.985 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:57.986 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -l -f /tmp/tmpXNkb6L/temp.pub execute /usr/lib/python2.7/dist-packages/nova/openstack/common/proce
ssutils.py:154
2014-08-15 07:41:57.998 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:58.002 3115 INFO nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] HTTP **HTTP exception thrown: Key pair 'isc01_InternalKey' already exists. exists.**
2014-08-15 07:41:58.003 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Returning 409 to user: Key pair 'isc01_InternalKey' already exists. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215

/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215

Any known nova keypair deletion bugs?

I am running icehouse openstack and using a heat stack to create resources, including a keypair. I have seen several times now where the nova keypair still exists after the heat stack is deleted.

I am wondering if this is a known bug, or if I should create a bug to track this?

I have looked in the nova-api.log file and see the following from my attempt to delete the keypair:

2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Matched **DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey** __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs/:(id)', defaults: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Match dict: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff', 'id': u'isc01_InternalKey'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:35:46.559 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.delete of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>' 
(Content-type='None', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:35:46.563 3115 INFO nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] **HTTP exception thrown: The resource could not be found.**
2014-08-15 07:35:46.564 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215
2014-08-15 07:35:46.564 3115 INFO nova.osapi_compute.wsgi.server [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] 135.1.194.1 "DELETE /v2/e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey HTTP/1.1" status: 404 len: 272 time: 0.0090990
2014-08-15 07:35:49.657 3103 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

yet...when I turn around and try to create the same keypair, it fails..

2014-08-15 07:41:57.781 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs', defaults: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:41:57.782 3115 DEBUG routes.middleware [-] Match dict: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:41:57.782 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **Action: 'create', body: {"keypair": {"name": "isc01_InternalKey"}}** _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:940
2014-08-15 07:41:57.783 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 **99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.create of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>'** 
(Content-type='application/json', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:41:57.800 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -N  -t rsa -f /tmp/tmpXNkb6L/temp -C Generated by Nova execute /usr/lib/python2.7/dist-packages/no
va/openstack/common/processutils.py:154
2014-08-15 07:41:57.985 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:57.986 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -l -f /tmp/tmpXNkb6L/temp.pub execute /usr/lib/python2.7/dist-packages/nova/openstack/common/proce
ssutils.py:154
2014-08-15 07:41:57.998 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:58.002 3115 INFO nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **HTTP exception thrown: Key pair 'isc01_InternalKey' already exists.**
2014-08-15 07:41:58.003 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Returning 409 to user: Key pair 'isc01_InternalKey' already exists. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215

Any Are there any known nova keypair deletion bugs?bugs where the keypair exists, but it is not found when trying to delete it??

I am running icehouse openstack and using a heat stack to create resources, including a keypair. I have seen several times now where the nova keypair still exists after the heat stack is deleted.deleted. If I repeatedly create/delete the stack, the keypair is deleted most of the time, but occasionally I find that the keypair does not get deleted. When this happens, I see the messages I have included below.

I am wondering if anyone has encountered this is a known bug, behavior as well? Is there an existing bug report for this or if I should I create a new bug to track this?report.

I have looked in the nova-api.log file and see the following from my attempt to delete the keypair:keypair named 'isc01_InternalKey'...and below that segment is another where I tried to recreate the heat stack and it shows a failure to create a keypair with the same name ('isc01_InternalKey') with the error that the same key already exists:

2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Matched **DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey** __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs/:(id)', defaults: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Match dict: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff', 'id': u'isc01_InternalKey'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:35:46.559 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.delete of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>' 
(Content-type='None', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:35:46.563 3115 INFO nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] **HTTP ****HTTP exception thrown: The resource could not be found.**
found.****
2014-08-15 07:35:46.564 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215
2014-08-15 07:35:46.564 3115 INFO nova.osapi_compute.wsgi.server [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] 135.1.194.1 "DELETE /v2/e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey HTTP/1.1" status: 404 len: 272 time: 0.0090990
2014-08-15 07:35:49.657 3103 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

yet...when I turn around and try to create the same keypair, it fails..

2014-08-15 07:41:57.781 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs', defaults: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:41:57.782 3115 DEBUG routes.middleware [-] Match dict: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:41:57.782 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **Action: 'create', body: {"keypair": {"name": "isc01_InternalKey"}}** _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:940
2014-08-15 07:41:57.783 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 **99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.create of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>'** 
(Content-type='application/json', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:41:57.800 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -N  -t rsa -f /tmp/tmpXNkb6L/temp -C Generated by Nova execute /usr/lib/python2.7/dist-packages/no
va/openstack/common/processutils.py:154
2014-08-15 07:41:57.985 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:57.986 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -l -f /tmp/tmpXNkb6L/temp.pub execute /usr/lib/python2.7/dist-packages/nova/openstack/common/proce
ssutils.py:154
2014-08-15 07:41:57.998 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:58.002 3115 INFO nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **HTTP exception thrown: Key pair 'isc01_InternalKey' already exists.**
2014-08-15 07:41:58.003 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Returning 409 to user: Key pair 'isc01_InternalKey' already exists. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215

Are there any known nova keypair deletion bugs where the keypair exists, Keypair is created with Heat stack but it is sometimes not found when trying to delete it??deleted

I am running icehouse openstack and using a heat stack to create resources, including a keypair. I have seen several times now where the nova keypair still exists after the heat stack is deleted. If I repeatedly create/delete the stack, the keypair is deleted most of the time, but occasionally I find that the keypair does not get deleted. When this happens, I see the messages I have included below.

I am wondering if anyone has encountered this behavior as well? Is there an existing bug report for this or should I create a new bug report.

I have looked in the nova-api.log file and see the following from my attempt to delete the keypair named 'isc01_InternalKey'...and below that segment is another where I tried to recreate the heat stack and it shows a failure to create a keypair with the same name ('isc01_InternalKey') with the error that the same key already exists:

2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Matched **DELETE /e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey** __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs/:(id)', defaults: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:35:46.559 3115 DEBUG routes.middleware [-] Match dict: {'action': u'delete', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff', 'id': u'isc01_InternalKey'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:35:46.559 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.delete of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>' 
(Content-type='None', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:35:46.563 3115 INFO nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] ****HTTP exception thrown: The resource could not be found.****
2014-08-15 07:35:46.564 3115 DEBUG nova.api.openstack.wsgi [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] Returning 404 to user: The resource could not be found. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215
2014-08-15 07:35:46.564 3115 INFO nova.osapi_compute.wsgi.server [req-d7eaab6b-0e7d-41d1-9dbe-e68849129ac6 53162a8bc5fe481d9aa5d2a40c1a6ed2 e2fa6ebdad824e839421e9cfb4089bff] 135.1.194.1 "DELETE /v2/e2fa6ebdad824e839421e9cfb4089bff/os-keypairs/isc01_InternalKey HTTP/1.1" status: 404 len: 272 time: 0.0090990
2014-08-15 07:35:49.657 3103 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

yet...when I turn around and try to create the same keypair, it fails..

2014-08-15 07:41:57.781 3115 DEBUG routes.middleware [-] Route path: '/{project_id}/os-keypairs', defaults: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2014-08-15 07:41:57.782 3115 DEBUG routes.middleware [-] Match dict: {'action': u'create', 'controller': <nova.api.openstack.wsgi.Resource object at 0x7f45d7f67790>, 'project_id': u'e2fa6ebdad824e839421e9cfb4089bff'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2014-08-15 07:41:57.782 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **Action: 'create', body: {"keypair": {"name": "isc01_InternalKey"}}** _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:940
2014-08-15 07:41:57.783 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 **99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Calling method '<bound method KeypairController.create of <nova.api.openstack.compute.contrib.keypairs.KeypairController object at 0x7f45d80e6250>>'** 
(Content-type='application/json', Accept='application/json') _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:945
2014-08-15 07:41:57.800 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -N  -t rsa -f /tmp/tmpXNkb6L/temp -C Generated by Nova execute /usr/lib/python2.7/dist-packages/no
va/openstack/common/processutils.py:154
2014-08-15 07:41:57.985 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:57.986 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Running cmd (subprocess): ssh-keygen -q -l -f /tmp/tmpXNkb6L/temp.pub execute /usr/lib/python2.7/dist-packages/nova/openstack/common/proce
ssutils.py:154
2014-08-15 07:41:57.998 3115 DEBUG nova.openstack.common.processutils [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:187
2014-08-15 07:41:58.002 3115 INFO nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] **HTTP exception thrown: Key pair 'isc01_InternalKey' already exists.**
2014-08-15 07:41:58.003 3115 DEBUG nova.api.openstack.wsgi [req-8f58ad48-238f-4126-bd35-a7894b215fd3 99a5e6e215074228889439212bced1c2 e2fa6ebdad824e839421e9cfb4089bff] Returning 409 to user: Key pair 'isc01_InternalKey' already exists. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1215