Ask Your Question
1

Complaint of conflicting resource provider name when old one is deleted.

asked 2019-03-14 19:57:22 -0500

bnordgren gravatar image

We're still cleaning up the mess from our upgrade from Queens to Rocky. Our current version is openstack-ansible 18.1.4.

This little nugget of joy is appearing in the nova-compute.log on two compute nodes:

Failed to create resource provider record in placement API for UUID 1b1d7963-d96b-4797-b7f4-32e1b0251c9b. Got 409: {"errors": [{"status": 409, "request_id": "req-635c6d1d-4dd0-4e58-abe3-da4b7ad5a0eb", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: compute-7.<something> already exists.  ", "title": "Conflict"}]}.

The placement api server has this in it's log, which was matched by UUID:

2019-03-14 17:03:06.922 107 INFO nova.api.openstack.placement.requestlog [req-dbcac0f2-24ee-4328-af32-33ab49c748af 5a5285ffa0b7462d829e0ad74044c0e1 1441821ca0f34bc6a5b4a2c8d30c977c - default default] 192.168.58.10 "GET /resource_providers/1b1d7963-d96b-4797-b7f4-32e1b0251c9b/allocations" status: 404 len: 303 microversion: 1.0

Looking at the database, I see that the two problem children (compute-7 and compute-8) have been deleted in the past and were re-created. Note that the compute node is trying-but-failing to update the resource tally on the correct (not deleted) uuid:

MariaDB [nova]> select id, host, hypervisor_hostname, created_at, deleted_at, deleted, uuid from compute_nodes order by host ;
+----+------------+----------------------------+---------------------+---------------------+---------+--------------------------------------+
| id | host       | hypervisor_hostname        | created_at          | deleted_at          | deleted | uuid                                 |
+----+------------+----------------------------+---------------------+---------------------+---------+--------------------------------------+
| 31 | compute-7  | compute-7.<something>      | 2017-05-01 23:27:10 | 2018-04-06 18:12:37 |      31 | 918e356d-fc0d-45d4-85fb-acdf5a008912 |
| 49 | compute-7  | compute-7.<something>      | 2018-10-17 20:16:17 | NULL                |       0 | 1b1d7963-d96b-4797-b7f4-32e1b0251c9b |
| 29 | compute-8  | compute-8.<something>      | 2017-05-01 23:26:49 | 2018-04-06 18:18:31 |      29 | 0a4089bd-e247-451c-bafc-229c9d492750 |
| 58 | compute-8  | compute-8.<something>      | 2018-10-17 20:16:18 | NULL                |       0 | 92c2c0ff-2246-4f76-8c24-2903c72b8945 |
+----+------------+----------------------------+---------------------+---------------------+---------+--------------------------------------+
17 rows in set (0.00 sec)

openstack hypervisor list and nova-manage cell_v2 list_hosts both report only a single entry for both hosts, and in the case of the openstack command, the correct (current) id is returned. And yet, there is something wrong with the placement api, which prevents updating the resource tally, which prevents deploying VMs to these two machines.

How do I address this situation? I'm hesitant to start deleting things out of databases because I don't want to create inconsistency. I'm also hesitant to delete the hosts with the command line tools, because won't that just add yet another deleted entry to the nova compute_nodes table?

Thanks in advance for your help!

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-03-15 04:45:48 -0500

Rather than messing around in the database, use the Placement CLI to correct the content of the Placement service.

The website says that you have to install the OpenStack client Placement plugin manually, but check first - perhaps it's already included.

edit flag offensive delete link more

Comments

This got me farther. Deleting the resource providers associated with the old UUIDs allowed the correct, current entries to be created. Unfortunately, now Horizon errors with: "Unable to retrieve instance list" whenever the list would contain a VM placed on one of those nodes...

bnordgren gravatar imagebnordgren ( 2019-03-15 14:15:15 -0500 )edit

This may or may not be related to your earlier Placement problem. Now it's time to dive into the log files. Start with the Nova API log, also see if there is anything in the corresponding Nova Compute logs.

Bernd Bausch gravatar imageBernd Bausch ( 2019-03-15 17:39:20 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-03-14 19:57:22 -0500

Seen: 239 times

Last updated: Mar 15