Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

2 second delay on OpenStack API

Having deployed a cloud using Tripleo, either Mitaka or Newton release, listing endpoints is slow. E.g.

[stack@undercloud ~]$ openstack endpoint list --timing +----------------------------------+-----------+--------------+----------------+ | ID | Region | Service Name | Service Type | +----------------------------------+-----------+--------------+----------------+ | 01d1f6098a9e4c4c874ac662e185f0b8 | regionOne | gnocchi | metric | | 63ec0d3d574444da96537769a447a70a | regionOne | heat | orchestration | | 376239edd44e4a5a9d08acf9b2034ada | regionOne | aodh | alarming | | cadc28d3dd1d4772a395ac12be61c7dc | regionOne | heat-cfn | cloudformation | | d8721e9e6e464463aacd35b39fbb5637 | regionOne | neutron | network | | d84984b938cf4ebbb481f9213faaac32 | regionOne | swift | object-store | | 71a53a84dc914ba6bea13366b00cceb5 | regionOne | cinderv2 | volumev2 | | 1c3dc22bb1a1478d8e8ae69216d908d0 | regionOne | cinderv3 | volumev3 | | aec55ba83cdd48509c7ec588259407e0 | regionOne | glance | image | | 660f840e9a874ab08a1bf1c14856570a | regionOne | ceilometer | metering | | baf1881f32ce467bbc8423abf0459481 | regionOne | nova | compute | | 54a8aed0cf9d4f9689d130370b9635de | regionOne | cinder | volume | | a0d74959916a4570af3a5e9258cf3c3e | regionOne | keystone | identity | +----------------------------------+-----------+--------------+----------------+

+-----------------------------------------------------------------------------------+-----------+ | URL | Seconds | +-----------------------------------------------------------------------------------+-----------+ | GET http://10.0.0.8:5000/v2.0 | 2.194681 | | POST http://10.0.0.8:5000/v2.0/tokens | 2.44672 | | POST http://10.0.0.8:5000/v2.0/tokens | 2.460463 | | GET http://10.0.0.8:35357/ | 0.007647 | | GET http://10.0.0.8:35357/v2.0/endpoints | 0.037036 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/6e57018bd8ba44d7b85a40bb29d792d7 | 0.025002 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/70d296346051444e987f29452e3d56b9 | 0.118714 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d50df18f2db94df9bd613e27fc57e6cc | 0.03642 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/549f7a4c1b13420fb7c08c2af9a07fe9 | 2.3991 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/ae944096e3664e92a7d5adf0c9b79878 | 0.118197 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/45a031be8ddf43e1b9add5c4ca624a64 | 0.0449 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d1beac8dab464f6992de9d3bf483b5a2 | 0.034414 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/4c0d3e812329468e9ebcbbb6f8958858 | 0.026568 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d6be081615a247a3a5c09f94b6062030 | 0.032789 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/42858357f6e340029bbd7eeabc9771fe | 0.140968 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/12883d88d9be4a6ca546f7134e9855db | 0.130114 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de0d004e184f462c87ad45ff77d610e8 | 2.36879 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de543a84e1184a45bce76786d4cf4916 | 0.026789 | | Total | 12.649312 | +-----------------------------------------------------------------------------------+-----------+

Some points to note

  1. We have tried various switch alterations (MTU, firmware, STP) but having reduced this down to a single controller and querying its own VIP, I'm assuming this doesn't even get to the switch.
  2. This only had 1 second delays with RHEL OSP 9
  3. Other installations take under a second to return a list of endpoints
  4. The problem seems to ease after a few hours, perhaps indicating a parameter/limit is set too low initially.

Any suggestions appreciated.

2 second delay on OpenStack API

Having deployed a cloud using Tripleo, either Mitaka or Newton release, listing endpoints is slow. E.g.

[stack@undercloud ~]$ openstack endpoint list --timing
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| 01d1f6098a9e4c4c874ac662e185f0b8 | regionOne | gnocchi      | metric         |
| 63ec0d3d574444da96537769a447a70a | regionOne | heat         | orchestration  |
| 376239edd44e4a5a9d08acf9b2034ada | regionOne | aodh         | alarming       |
| cadc28d3dd1d4772a395ac12be61c7dc | regionOne | heat-cfn     | cloudformation |
| d8721e9e6e464463aacd35b39fbb5637 | regionOne | neutron      | network        |
| d84984b938cf4ebbb481f9213faaac32 | regionOne | swift        | object-store   |
| 71a53a84dc914ba6bea13366b00cceb5 | regionOne | cinderv2     | volumev2       |
| 1c3dc22bb1a1478d8e8ae69216d908d0 | regionOne | cinderv3     | volumev3       |
| aec55ba83cdd48509c7ec588259407e0 | regionOne | glance       | image          |
| 660f840e9a874ab08a1bf1c14856570a | regionOne | ceilometer   | metering       |
| baf1881f32ce467bbc8423abf0459481 | regionOne | nova         | compute        |
| 54a8aed0cf9d4f9689d130370b9635de | regionOne | cinder       | volume         |
| a0d74959916a4570af3a5e9258cf3c3e | regionOne | keystone     | identity       |
+----------------------------------+-----------+--------------+----------------+

+----------------------------------+-----------+--------------+----------------+ +-----------------------------------------------------------------------------------+-----------+ | URL | Seconds | +-----------------------------------------------------------------------------------+-----------+ | GET http://10.0.0.8:5000/v2.0 | 2.194681 | | POST http://10.0.0.8:5000/v2.0/tokens | 2.44672 | | POST http://10.0.0.8:5000/v2.0/tokens | 2.460463 | | GET http://10.0.0.8:35357/ | 0.007647 | | GET http://10.0.0.8:35357/v2.0/endpoints | 0.037036 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/6e57018bd8ba44d7b85a40bb29d792d7 | 0.025002 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/70d296346051444e987f29452e3d56b9 | 0.118714 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d50df18f2db94df9bd613e27fc57e6cc | 0.03642 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/549f7a4c1b13420fb7c08c2af9a07fe9 | 2.3991 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/ae944096e3664e92a7d5adf0c9b79878 | 0.118197 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/45a031be8ddf43e1b9add5c4ca624a64 | 0.0449 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d1beac8dab464f6992de9d3bf483b5a2 | 0.034414 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/4c0d3e812329468e9ebcbbb6f8958858 | 0.026568 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d6be081615a247a3a5c09f94b6062030 | 0.032789 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/42858357f6e340029bbd7eeabc9771fe | 0.140968 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/12883d88d9be4a6ca546f7134e9855db | 0.130114 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de0d004e184f462c87ad45ff77d610e8 | 2.36879 | | GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de543a84e1184a45bce76786d4cf4916 | 0.026789 | | Total | 12.649312 | +-----------------------------------------------------------------------------------+-----------+

+-----------------------------------------------------------------------------------+-----------+

Some points to note

  1. We have tried various switch alterations (MTU, firmware, STP) but having reduced this down to a single controller and querying its own VIP, I'm assuming this doesn't even get to the switch.
  2. This only had 1 second delays with RHEL OSP 9
  3. Other installations take under a second to return a list of endpoints
  4. The problem seems to ease after a few hours, perhaps indicating a parameter/limit is set too low initially.

Any suggestions appreciated.