2 second delay on OpenStack API

asked 2016-11-29 02:57:10 -0500

snecklifter gravatar image

updated 2016-12-02 09:16:29 -0500

rbowen gravatar image

Having deployed a cloud using Tripleo, either Mitaka or Newton release, listing endpoints is slow. E.g.

[stack@undercloud ~]$ openstack endpoint list --timing
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| 01d1f6098a9e4c4c874ac662e185f0b8 | regionOne | gnocchi      | metric         |
| 63ec0d3d574444da96537769a447a70a | regionOne | heat         | orchestration  |
| 376239edd44e4a5a9d08acf9b2034ada | regionOne | aodh         | alarming       |
| cadc28d3dd1d4772a395ac12be61c7dc | regionOne | heat-cfn     | cloudformation |
| d8721e9e6e464463aacd35b39fbb5637 | regionOne | neutron      | network        |
| d84984b938cf4ebbb481f9213faaac32 | regionOne | swift        | object-store   |
| 71a53a84dc914ba6bea13366b00cceb5 | regionOne | cinderv2     | volumev2       |
| 1c3dc22bb1a1478d8e8ae69216d908d0 | regionOne | cinderv3     | volumev3       |
| aec55ba83cdd48509c7ec588259407e0 | regionOne | glance       | image          |
| 660f840e9a874ab08a1bf1c14856570a | regionOne | ceilometer   | metering       |
| baf1881f32ce467bbc8423abf0459481 | regionOne | nova         | compute        |
| 54a8aed0cf9d4f9689d130370b9635de | regionOne | cinder       | volume         |
| a0d74959916a4570af3a5e9258cf3c3e | regionOne | keystone     | identity       |
+----------------------------------+-----------+--------------+----------------+

+-----------------------------------------------------------------------------------+-----------+
| URL                                                                               |   Seconds |
+-----------------------------------------------------------------------------------+-----------+
| GET http://10.0.0.8:5000/v2.0                                                     |  2.194681 |
| POST http://10.0.0.8:5000/v2.0/tokens                                             |   2.44672 |
| POST http://10.0.0.8:5000/v2.0/tokens                                             |  2.460463 |
| GET http://10.0.0.8:35357/                                                        |  0.007647 |
| GET http://10.0.0.8:35357/v2.0/endpoints                                          |  0.037036 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/6e57018bd8ba44d7b85a40bb29d792d7 |  0.025002 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/70d296346051444e987f29452e3d56b9 |  0.118714 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d50df18f2db94df9bd613e27fc57e6cc |   0.03642 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/549f7a4c1b13420fb7c08c2af9a07fe9 |    2.3991 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/ae944096e3664e92a7d5adf0c9b79878 |  0.118197 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/45a031be8ddf43e1b9add5c4ca624a64 |    0.0449 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d1beac8dab464f6992de9d3bf483b5a2 |  0.034414 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/4c0d3e812329468e9ebcbbb6f8958858 |  0.026568 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d6be081615a247a3a5c09f94b6062030 |  0.032789 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/42858357f6e340029bbd7eeabc9771fe |  0.140968 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/12883d88d9be4a6ca546f7134e9855db |  0.130114 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de0d004e184f462c87ad45ff77d610e8 |   2.36879 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de543a84e1184a45bce76786d4cf4916 |  0.026789 |
| Total                                                                             | 12.649312 |
+-----------------------------------------------------------------------------------+-----------+

Some points to note

  1. We have tried various switch alterations (MTU, firmware, STP) but having reduced this down to a single controller and querying its own VIP, I'm assuming this doesn't even get to the switch.
  2. This only had 1 second delays with RHEL OSP 9
  3. Other installations take under a second to return a list of endpoints
  4. The problem seems to ease after a few hours, perhaps indicating a parameter/limit is set too low initially.

Any suggestions appreciated.

edit retag flag offensive close merge delete

Comments

Few thoughts: -token (more probable)-database operations You can probably try to exclude token by issuing token and then using that You can activate slow query logging in my.cnf To understand RHEL vs. your current behaviour - compare keystone.conf and my.cnf +DEBUG logging in Keystone/check logs

volenbovsky gravatar imagevolenbovsky ( 2016-11-29 05:35:55 -0500 )edit