Ask Your Question
0

Slow control path (API completion time)

asked 2015-10-07 20:48:20 -0500

Nodir gravatar image

I'd like to understand OpenStack control path performance and get some suggestions to it faster.

I am facing issue where subnet attachment to the router takes a long time, order of 10 seconds. I am creating 30 networks with one subnet each (30 subnets total), then I am attaching [1] all of these subnets to one router which is already attached to the external network. Here, initial 7-8 subnets get attached quickly, less than a second, but after 15-20 subnets API completion takes really long, ~10 seconds.

I analyzed logs at neutron-server.log (at the network node) to understand bottlenecks. I am using Kilo with DVR, OVS and GRE. I have 1 network node (host name netnode) and 1 compute node (host name compnode1). Here is a timeline and events I see relevant to the port which got created and is used to attach the subnet to the router (replaced port UUID 980213a0-f07c-4773-a154-a32ab6e2b269 with "UUID" for brevity):

15:22:22.409 neutron.db.db_base_plugin_v2 -- Allocated IP 10.0.1.1 for port UUID 
15:22:25.752 neutron.plugins.ml2.db -- No binding for DVR port UUID with host compnode1 get_dvr_port_binding_by_host 15:22:25.823 neutron.plugins.ml2.db -- For port UUID host compnode1, got binding levels 
15:22:25.875 neutron.plugins.ml2.managers -- Attempting to bind port UUID on host host0 for vnic_type normal with profile  bind_port 
15:22:25.882 neutron.plugins.ml2.drivers.mech_agent -- Attempting to bind port UUID on network 
15:22:25.892 neutron.plugins.ml2.db -- No binding for DVR port UUID with host netnode get_dvr_port_binding_by_host  15:22:26.021 neutron.plugins.ml2.db -- For port UUID, host netnode, got binding levels [] get_binding_levels  
15:22:26.172 neutron.plugins.ml2.managers -- Bound port: UUID, host: compnode1, vif_type: ovs, vif_details:
15:22:26.210 neutron.plugins.ml2.db -- For port UUID, host compnode1, cleared binding levels clear_binding_levels
15:22:26.215 neutron.plugins.ml2.db -- For port UUID, host compnode1, set binding levels  
15:22:26.228 neutron.plugins.ml2.managers -- Attempting to bind port UUID on host netnode for vnic_type normal with profile  bind_port  
15:22:26.230 neutron.plugins.ml2.managers -- Attempting to bind port UUID host netnode at level 0 using segments [{'segmentation_id': 2L, 'physical_network': None, 'id': u'd73b88e f-41cb-41fe-88d8-5f6e48df2470', 'network_type': u'gre'}]
_bind_port_level  15:22:26.232 neutron.plugins.ml2.drivers.mech_agent
-- Attempting to bind port UUID on network 5433b880-077d-4d7f-a799-03f3ea7ccd7f bind_port  
15:22:26.270 neutron.plugins.ml2.managers -- Bound port: UUID, host: netnode, vif_type: ovs, vif_details
15:22:26.316 neutron.plugins.ml2.db -- For port UUID, host netnode, cleared binding levels clear_binding_levels
15:22:26.320 neutron.plugins.ml2.db -- For port UUID, host netnode, set binding levels 
15:22:26.382 neutron.plugins.ml2.drivers.l2pop.rpc -- Fanout notify l2population agents at q-agent-notifier the message remove_fdb_entries 
15:22:26.405 neutron.api.rpc.handlers.l3_rpc -- Routers returned to l3 agent: -- returns list of routers with bound new port (for the subnet being attached) 
15:22:27 ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-10-13 16:25:22 -0500

Should this be filed as a bug? I'm not sure we as a community have a good process for ensuring these issue get a good follow up when they're brought up in this forum. I, for one, do not spend much time here. However, our new PTL is improving our process for triaging and following up on bugs as we speak with good success.

We are currently discussing a performance team within Neutron. It could take a look at things like this. I'd file a bug and tag it with 'loadimpact'.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2015-10-07 20:48:20 -0500

Seen: 197 times

Last updated: Oct 07 '15