Ask Your Question
0

"Deadlock found" and "DuplicateEntry" errors when trying to attach a neutron L3 HA router to an internal network

asked 2015-07-31 02:53:29 -0500

arcimboldo gravatar image

On a setup with 2 neutron nodes, L3 agent in HA (VRRP), we:

  • create the external network (vlan)
  • create an internal network (vxlan)
  • create a router
  • attach the router to the external network
  • attach the router to the internal network

After attaching the router to the internal network, the following happens:

a) some router _and_ dhcp ports of the external networks go from ACTIVE to BUILD status b) dhcp ports of the internal networks go from ACTIVE to BUILD c) two type of errors are found on the neutron log file: - a Deadlock found when trying to get a lock error, on one of the network nodes - multiple Duplicate entry ... for key PRIMARY on both network nodes

How we replicated the issue

root@kenny:~# neutron  net-create uzh-only \
  --shared --provider:network_type vlan \
  --provider:segmentation_id 842 \
  --provider:physical_network vlannet \
  --router:external 

root@kenny:~# neutron  subnet-create uzh-only \
    --name vlan842 \
    172.23.0.0/16 \
    --gateway 172.23.0.1 \
    --allocation-pool start=172.23.2.0,end=172.23.200.255 \
    --enable-dhcp \
    --dns-nameservers 130.60.128.3 130.60.64.51 \
    --   --host-routes type=dict list=true \
    destination=130.60.206.69/32,nexthop=172.23.0.5

root@kenny:~# neutron net-create test 
root@kenny:~# neutron subnet-create test --name subtest 10.0.0.0/24 
root@kenny:~# neutron router-create test
root@kenny:~# neutron router-gateway-set test uzh-only

root@kenny:~# neutron port-list -c id -c fixed_ips -c status
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+
| id                                   | fixed_ips                                                                            | status | device_owner                |
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+
| 1463b78c-f640-4e41-8831-927679be380b | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.3"}      | ACTIVE | network:dhcp                |
| 1e7a0ac2-268c-48b7-9c70-e7e03126d9ed | {"subnet_id": "178f49ec-383e-4516-b6c3-814e7f459497", "ip_address": "169.254.192.1"} | ACTIVE | network:router_ha_interface |
| 234ddb9b-16cc-48f2-b646-fee5f6d5def2 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.2"}    | BUILD  | network:router_gateway      |
| 3e755174-14b3-48e9-af84-489cce8985be | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.2"}      | ACTIVE | network:dhcp                |
| 4df3d3ab-1967-4bf1-bc3f-31ec1addd8d1 | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.1"}      | BUILD  | network:router_interface    |
| 5a2e6cbd-4c69-47e0-bfa6-92fe6242aba4 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.0"}    | ACTIVE | network:dhcp                |
| 881882f0-4ad7-4294-bec4-93f869cb40d8 | {"subnet_id": "178f49ec-383e-4516-b6c3-814e7f459497", "ip_address": "169.254.192.2"} | ACTIVE | network:router_ha_interface |
| ac0e1761-335e-431f-ae89-a2703f2bafa5 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.1"}    | ACTIVE | network:dhcp                |
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+

root@kenny:~# neutron router-interface-add test subtest

root@kenny:~# neutron port-list -c id -c fixed_ips -c status

root@kenny:~# neutron port-list -c id -c fixed_ips -c status -c device_owner
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+
| id                                   | fixed_ips                                                                            | status | device_owner                |
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+
| 1463b78c-f640-4e41-8831-927679be380b | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.3"}      | BUILD  | network:dhcp                |
| 1e7a0ac2-268c-48b7-9c70-e7e03126d9ed | {"subnet_id": "178f49ec-383e-4516-b6c3-814e7f459497", "ip_address": "169.254.192.1"} | ACTIVE | network:router_ha_interface |
| 234ddb9b-16cc-48f2-b646-fee5f6d5def2 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.2"}    | BUILD  | network:router_gateway      |
| 3e755174-14b3-48e9-af84-489cce8985be | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.2"}      | ACTIVE | network:dhcp                |
| 4df3d3ab-1967-4bf1-bc3f-31ec1addd8d1 | {"subnet_id": "e5e43019-58e4-4203-a51b-8e0dce7ca829", "ip_address": "10.0.0.1"}      | BUILD  | network:router_interface    |
| 5a2e6cbd-4c69-47e0-bfa6-92fe6242aba4 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.0"}    | ACTIVE | network:dhcp                |
| 881882f0-4ad7-4294-bec4-93f869cb40d8 | {"subnet_id": "178f49ec-383e-4516-b6c3-814e7f459497", "ip_address": "169.254.192.2"} | BUILD  | network:router_ha_interface |
| ac0e1761-335e-431f-ae89-a2703f2bafa5 | {"subnet_id": "88a38e12-9402-418f-b0b7-1deeb843edf2", "ip_address": "172.23.2.1"}    | BUILD  | network:dhcp                |
+--------------------------------------+--------------------------------------------------------------------------------------+--------+-----------------------------+

excerpt from the logs on controller node A (only WARNING, ERROR and TRACE)

2015-07-30 15:50:59.322 251400 WARNING neutron.api.rpc.handlers.dhcp_rpc [req-203d6d37-4643-46e1-baa4-eb58c5f45750 ] Action create_port for network 9f7e9af9-72fc-44f1-89e3-be5cdf3d5bc5 could not complete successfully: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'INSERT INTO ml2_port_binding_levels (port_id, host, level, driver, segment_id) VALUES (%s ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-08-03 03:31:49 -0500

arcimboldo gravatar image

Sorry for the noise, this error was caused by a trigger in MySQL added by us.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-07-31 02:53:29 -0500

Seen: 319 times

Last updated: Aug 03 '15