Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Why TAP-interface (qr-) not created by OVS-agent?

Hello! My system: Ubuntu 14.04 LTS x64. Openstack Mitaka.

My problem: I create instance from Horizon. It have ip 10.0.0.30 But from Host it is unreachable:

root@openstack-2:/opt/stack# ping 10.0.0.30
PING 10.0.0.30 (10.0.0.30) 56(84) bytes of data.
^C
--- 10.0.0.30 ping statistics ---
24 packets transmitted, 0 received, 100% packet loss, time 23105ms

I think that is because there is no TAP-interfaces on OVS-bridge:

root@openstack-2:/opt/stack# ovs-vsctl show
99ad01f7-28ac-4bdc-b0b3-eb12c6e2a080
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth1"
            Interface "eth1"
    ovs_version: "2.5.0"

Output of command "ip netns list" is empty.

Running processes:

root@openstack-2:/opt/stack# pstree
init─┬─acpid
     ├─apache2─┬─2*[apache2───26*[{apache2}]]
     │         ├─10*[apache2───3*[{apache2}]]
     │         └─3*[apache2───12*[{apache2}]]
     ├─atd
     ├─cgmanager
     ├─cron
     ├─dbus-daemon
     ├─dnsmasq───dnsmasq
     ├─epmd
     ├─6*[getty]
     ├─glance-api───8*[glance-api]
     ├─glance-registry───8*[glance-registry]
     ├─irqbalance
     ├─2*[iscsid]
     ├─libvirtd───16*[{libvirtd}]
     ├─logger
     ├─memcached───5*[{memcached}]
     ├─mongod───10*[{mongod}]
     ├─monitor───ovsdb-server
     ├─monitor───ovs-vswitchd───9*[{ovs-vswitchd}]
     ├─mysqld_safe───mysqld───80*[{mysqld}]
     ├─neutron-dhcp-ag
     ├─neutron-l3-agen
     ├─neutron-linuxbr
     ├─neutron-metadat───4*[neutron-metadat]
     ├─neutron-openvsw───sudo───neutron-rootwra───ovsdb-client
     ├─neutron-server───10*[neutron-server]
     ├─nova-api───16*[nova-api]
     ├─nova-compute───21*[{nova-compute}]
     ├─nova-conductor───8*[nova-conductor]
     ├─nova-consoleaut
     ├─nova-novncproxy
     ├─nova-scheduler
     ├─ntpd
     ├─qemu-system-x86───3*[{qemu-system-x86}]
     ├─rabbitmq-server───rabbitmq-server───beam.smp─┬─inet_gethost───inet_gethost
     │                                              └─76*[{beam.smp}]
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd───sshd───sshd───bash───sudo───bash───pstree
     ├─systemd-logind
     ├─systemd-udevd
     ├─upstart-file-br
     ├─upstart-socket-
     ├─upstart-udev-br
     ├─virtlockd───{virtlockd}
     └─virtlogd───{virtlogd}

root@openstack-2:/opt/stack# openstack service list
+----------------------------------+----------+----------+
| ID                               | Name     | Type     |
+----------------------------------+----------+----------+
| 0656ba73a84d478a82e71cda52db6332 | neutron  | network  |
| 2b3b893769ec48918056bb03efe3c35c | keystone | identity |
| 92c9541506034ee094372de6207886d6 | glance   | image    |
| fb7344c7e88347de86ea923f71dec059 | nova     | compute  |
+----------------------------------+----------+----------+

root@openstack-2:/opt/stack# neutron agent-list
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host        | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| 3b8b06ec-6b99-45f0-9c71-f5c4c58129e7 | Open vSwitch agent | openstack-2 |                   | :-)   | True           | neutron-openvswitch-agent |
| ae0eae88-0326-41c8-876c-8387cdbf9941 | Linux bridge agent | openstack-2 |                   | :-)   | True           | neutron-linuxbridge-agent |
| d2768135-6aa0-4334-9cdf-176cb62e5a5f | Metadata agent     | openstack-2 |                   | :-)   | True           | neutron-metadata-agent    |
| eb7db378-fd15-4d40-ba6b-a44fd0fdb0ea | DHCP agent         | openstack-2 | nova              | :-)   | True           | neutron-dhcp-agent        |
| efe5cd21-c65e-4c03-8f8c-b251f18e2dbb | L3 agent           | openstack-2 | nova              | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+

root@openstack-2:/opt/stack# neutron net-show private-network-test
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2016-05-13T08:19:11                  |
| description               |                                      |
| id                        | 13f65443-b111-4c82-afba-f86ac5430a9f |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450                                 |
| name                      | private-network-test                 |
| port_security_enabled     | True                                 |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 75                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 478bc657-88b0-4e69-be65-bb3de3ebd7bf |
| tags                      |                                      |
| tenant_id                 | d80ebc72ac9e49eba0e9a0fc62d48474     |
| updated_at                | 2016-05-13T08:19:11                  |
+---------------------------+--------------------------------------+

========================================================

cat /etc/neutron/l3_agent.ini

[DEFAULT]
l3_agent_manager = neutron.agent.l3_agent.L3NATAgentWithStateReport
ovs_use_veth = false
interface_driver = openvswitch
router_delete_namespaces = False
verbose = true
[AGENT]
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

========================================================

cat /etc/neutron/neutron.conf

[DEFAULT]
auth_strategy = keystone
core_plugin = ml2
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.metering.metering_plugin.MeteringPlugin
allow_overlapping_ips = True
notify_nova_on_port_status_changes = true
notify_nova_on_port_data_changes = true
verbose = true
rpc_backend = rabbit
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://neutron:ghjcnjgfhjkm@controller/neutron
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = ghjcnjgfhjkm
[matchmaker_redis]
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = nova
password = ghjcnjgfhjkm
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = ghjcnjgfhjkm
[oslo_policy]
[quotas]
[ssl]

========================================================

cat /etc/neutron/plugins/ml2/ml2_conf.ini

[DEFAULT]
[ml2]
type_drivers = local,flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,linuxbridge
extension_drivers = port_security
[ml2_type_flat]
[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]
network_vlan_ranges = extnet:3000:3600
[ml2_type_vxlan]
ivni_ranges = 1001:3600
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_ipset = true
[agent]
tunnel_types = vxlan
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[ovs]
datapath_type = system
tunnel_bridge = br-tun
local_ip = 172.17.44.62
integration_bridge = br-int
bridge_mappings = external:br-ex

========================================================

Logs:

l3-agent.log http://pastebin.com/iDy0ymee

neutron-server.log http://pastebin.com/JgrDXv2v

openvswitch-agent.log http://pastebin.com/uHJsjWww

In my opinion problem is there (from /var/log/neutron/openvswitch-agent.log): Port 50b07103-a0f0-47c2-b85c-b770ab4dec95 not present in bridge br-int

Why TAP-interface (qr-) not created by OVS-agent?

Best regards, Mikhail

Why TAP-interface (qr-) not created by OVS-agent?

Hello! My system: Ubuntu 14.04 LTS x64. Openstack Mitaka.

My problem: I create instance from Horizon. It have ip 10.0.0.30 But from Host it is unreachable:

root@openstack-2:/opt/stack# ping 10.0.0.30
PING 10.0.0.30 (10.0.0.30) 56(84) bytes of data.
^C
--- 10.0.0.30 ping statistics ---
24 packets transmitted, 0 received, 100% packet loss, time 23105ms

I think that is because there is no TAP-interfaces on OVS-bridge:

root@openstack-2:/opt/stack# ovs-vsctl show
99ad01f7-28ac-4bdc-b0b3-eb12c6e2a080
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth1"
            Interface "eth1"
    ovs_version: "2.5.0"

Output of command "ip netns list" is empty.

Running processes:

root@openstack-2:/opt/stack# pstree
init─┬─acpid
     ├─apache2─┬─2*[apache2───26*[{apache2}]]
     │         ├─10*[apache2───3*[{apache2}]]
     │         └─3*[apache2───12*[{apache2}]]
     ├─atd
     ├─cgmanager
     ├─cron
     ├─dbus-daemon
     ├─dnsmasq───dnsmasq
     ├─epmd
     ├─6*[getty]
     ├─glance-api───8*[glance-api]
     ├─glance-registry───8*[glance-registry]
     ├─irqbalance
     ├─2*[iscsid]
     ├─libvirtd───16*[{libvirtd}]
     ├─logger
     ├─memcached───5*[{memcached}]
     ├─mongod───10*[{mongod}]
     ├─monitor───ovsdb-server
     ├─monitor───ovs-vswitchd───9*[{ovs-vswitchd}]
     ├─mysqld_safe───mysqld───80*[{mysqld}]
     ├─neutron-dhcp-ag
     ├─neutron-l3-agen
     ├─neutron-linuxbr
     ├─neutron-metadat───4*[neutron-metadat]
     ├─neutron-openvsw───sudo───neutron-rootwra───ovsdb-client
     ├─neutron-server───10*[neutron-server]
     ├─nova-api───16*[nova-api]
     ├─nova-compute───21*[{nova-compute}]
     ├─nova-conductor───8*[nova-conductor]
     ├─nova-consoleaut
     ├─nova-novncproxy
     ├─nova-scheduler
     ├─ntpd
     ├─qemu-system-x86───3*[{qemu-system-x86}]
     ├─rabbitmq-server───rabbitmq-server───beam.smp─┬─inet_gethost───inet_gethost
     │                                              └─76*[{beam.smp}]
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd───sshd───sshd───bash───sudo───bash───pstree
     ├─systemd-logind
     ├─systemd-udevd
     ├─upstart-file-br
     ├─upstart-socket-
     ├─upstart-udev-br
     ├─virtlockd───{virtlockd}
     └─virtlogd───{virtlogd}

root@openstack-2:/opt/stack# openstack service list
+----------------------------------+----------+----------+
| ID                               | Name     | Type     |
+----------------------------------+----------+----------+
| 0656ba73a84d478a82e71cda52db6332 | neutron  | network  |
| 2b3b893769ec48918056bb03efe3c35c | keystone | identity |
| 92c9541506034ee094372de6207886d6 | glance   | image    |
| fb7344c7e88347de86ea923f71dec059 | nova     | compute  |
+----------------------------------+----------+----------+

root@openstack-2:/opt/stack# neutron agent-list
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host        | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| 3b8b06ec-6b99-45f0-9c71-f5c4c58129e7 | Open vSwitch agent | openstack-2 |                   | :-)   | True           | neutron-openvswitch-agent |
| ae0eae88-0326-41c8-876c-8387cdbf9941 | Linux bridge agent | openstack-2 |                   | :-)   | True           | neutron-linuxbridge-agent |
| d2768135-6aa0-4334-9cdf-176cb62e5a5f | Metadata agent     | openstack-2 |                   | :-)   | True           | neutron-metadata-agent    |
| eb7db378-fd15-4d40-ba6b-a44fd0fdb0ea | DHCP agent         | openstack-2 | nova              | :-)   | True           | neutron-dhcp-agent        |
| efe5cd21-c65e-4c03-8f8c-b251f18e2dbb | L3 agent           | openstack-2 | nova              | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+

root@openstack-2:/opt/stack# neutron net-show private-network-test
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2016-05-13T08:19:11                  |
| description               |                                      |
| id                        | 13f65443-b111-4c82-afba-f86ac5430a9f |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450                                 |
| name                      | private-network-test                 |
| port_security_enabled     | True                                 |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 75                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 478bc657-88b0-4e69-be65-bb3de3ebd7bf |
| tags                      |                                      |
| tenant_id                 | d80ebc72ac9e49eba0e9a0fc62d48474     |
| updated_at                | 2016-05-13T08:19:11                  |
+---------------------------+--------------------------------------+

========================================================

cat /etc/neutron/l3_agent.ini

[DEFAULT]
l3_agent_manager = neutron.agent.l3_agent.L3NATAgentWithStateReport
ovs_use_veth = false
interface_driver = openvswitch
router_delete_namespaces = False
verbose = true
[AGENT]
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

========================================================

cat /etc/neutron/neutron.conf

[DEFAULT]
auth_strategy = keystone
core_plugin = ml2
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.metering.metering_plugin.MeteringPlugin
allow_overlapping_ips = True
notify_nova_on_port_status_changes = true
notify_nova_on_port_data_changes = true
verbose = true
rpc_backend = rabbit
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://neutron:ghjcnjgfhjkm@controller/neutron
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = ghjcnjgfhjkm
[matchmaker_redis]
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = nova
password = ghjcnjgfhjkm
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = ghjcnjgfhjkm
[oslo_policy]
[quotas]
[ssl]

========================================================

cat /etc/neutron/plugins/ml2/ml2_conf.ini

[DEFAULT]
[ml2]
type_drivers = local,flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,linuxbridge
extension_drivers = port_security
[ml2_type_flat]
[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]
network_vlan_ranges = extnet:3000:3600
[ml2_type_vxlan]
ivni_ranges = 1001:3600
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_ipset = true
[agent]
tunnel_types = vxlan
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[ovs]
datapath_type = system
tunnel_bridge = br-tun
local_ip = 172.17.44.62
integration_bridge = br-int
bridge_mappings = external:br-ex

========================================================

rootwrap.conf and rootwrap.d/*.filters are same as on devstack installation of OpenStack ( I else install devstack which works fine). Configuration:

cat rootwrap.conf
[DEFAULT]
filters_path=/etc/neutron/rootwrap.d,/usr/share/neutron/rootwrap
exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin
use_syslog=False
syslog_log_facility=syslog
syslog_log_level=ERROR

cat rootwrap.d/openvswitch-plugin.filters
[Filters]
ovs-vsctl: CommandFilter, ovs-vsctl, root
ovs-ofctl: CommandFilter, ovs-ofctl, root
kill_ovsdb_client: KillFilter, root, /usr/bin/ovsdb-client, -9
ovsdb-client: CommandFilter, ovsdb-client, root
xe: CommandFilter, xe, root
ip: IpFilter, ip, root
find: RegExpFilter, find, root, find, /sys/class/net, -maxdepth, 1, -type, l, -printf, %.*
ip_exec: IpNetnsExecFilter, ip, root


cat rootwrap.d/l3.filters
[Filters]
arping: CommandFilter, arping, root
sysctl: CommandFilter, sysctl, root
route: CommandFilter, route, root
radvd: CommandFilter, radvd, root
metadata_proxy: CommandFilter, neutron-ns-metadata-proxy, root
kill_metadata: KillFilter, root, python, -9
kill_metadata7: KillFilter, root, python2.7, -9
kill_radvd_usr: KillFilter, root, /usr/sbin/radvd, -9, -HUP
kill_radvd: KillFilter, root, /sbin/radvd, -9, -HUP
ip: IpFilter, ip, root
find: RegExpFilter, find, root, find, /sys/class/net, -maxdepth, 1, -type, l, -printf, %.*
ip_exec: IpNetnsExecFilter, ip, root
kill_ip_monitor: KillFilter, root, ip, -9
ovs-vsctl: CommandFilter, ovs-vsctl, root
iptables-save: CommandFilter, iptables-save, root
iptables-restore: CommandFilter, iptables-restore, root
ip6tables-save: CommandFilter, ip6tables-save, root
ip6tables-restore: CommandFilter, ip6tables-restore, root
keepalived: CommandFilter, keepalived, root
kill_keepalived: KillFilter, root, /usr/sbin/keepalived, -HUP, -15, -9
conntrack: CommandFilter, conntrack, root
keepalived_state_change: CommandFilter, neutron-keepalived-state-change, root

In l3-agent.log i found

"ERROR oslo_service.periodic_task [req-cc4f36b1-e3d1-4e84-832d-0679fa42eddf - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task"

and then

"Exception: Failed to spawn rootwrap process."

I think this is reason why TAP interfaces (qr-) not created. But I do not know the reason may be what else.

======================================================== Logs:

l3-agent.log http://pastebin.com/iDy0ymeehttp://pastebin.com/ZxC6Hpid

neutron-server.log http://pastebin.com/JgrDXv2v

openvswitch-agent.log http://pastebin.com/uHJsjWww

In my opinion problem is there (from /var/log/neutron/openvswitch-agent.log): Port 50b07103-a0f0-47c2-b85c-b770ab4dec95 not present in bridge br-int

Why TAP-interface (qr-) not created by OVS-agent?

Best regards, Mikhail

Why TAP-interface (qr-) not created by OVS-agent?

Hello! My system: Ubuntu 14.04 LTS x64. Openstack Mitaka.

My problem: I create instance from Horizon. It have ip 10.0.0.30 But from Host it is unreachable:

root@openstack-2:/opt/stack# ping 10.0.0.30
PING 10.0.0.30 (10.0.0.30) 56(84) bytes of data.
^C
--- 10.0.0.30 ping statistics ---
24 packets transmitted, 0 received, 100% packet loss, time 23105ms

I think that is because there is no TAP-interfaces on OVS-bridge:

root@openstack-2:/opt/stack# ovs-vsctl show
99ad01f7-28ac-4bdc-b0b3-eb12c6e2a080
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth1"
            Interface "eth1"
    ovs_version: "2.5.0"

Output of command "ip netns list" is empty.

Running processes:

root@openstack-2:/opt/stack# pstree
init─┬─acpid
     ├─apache2─┬─2*[apache2───26*[{apache2}]]
     │         ├─10*[apache2───3*[{apache2}]]
     │         └─3*[apache2───12*[{apache2}]]
     ├─atd
     ├─cgmanager
     ├─cron
     ├─dbus-daemon
     ├─dnsmasq───dnsmasq
     ├─epmd
     ├─6*[getty]
     ├─glance-api───8*[glance-api]
     ├─glance-registry───8*[glance-registry]
     ├─irqbalance
     ├─2*[iscsid]
     ├─libvirtd───16*[{libvirtd}]
     ├─logger
     ├─memcached───5*[{memcached}]
     ├─mongod───10*[{mongod}]
     ├─monitor───ovsdb-server
     ├─monitor───ovs-vswitchd───9*[{ovs-vswitchd}]
     ├─mysqld_safe───mysqld───80*[{mysqld}]
     ├─neutron-dhcp-ag
     ├─neutron-l3-agen
     ├─neutron-linuxbr
     ├─neutron-metadat───4*[neutron-metadat]
     ├─neutron-openvsw───sudo───neutron-rootwra───ovsdb-client
     ├─neutron-server───10*[neutron-server]
     ├─nova-api───16*[nova-api]
     ├─nova-compute───21*[{nova-compute}]
     ├─nova-conductor───8*[nova-conductor]
     ├─nova-consoleaut
     ├─nova-novncproxy
     ├─nova-scheduler
     ├─ntpd
     ├─qemu-system-x86───3*[{qemu-system-x86}]
     ├─rabbitmq-server───rabbitmq-server───beam.smp─┬─inet_gethost───inet_gethost
     │                                              └─76*[{beam.smp}]
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd───sshd───sshd───bash───sudo───bash───pstree
     ├─systemd-logind
     ├─systemd-udevd
     ├─upstart-file-br
     ├─upstart-socket-
     ├─upstart-udev-br
     ├─virtlockd───{virtlockd}
     └─virtlogd───{virtlogd}

root@openstack-2:/opt/stack# openstack service list
+----------------------------------+----------+----------+
| ID                               | Name     | Type     |
+----------------------------------+----------+----------+
| 0656ba73a84d478a82e71cda52db6332 | neutron  | network  |
| 2b3b893769ec48918056bb03efe3c35c | keystone | identity |
| 92c9541506034ee094372de6207886d6 | glance   | image    |
| fb7344c7e88347de86ea923f71dec059 | nova     | compute  |
+----------------------------------+----------+----------+

root@openstack-2:/opt/stack# neutron agent-list
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host        | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+
| 3b8b06ec-6b99-45f0-9c71-f5c4c58129e7 | Open vSwitch agent | openstack-2 |                   | :-)   | True           | neutron-openvswitch-agent |
| ae0eae88-0326-41c8-876c-8387cdbf9941 | Linux bridge agent | openstack-2 |                   | :-)   | True           | neutron-linuxbridge-agent |
| d2768135-6aa0-4334-9cdf-176cb62e5a5f | Metadata agent     | openstack-2 |                   | :-)   | True           | neutron-metadata-agent    |
| eb7db378-fd15-4d40-ba6b-a44fd0fdb0ea | DHCP agent         | openstack-2 | nova              | :-)   | True           | neutron-dhcp-agent        |
| efe5cd21-c65e-4c03-8f8c-b251f18e2dbb | L3 agent           | openstack-2 | nova              | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+-------------+-------------------+-------+----------------+---------------------------+

root@openstack-2:/opt/stack# neutron net-show private-network-test
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2016-05-13T08:19:11                  |
| description               |                                      |
| id                        | 13f65443-b111-4c82-afba-f86ac5430a9f |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450                                 |
| name                      | private-network-test                 |
| port_security_enabled     | True                                 |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 75                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 478bc657-88b0-4e69-be65-bb3de3ebd7bf |
| tags                      |                                      |
| tenant_id                 | d80ebc72ac9e49eba0e9a0fc62d48474     |
| updated_at                | 2016-05-13T08:19:11                  |
+---------------------------+--------------------------------------+

========================================================

cat /etc/neutron/l3_agent.ini

[DEFAULT]
l3_agent_manager = neutron.agent.l3_agent.L3NATAgentWithStateReport
ovs_use_veth = false
interface_driver = openvswitch
router_delete_namespaces = False
verbose = true
[AGENT]
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

========================================================

cat /etc/neutron/neutron.conf

[DEFAULT]
auth_strategy = keystone
core_plugin = ml2
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.metering.metering_plugin.MeteringPlugin
allow_overlapping_ips = True
notify_nova_on_port_status_changes = true
notify_nova_on_port_data_changes = true
verbose = true
rpc_backend = rabbit
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://neutron:ghjcnjgfhjkm@controller/neutron
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = ghjcnjgfhjkm
[matchmaker_redis]
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = nova
password = ghjcnjgfhjkm
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = ghjcnjgfhjkm
[oslo_policy]
[quotas]
[ssl]

========================================================

cat /etc/neutron/plugins/ml2/ml2_conf.ini

[DEFAULT]
[ml2]
type_drivers = local,flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,linuxbridge
extension_drivers = port_security
[ml2_type_flat]
[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]
network_vlan_ranges = extnet:3000:3600
[ml2_type_vxlan]
ivni_ranges = 1001:3600
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_ipset = true
[agent]
tunnel_types = vxlan
root_helper_daemon = sudo /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
root_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[ovs]
datapath_type = system
tunnel_bridge = br-tun
local_ip = 172.17.44.62
integration_bridge = br-int
bridge_mappings = external:br-ex

========================================================

rootwrap.conf and rootwrap.d/*.filters are same as on devstack installation of OpenStack ( I else install devstack which works fine). Configuration:

cat rootwrap.conf
[DEFAULT]
filters_path=/etc/neutron/rootwrap.d,/usr/share/neutron/rootwrap
exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin
use_syslog=False
syslog_log_facility=syslog
syslog_log_level=ERROR

cat rootwrap.d/openvswitch-plugin.filters
[Filters]
ovs-vsctl: CommandFilter, ovs-vsctl, root
ovs-ofctl: CommandFilter, ovs-ofctl, root
kill_ovsdb_client: KillFilter, root, /usr/bin/ovsdb-client, -9
ovsdb-client: CommandFilter, ovsdb-client, root
xe: CommandFilter, xe, root
ip: IpFilter, ip, root
find: RegExpFilter, find, root, find, /sys/class/net, -maxdepth, 1, -type, l, -printf, %.*
ip_exec: IpNetnsExecFilter, ip, root


cat rootwrap.d/l3.filters
[Filters]
arping: CommandFilter, arping, root
sysctl: CommandFilter, sysctl, root
route: CommandFilter, route, root
radvd: CommandFilter, radvd, root
metadata_proxy: CommandFilter, neutron-ns-metadata-proxy, root
kill_metadata: KillFilter, root, python, -9
kill_metadata7: KillFilter, root, python2.7, -9
kill_radvd_usr: KillFilter, root, /usr/sbin/radvd, -9, -HUP
kill_radvd: KillFilter, root, /sbin/radvd, -9, -HUP
ip: IpFilter, ip, root
find: RegExpFilter, find, root, find, /sys/class/net, -maxdepth, 1, -type, l, -printf, %.*
ip_exec: IpNetnsExecFilter, ip, root
kill_ip_monitor: KillFilter, root, ip, -9
ovs-vsctl: CommandFilter, ovs-vsctl, root
iptables-save: CommandFilter, iptables-save, root
iptables-restore: CommandFilter, iptables-restore, root
ip6tables-save: CommandFilter, ip6tables-save, root
ip6tables-restore: CommandFilter, ip6tables-restore, root
keepalived: CommandFilter, keepalived, root
kill_keepalived: KillFilter, root, /usr/sbin/keepalived, -HUP, -15, -9
conntrack: CommandFilter, conntrack, root
keepalived_state_change: CommandFilter, neutron-keepalived-state-change, root

In l3-agent.log i found

"ERROR oslo_service.periodic_task [req-cc4f36b1-e3d1-4e84-832d-0679fa42eddf - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task"

and then

"Exception: Failed to spawn rootwrap process."

I think this is reason why TAP interfaces (qr-) not created. But I do not know the reason may be what else.

======================================================== Logs:

l3-agent.log http://pastebin.com/ZxC6Hpid

neutron-server.log http://pastebin.com/JgrDXv2v

openvswitch-agent.log http://pastebin.com/uHJsjWww

In my opinion problem is there (from /var/log/neutron/openvswitch-agent.log): Port 50b07103-a0f0-47c2-b85c-b770ab4dec95 not present in bridge br-int

Why TAP-interface (qr-) not created by OVS-agent?

Best regards, Mikhail

UPD. There is mistake in nova.conf authenticate parameters for section [neutron]. Some logs in /var/log/neutron help me.