Revision history [back]

click to hide/show revision 1
initial version

Instance Launching Issue in Openstack - No Valid Host was found

Issue : Getting Error in launching Instance VM in Openstack

Error: "No valid host was found" Scenario: One Controller, one Compute VM machines are integrated with APIC Controller

Release JUNO, distro: CentOS 7

Setup is showing all Openstack created tenants information in APIC but triggering error for launching instance, it may not find/discover Compute Node. All services on controller and compute are running nicely.

Suspicious: TRACE neutron.plugins.ml2.managers ApicHostNotConfigured: The switch and port for host 'juno-controller-ali' are not configured"

File: /etc/neutron/plugins/ml2/ml2_config_cisco.ini [Default] apic_system_id = ABC

[ml2_cisco_apic] apic_hosts = 10.3.1.207:80,10.3.1.207:443 apic_username = admin apic_password = MyPassword apic_name_mapping = use_name apic_use_ssl = True

[apic_external_network:POD7-ext] switch = 101 port = 1/19 cidr_exposed = 10.3.1.0/24 gateway_ip = 10.3.1.254

[apic_switch:101] # when I add these compute nodes.. Nuetron service fails

File: /etc/neutron/plugins/ml2/ml2_conf.ini

[ml2] type_drivers = opflex,local,flat,vlan,gre,vxlan tenant_network_types = vlan mechanism_drivers =openvswitch,cisco_apic

[ml2_type_vlan] network_vlan_ranges = anx:790:795

[ovs] bridge_mappings = anx:br-anx enable_tunneling = False integration_bridge = br-int

[agent] polling_interval = 2 l2_population = False arp_responder = False

File: /etc/neutron/neutron.conf

[DEFAULT] osapi_compute_workers = 4 workers = 4 ec2_workers = 4 metadata_workers = 4 rpc_backend = rabbit rabbit_host = 10.3.1.175 rabbit_password = MyPassword auth_strategy = keystone notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True nova_url = http://10.3.1.175:8774/v2 nova_admin_auth_url = http://10.3.1.175:35357/v2.0/ nova_region_name = regionOne nova_admin_username = nova nova_admin_tenant_id = 4f6cf805788b4d4f9539563889e41d53 nova_admin_password = MyPassword core_plugin = ml2 service_plugins = router allow_overlapping_ips = True

[keystone_authtoken] auth_uri = http://10.3.1.175:5000/v2.0/ identity_uri = http://10.3.1.175:35357 admin_tenant_name = service admin_user = neutron admin_password = MyPassword auth_protocol = http auth_host = 10.3.1.175 auth_port = 35357

LOGS: TAIL -F /VAR/LOG/NEUTRON/SERVER.LOG [root@juno-controller-ali ~(keystone_admin)]# tail -f /var/log/neutron/server.log TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1147, in update_port_status TRACE oslo.messaging.rpc.dispatcher self.mechanism_manager.update_port_postcommit(mech_context) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 527, in update_port_postcommit TRACE oslo.messaging.rpc.dispatcher continue_on_failure=True) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 302, in _call_on_drivers TRACE oslo.messaging.rpc.dispatcher method=method_name TRACE oslo.messaging.rpc.dispatcher MechanismDriverError: update_port_postcommit failed. TRACE oslo.messaging.rpc.dispatcher ERROR oslo.messaging._drivers.common [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 ] Returning exception update_port_postcommit failed. to caller ERROR oslo.messaging._drivers.common [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 167, in update_device_up\n host)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1147, in update_port_status\n self.mechanism_manager.update_port_postcommit(mech_context)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 527, in update_port_postcommit\n continue_on_failure=True)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 302, in _call_on_drivers\n method=method_name\n', 'MechanismDriverError: update_port_postcommit failed.\n'] ERROR neutron.plugins.ml2.managers [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 None] Mechanism driver 'cisco_apic' failed in update_port_postcommit TRACE neutron.plugins.ml2.managers Traceback (most recent call last): TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 291, in _call_on_drivers TRACE neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/apic/mechanism_apic.py", line 81, in inner TRACE neutron.plugins.ml2.managers return f(inst, *args, *kwargs) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/apic/mechanism_apic.py", line 201, in update_port_postcommit TRACE neutron.plugins.ml2.managers self._perform_port_operations(context) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/apic/mechanism_apic.py", line 146, in _perform_port_operations TRACE neutron.plugins.ml2.managers self._perform_path_port_operations(context, port) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner TRACE neutron.plugins.ml2.managers return f(args, *kwargs) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/apic/mechanism_apic.py", line 106, in _perform_path_port_operations TRACE neutron.plugins.ml2.managers tenant_id, anetwork_id, host, seg, transaction=trs) TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/apicapi/apic_manager.py", line 669, in ensure_path_created_for_port TRACE neutron.plugins.ml2.managers raise cexc.ApicHostNotConfigured(host=host_id) TRACE neutron.plugins.ml2.managers ApicHostNotConfigured: The switch and port for host 'juno-controller-ali' are not configured TRACE neutron.plugins.ml2.managers ERROR oslo.messaging.rpc.dispatcher [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 ] Exception during message handling: update_port_postcommit failed. TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply TRACE oslo.messaging.rpc.dispatcher incoming.message)) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, *new_args) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 115, in get_devices_details_list TRACE oslo.messaging.rpc.dispatcher for device in kwargs.pop('devices', []) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 92, in get_device_details TRACE oslo.messaging.rpc.dispatcher host) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1147, in update_port_status TRACE oslo.messaging.rpc.dispatcher self.mechanism_manager.update_port_postcommit(mech_context) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 527, in update_port_postcommit TRACE oslo.messaging.rpc.dispatcher continue_on_failure=True) TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 302, in _call_on_drivers TRACE oslo.messaging.rpc.dispatcher method=method_name TRACE oslo.messaging.rpc.dispatcher MechanismDriverError: update_port_postcommit failed. TRACE oslo.messaging.rpc.dispatcher ERROR oslo.messaging._drivers.common [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 ] Returning exception update_port_postcommit failed. to caller ERROR oslo.messaging._drivers.common [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 115, in get_devices_details_list\n for device in kwargs.pop(\'devices\', [])\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 92, in get_device_details\n host)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1147, in update_port_status\n self.mechanism_manager.update_port_postcommit(mech_context)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 527, in update_port_postcommit\n continue_on_failure=True)\n', ' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 302, in _call_on_drivers\n method=method_name\n', 'MechanismDriverError: update_port_postcommit failed.\n']

==================================================================================================

tail -f /var/log/neutron/openvswitch-agent.log

TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 None] Agent out of sync with plugin! INFO neutron.agent.securitygroups_rpc [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 None] Skipping method prepare_devices_filter as firewall is disabled or configured as NoopFirewallDriver. ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-5e8e3cbb-69fd-494e-b27c-981f0311e810 None] process_network_ports - iteration:1745 - failure while retrieving port details from server TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1228, in process_network_ports TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices_added_updated, ovs_restarted) TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1101, in treat_devices_added_or_updated TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise DeviceListRetrievalError(devices=devices, error=e) TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent DeviceListRetrievalError: Unable to retrieve port details for devices: set([u'b11c07f8-1aec-449c-8d65-1d6ed31cabf2', u'e1778bbc-32ff-490d-9adf-6ea16fa181cf', u'437d1e45-47ea-4485-bd6c-449ce0bf8df2', u'c9bef4cc-b0f8-4671-82f9-25843bf39a6f', u'f0da8c58-7d2a-4c18-9239-0b1bada6005f', u'19a96aa5-478e-45d1-b890-6a2d9b359fee']) because of error: Remote error: MechanismDriverError update_port_postcommit failed. TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 115, in get_devices_details_list\n for device in kwargs.pop(\'devices\', [])\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 92, in get_device_details\n host)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1147, in update_port_status\n self.mechanism_manager.update_port_postcommit(mech_context)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 527, in update_port_postcommit\n continue_on_failure=True)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 302, in _call_on_drivers\n method=method_name\n', u'MechanismDriverError: update_port_postcommit failed.\n'].

================================================================================================== [root@juno-controller-ali ~(keystone_admin)]# tail -f /var/log/nova/nova-consoleauth.log INFO oslo.messaging._drivers.impl_rabbit [req-a42c4c92-0593-4490-9317-cc966421e8ab ] Connected to AMQP server on 10.3.1.175:5672 AUDIT nova.service [-] Starting consoleauth node (version 2014.2.2-1.el7) WARNING oslo.db.sqlalchemy.session [req-830cb580-1cf6-44e5-9c44-ff55eb6499ee ] SQL connection failed. -1 attempts left. INFO oslo.messaging._drivers.impl_rabbit [req-830cb580-1cf6-44e5-9c44-ff55eb6499ee ] Connecting to AMQP server on 10.3.1.175:5672 INFO oslo.messaging._drivers.impl_rabbit [req-830cb580-1cf6-44e5-9c44-ff55eb6499ee ] Connected to AMQP server on 10.3.1.175:5672 INFO nova.openstack.common.service [-] Caught SIGTERM, exiting AUDIT nova.service [-] Starting consoleauth node (version 2014.2.2-1.el7) INFO oslo.messaging._drivers.impl_rabbit [req-7e11f505-8a97-4c84-a180-2c8a3506bfc6 ] Connecting to AMQP server on 10.3.1.175:5672 INFO oslo.messaging._drivers.impl_rabbit [req-7e11f505-8a97-4c84-a180-2c8a3506bfc6 ] Connected to AMQP server on 10.3.1.175:5672

WARNING nova.openstack.common.loopingcall [-] task <bound method="" dbdriver._report_state="" of="" <nova.servicegroup.drivers.db.dbdriver="" object="" at="" 0x3ed3e10="">> run outlasted interval by 37.89 sec

Any help will be highly appreciable.

Thanks & Best Regards

Ali Murtaza Solangi