Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Triple-O Introspection failure, ironic-python-agent ConnectionError

Hello,

I am currently setting up a test baremetal lab environment using tripleO (pike release). However I am stuck at the introspection part of the overcloud setup.

During introspection, the overcloud node boots up, but then the ironic-python-agent service fails to start with the following error;

ERROR ironic-python-agent ConnectionError: HTTPConnectionPool(host='192.168.24.1', port=5050'): Max retries exceeded with url: /v1/continue (Caused by NewConnectionError('<request.packages.urllib3.connection.httpconnection object="" at="" 0x2d9f6d0="">: Failed to establish a new connection: (Errno 101) Network is unreachable',))

Now when I login into ramdisk image and run the " ip addr " command; I see no IPs assigned to any of the node interfaces which explains the error above; however when I run the " sudo journalctl -u openstack-ironic-inspector -u openstack-ironic-inspector-dnsmasq " command, I am to see the complete DHCP handshake; logs are as follows,

*

Dec 15 12:16:50 undercloud.inara dnsmasq-dhcp[1434]: DHCPDISCOVER(br-ctlplane) 0c:c4:7a:9f:f1:78

Dec 15 12:16:50 undercloud.inara dnsmasq-dhcp[1434]: DHCPOFFER(br-ctlplane) 192.168.24.111 0c:c4:7a:9f:f1:78

Dec 15 12:16:50 undercloud.inara dnsmasq-dhcp[1434]: DHCPDISCOVER(br-ctlplane) 0c:c4:7a:9f:f1:78

Dec 15 12:16:50 undercloud.inara dnsmasq-dhcp[1434]: DHCPOFFER(br-ctlplane) 192.168.24.111 0c:c4:7a:9f:f1:78

Dec 15 12:16:53 undercloud.inara dnsmasq-dhcp[1434]: DHCPREQUEST(br-ctlplane) 192.168.24.111 0c:c4:7a:9f:f1:78

Dec 15 12:16:53 undercloud.inara dnsmasq-dhcp[1434]: DHCPACK(br-ctlplane) 192.168.24.111 0c:c4:7a:9f:f1:78

Dec 15 12:16:59 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:16:59.465 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:16:59] "GET /v1/introspection/0

Dec 15 12:17:09 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:09.483 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:09] "GET /v1/introspection/0

Dec 15 12:17:19 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:19.503 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:19] "GET /v1/introspection/0

Dec 15 12:17:29 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:29.522 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:29] "GET /v1/introspection/0

Dec 15 12:17:30 undercloud.inara dnsmasq-dhcp[1434]: DHCPDISCOVER(br-ctlplane) 0c:c4:7a:9f:f1:78

Dec 15 12:17:30 undercloud.inara dnsmasq-dhcp[1434]: DHCPOFFER(br-ctlplane) 192.168.24.112 0c:c4:7a:9f:f1:78

Dec 15 12:17:30 undercloud.inara dnsmasq-dhcp[1434]: DHCPREQUEST(br-ctlplane) 192.168.24.112 0c:c4:7a:9f:f1:78

Dec 15 12:17:30 undercloud.inara dnsmasq-dhcp[1434]: DHCPACK(br-ctlplane) 192.168.24.112 0c:c4:7a:9f:f1:78

Dec 15 12:17:39 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:39.539 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:39] "GET /v1/introspection/0

Dec 15 12:17:45 undercloud.inara dnsmasq-dhcp[1434]: DHCPREQUEST(br-ctlplane) 192.168.24.110 ac:1f:6b:1d:14:a9

Dec 15 12:17:45 undercloud.inara dnsmasq-dhcp[1434]: DHCPACK(br-ctlplane) 192.168.24.110 ac:1f:6b:1d:14:a9

Dec 15 12:17:49 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:49.555 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:49] "GET /v1/introspection/0

Dec 15 12:17:59 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:17:59.576 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:17:59] "GET /v1/introspection/0

Dec 15 12:18:09 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:18:09.591 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:18:09] "GET /v1/introspection/0

Dec 15 12:18:19 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:18:19.609 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:18:19] "GET /v1/introspection/0

Dec 15 12:18:29 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:18:29.629 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:18:29] "GET /v1/introspection/0

Dec 15 12:18:39 undercloud.inara ironic-inspector[1381]: 2017-12-15 12:18:39.648 1381 INFO werkzeug [-] 192.168.24.1 - - [15/Dec/2017 12:18:39] "GET /v1/introspection/0

*

This is a complete DHCP handshake and why the nodes do not receive IPs baffles me. (P.S: I connected my computer's interface in the provisioning VLAN during introspection and successfully got an IP from the dhcp run by undercloud node.)

Doing a tcpdump on the undercloud node's provisioning interface gives the follwing results; 14:23:06.868792 IP 192.168.24.110.bootpc > 192.168.24.1.bootps: BOOTP/DHCP, Request from ac:1f:6b:1d:14:a9, length 300 14:23:06.870795 IP 192.168.24.1.bootps > 192.168.24.110.bootpc: BOOTP/DHCP, Reply, length 300 14:23:08.176130 IP 192.168.24.110 > 192.168.24.1: ICMP echo request, id 22025, seq 0, length 64 14:23:08.176218 IP 192.168.24.1 > 192.168.24.110: ICMP echo reply, id 22025, seq 0, length 64 14:23:11.868104 ARP, Request who-has 192.168.24.1 tell 192.168.24.110, length 46 14:23:11.868135 ARP, Reply 192.168.24.1 is-at ac:1f:6b:1d:16:fc, length 28

My network configuration consists of separate VLANs for the management and provisioning networks (on a cisco switch)

P.S: I noted that the ramdisk image does not have udp ports 67 and 68 open in the iptable list. Could this be the issue?