Ask Your Question
0

octavia controller cant connect to amphora

asked 2019-08-24 12:16:40 -0500

stacks gravatar image

I followed this blog post to try and deploy Octavia on Openstack created with Juju and Maas.

https://ubuntu.com/blog/taking-octavia-for-a-ride-with-kubernetes-on-openstack (https://ubuntu.com/blog/taking-octavi...)

Everything seems to be working but when I try to create a load balancer the amphora instance fails. I've tried creating them manually with the Openstack CLI, via the Openstack dashboard and via a Kubernetes cluster that was deployed on Openstack with Juju. The problem may be with Openstack but I thought I'd start here first.

I thought maybe the issue was with creating the instance in Nova so I watched the logs there and everything seems fine.

Nova Log - https://gist.github.com/2stacks/152918ec7f7d60c61ba17d5809211c97 (https://gist.github.com/2stacks/15291...)

The only place I see an indication of an issue is in the Octavia worker logs. There seems to be a clear indication that the controller can't reach the newly created instance over the lb-mgmt-net.

2019-08-24 16:23:37.528 675855 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='fc00:8f36:122:78d8:f816:3eff:fe35:60e1', port=9443): Max retries exceeded with url: /0.5/info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbc2549d048>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-08-24 16:23:42.585 675855 ERROR octavia.amphorae.drivers.haproxy.exceptions [-] Amphora agent returned unexpected result code 404 with response {'error': '404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.', 'http_code': 404}
2019-08-24 16:23:43.013 675855 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait' (1858607d-f808-494f-8ead-8e4bc012a3e2) transitioned into state 'FAILURE' from state 'RUNNING'

Octavia Log - https://gist.github.com/2stacks/736d6b36f97cec0dc05025a7b198bdb2 (https://gist.github.com/2stacks/736d6...)

I verified that all of the necessary networks, security groups, flavors etc. that are created when running

juju run-action --wait octavia/0 configure-resources

have been created in Openstack as expected. I also double check that the Octavia controller can reach other resources on the IPv6 lb-mgmt-net.

root@juju-9e01a5-0-lxd-10:/var/log/octavia# ping fc00:8f36:122:78d8::1
PING fc00:8f36:122:78d8::1(fc00:8f36:122:78d8::1) 56 data bytes
64 bytes from fc00:8f36:122:78d8::1: icmp_seq=1 ttl=64 time=1.69 ms
64 bytes from fc00:8f36:122:78d8::1: icmp_seq=2 ttl=64 time=0.181 ms

root@juju-9e01a5-0-lxd-10:/var/log/octavia# ping fc00:8f36:122:78d8:f816:3eff:fe05:884c
PING fc00:8f36:122:78d8:f816:3eff:fe05:884c(fc00:8f36:122:78d8:f816:3eff:fe05:884c) 56 data bytes
64 bytes from fc00:8f36:122:78d8:f816:3eff:fe05:884c: icmp_seq=1 ttl=64 time=0.525 ms
64 bytes from fc00:8f36:122:78d8:f816:3eff:fe05:884c: icmp_seq=2 ttl=64 time=0.180 ms

I saw a similar post regarding that issue that suggesting creating an instance on the lb-mgmt-net so I tried that using the network, flavor ... (more)

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2019-08-30 15:08:32 -0500

HI stacks !

I have the same issue and it was driving me crazy. Until I've realized that I was using the master (train) and the variables by default that points to the train configurations. that means it was using the Train octavia.agent on the instances and ended up always in ERROR.

For example:

My env is Stein and I was using the agent for Train and give me the same errors.

Solution:

Set the correct variables as Jhon commented. To go easier go like this, export the variables required to set the versioning and the repo you want to use.

export DIB_REPOREF_amphora_agent=stable/stein
export DIB_REPOLOCATION_amphora_agent=https://opendev.org/openstack/octavia

Be sure to use the stable/ and not the origin/stable/ ones.

stable/ocata
stable/pike
stable/queens
stable/rocky
stable/stein

And normally with opendev repo will work

https://opendev.org/openstack/octavia

Good luck :P

edit flag offensive delete link more
0

answered 2019-08-28 14:18:21 -0500

johnsom gravatar image

Hi there. Thank you for the in depth information! Super helpful.

The worker log indicates that it was able to connect, but the amphora agent is not responding as expected.

I suspect what has happened is you are running an version of Octavia that is older than the master branch (train) with the master branch version of the amphora-agent.

The master branch required a version bump of the amphora agent API that is not compatible with older controllers. Older images are compatible with the new controllers.

You will need to get an amphora image that matches the release of your Octavia controllers.

One option is to build one: git clone https://opendev.org/openstack/octavia export DIB_REPOLOCATION_amphora_agent=<path above="" checked="" git="" octavia="" out="" repo="" the="" to=""> export DIB_REPOREF_amphora_agent=stable/stein diskimage-create.sh</path>

This will create an Ubuntu 18.04 image with the stable/stein amphora agent.

For more information on image building see the readme: https://opendev.org/openstack/octavia...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-08-24 12:16:40 -0500

Seen: 75 times

Last updated: Aug 30