Ask Your Question

johnsom's profile - activity

2019-09-08 14:12:48 -0500 answered a question Octavia LB flavor recommendation for Amphora VMs

For small deployments, the 1GB RAM, 1vCPU, 2GB disk (3GB with centos, etc) should work fine for you. You might even be able to drop the RAM lower if you will not be doing TLS. For example, my devstack amphora instance is allocated 1GB RAM, but is only using less than half that. (just because the flavor says 1GB it doesn't mean it uses all of that all of the time) Kernel page de-duplication will also help with actual consumption as the amphora images are mostly the same.

If you are doing really large numbers of connections, and you are logging the tenant traffic flows locally, you might want to increase the available disk. Normal workloads will be fine with a smaller disk as the amphora do include log rotation. If you do not need the flow logs, there is a configuration setting to disable them.

The main tuning you might want to do is setting the maximum amount of RAM it can consume. If you have a very large number of concurrent connections or are using TLS offloading, you might want to consider increasing the amount of RAM the amphora can consume. The HAProxy documentation states that it normally(non-TLS offload) uses around 32kB of RAM per established connection. You might start with that and see how that aligns to your application/use case.

In testing I have done, adding additional vCPUs has very little impact on the performance(a small bump with the second CPU as the NIC interrupts can be split from the HAProxy processes). You can get pretty high throughput with a single vCPU. We expect once HAProxy 2.0 stabilizes and is available (the distros are not yet shipping it), we will look at enabling the threading support to vertically scale the amphora by adding vCPUs. Versions prior to 2.0 did not have good threading and the multi-process model breaks a bunch of features. If you really need more CPU now, you can always build a custom image with 2.0.x in it and use the "custom HAProxy template" configuration setting to add the threading settings.

Now with Octavia flavors, you can define flavors that select different nova flavors for the amphora at load balancer creation. For example, you can have a "bronze", "silver", "gold", each with different RAM allocations.

We would also love to hear what you find with your deployment and applications.

2019-08-28 14:18:21 -0500 answered a question octavia controller cant connect to amphora

Hi there. Thank you for the in depth information! Super helpful.

The worker log indicates that it was able to connect, but the amphora agent is not responding as expected.

I suspect what has happened is you are running an version of Octavia that is older than the master branch (train) with the master branch version of the amphora-agent.

The master branch required a version bump of the amphora agent API that is not compatible with older controllers. Older images are compatible with the new controllers.

You will need to get an amphora image that matches the release of your Octavia controllers.

One option is to build one: git clone https://opendev.org/openstack/octavia export DIB_REPOLOCATION_amphora_agent=<path above="" checked="" git="" octavia="" out="" repo="" the="" to=""> export DIB_REPOREF_amphora_agent=stable/stein diskimage-create.sh</path>

This will create an Ubuntu 18.04 image with the stable/stein amphora agent.

For more information on image building see the readme: https://opendev.org/openstack/octavia...

2019-08-28 12:25:07 -0500 answered a question Octavia amphora curl Got a 404

One thing to note, the "Train" version of the amphora agent now is at version 1.0, so the URL would be https://192.168.0.15:9443/1.0/info There is also now a version discovery document at /.

Aside from that, did you figure out what is causing the amphora agent service to fail? We are not aware of any issues in the amphora-agent at this time.

2019-08-28 12:20:47 -0500 answered a question Octavia: Could not retrieve certificate when create HTTPS listener using application credentials

This depends on the version of Octavia you are running.

As of the Rocky release, Octavia will set the appropriate ACLs in barbican on behalf of the user.

If you are using an older version of Octavia, you will need to add the ACLs manually. This is documented in the Queens version of the Octavia documentation here: https://docs.openstack.org/octavia/qu...

2019-08-21 15:34:22 -0500 answered a question Octavia Health-check IP and Security group

Hi there, and thank you for the kind words.

You are correct that requests from the load balancer to the "member" servers comes from a dynamic IP address on the backend of the load balancer.

This allows users to add or remove members that are on both public and private subnets alike. When a user adds a member to the load balancer pool, we hot plug the network and subnet into the load balancer (If it is not already). This hot-plug process causes neutron to issue us an IP address on that subnet.

The other action that makes this tricky is the load balancer failover mechanisms, where should a load balancer have a failure, the Octavia controllers with replace it with a working load balancer. This applies to both standalone and active/standby load balancers. When this failover occurs, the source IP will change.

Currently we don't have a mechanism in Octavia that would allow you to set a security group on the member server ports that would restrict it down to only the load balancer source IP. I think there is an open bug for this use case, but I was unable to find it in storyboard. One proposal was to leverage FWaaS shared security groups, but this functionality has not yet landed in the FWaaS project.

There are a couple of workarounds until we come up with a solution to this: Put the members on a private network/subnet and firewall this at the router or restrict access to members of the subnet only. Since Octavia will plug a port into the network/subnet, it will be local behind this firewall. Add a security group to the members that only allows the member subnet access.

I hope that helps.

2019-08-21 10:07:14 -0500 answered a question ERROR neutron.db.metering.metering_rpc [req-4d900035-57dd-47ca-880c-0df380436e7a - - - - -] Unable to find agent on host

This is an RDO and/or neutron issue, so I am going to retag this.

2019-08-21 10:02:51 -0500 answered a question how to install and configure openstack octavia (LBaaS) using rdo poc?)

You will need to tag the neutron/OVN and/or RDO for this as Octavia has no limitation on using OVN L2.

2019-08-21 10:01:32 -0500 answered a question How to Install and configure Open stack LBaaS (Octavia) using rdo project with ovn neutron L2 backgroungd agent?

You will need to tag the neutron/OVN and/or RDO for this as Octavia has no limitation on using OVN L2.

2019-04-03 19:49:56 -0500 commented answer Octavia - instance is not reachable via the lb-mgmt-net

Yes, I have commented there. The network message is likely due to the SSL error since I is not able to successfully connect to the instance due to the SSL issue.

2019-04-03 19:47:02 -0500 commented answer OCTAVIA SSL ERROR

No, this error is clear (as openssl is with errors) that the client_ca.cert.pem file is bad or the client.cert-and-key.pem file is bad.

2019-04-03 19:44:51 -0500 answered a question How to disable failover fuction of amphora in Octavia?

Hello,

Yes, that setting would cause the health manager to not failover any instance for up to 604800 seconds.

We don't use AMQP for the health monitoring, so that should not impact this, but having the compute nodes down will certainly cause Octavia to attempt to repair the load balancer amphora on those nodes.

With the active/standby load balancers you should "fail safe" if it is unable to rebuild the failed amphora. This state will have the Load Balancer in provisioning status ERROR, but one of the two amphora instances will still be up and handling traffic. In the case of the node being down, it will resume operation once the compute node is brought back up (aside from TLS offloading as the secure content will likely be gone).

I have put in an RFE to make a more clear setting for this: https://storyboard.openstack.org/#!/s...

Michael

2019-04-03 19:27:29 -0500 answered a question Any document providing step by step Octavia installation on ubnutu?

Hi there, sorry to hear you are having trouble.

First I want to mention that neutron-lbaas is deprecated and scheduled to be retired this year. See the FAQ for more information: https://wiki.openstack.org/wiki/Neutr...

I have updated that page you referenced (I didn't even know it existed and it's very old/outdated) to reflect the status of neutron-lbaas.

Now, on to Octavia which is what you want.

All of the Octavia documentation is located here: https://docs.openstack.org/octavia/la...

There is an installation overview guide here: https://docs.openstack.org/octavia/la... You will also find helpful guides in this section: https://docs.openstack.org/octavia/la... Specifically the "Octavia Certificate Configuration Guide".

Unfortunately we have not yet been able to write a detailed install guide for doing it by hand.

A reference to the steps we execute to install in for our testing gates is the devstack plugin script here: https://github.com/openstack/octavia/...

There are also folks in the #openstack-lbaas IRC channel that can help answer questions.

I hope this helps, Michael

2019-02-19 10:08:37 -0500 answered a question lbaasv2 reference beetwen HM and Pool

Hi Gleb,

The quick answer to your question is yes, when you delete a pool it cascade deletes the health monitor as the health monitor is a child of the pool.

The current documentation is available here: https://docs.openstack.org/octavia/la... From this main page, you may find the API reference: https://developer.openstack.org/api-r... and the cookbook: https://docs.openstack.org/octavia/la... helpful.

Also note, neutron-lbaas will be retired this year: https://wiki.openstack.org/wiki/Neutr...

I hope this helps,

Michael

2019-02-11 11:10:12 -0500 answered a question OCTAVIA SSL ERROR

This implies that either your client_ca.cert.pem file is bad or the client.cert-and-key.pem file is bad.

Please double check those files against the instructions here: https://docs.openstack.org/octavia/la...

2019-01-23 09:31:43 -0500 answered a question Octavia - instance is not reachable via the lb-mgmt-net

Hi there,

From a configuration perspective, yes, the lb-mgmt-net ID goes into the amp_boot_netowrk_list configuration setting in the octavia.conf.

As for the TLS configuration, there is a guide that covers the only required configuration for this: https://docs.openstack.org/octavia/la...

The /var/lib/octavia/certs folder is inside the amphora instances and is fully managed by the controller processes. There is no manual configuration required there.

2019-01-07 15:34:44 -0500 answered a question octavia SSLError : BAD SIGNATURE

There were a few last minute typos in that patch that was under review. It has now merged here: https://docs.openstack.org/octavia/la...

2018-12-10 11:16:43 -0500 commented answer SSL BAD SIGNATURE octavia amphora

BAD_SIGNATURE is openssl saying it cannot validate the certificate that was presented to it. So either the cert being presented is bad/incorrect, or the CA certificate is not correct on the controller.

2018-12-03 18:13:29 -0500 commented answer SSL BAD SIGNATURE octavia amphora

Those are automatically filled in at amphora boot time when the configuration file is create. Those should not be set on the controllers.

2018-11-30 16:05:28 -0500 answered a question SSL BAD SIGNATURE octavia amphora

Yes, those certificates are installed into the amphora at nova boot time, so they will not be updated after a controller configuration change. A new amphora will need to be booted, either by rebuilding the loadbalancer, or using the amphroa failover API.

2018-11-30 16:03:56 -0500 received badge  Organizer (source)
2018-11-15 18:57:44 -0500 answered a question Could not connect to instance. Retrying.: SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)

This means that the server CA is mis-configured on your controllers.

[haproxy_amphora] server_ca and/or [certificates] ca_certificate is pointing to the wrong CA certificate.

I would go through the certificates document and double check your CA certificate configuration. https://review.openstack.org/#/c/613454/

Michael

2018-10-30 13:22:25 -0500 commented answer Could not connect to instance. Retrying.: ... [Errno 113] No route to host. provisioning_status stays on PENDING_CREATE

Yes, ACTIVE in nova just means it started the hypervisor, it does not mean the virtual machine has booted successfully. For the SSL issue, I have commented on the other Ask OpenStack topic for that issue. Please see https://review.openstack.org/#/c/613454/

2018-10-30 13:18:30 -0500 answered a question Could not connect to instance. Retrying.: SSLError: [SSL] PEM lib (_ssl.c:2554)

Without the rest of the warning message there, it's hard to tell what the exact SSL issue is, but I recently wrote a step-by-step certificates configuration guide for Octaiva that should address this issue.

https://review.openstack.org/#/c/613454/

There is some configuration issue with how the controller and amphora certificates are setup.

Michael

2018-10-30 10:50:05 -0500 received badge  Nice Answer (source)
2018-10-29 13:35:45 -0500 answered a question how to create an amphora image for octavia

Please see the README file available here: https://github.com/openstack/octavia/...

It explains the usage of the diskimage-create.sh script for building amphora images.

If you want a CentOS 7 amphora image, you would run the following command line:

diskimage-create -i centos -d 7 -s 3

When it finishes, you will have the image in the diskimage-create directory with the name of "amphora-x64-haproxy.qcow2"

Michael

2018-10-29 13:31:01 -0500 answered a question Could not connect to instance. Retrying.: ... [Errno 113] No route to host. provisioning_status stays on PENDING_CREATE

Here are a couple of thoughts:

  1. VirtualBox is VERY slow to boot service VMs. I would check the nova console logs to make sure your service VM has finished booting. While it is booting, Octavia will keep retrying to connect with the above message.
  2. Octavia does not have any dependency on the neutron ML2 selection. Linux bridge and OVS both work just fine.
  3. This initial network connection from the Octavia worker process to the amphora is over the "lb-mgmt-net", which is a neutron network setup by the operator to handle the command/control traffic to/from the service VMs. It uses the network configured in the "[controller_worker]amp_boot_network_list" setting. Check that this network is the network you intend the controller processes to communicate with the amphora over.

I suspect the lb-mgmt-network is not setup correctly or the configuration has the wrong network ID in it.

Michael

2018-10-04 17:24:04 -0500 answered a question octavia loadbalancer ip (198.51.100.1) is given by noop_driver

Oh, It looks like you have an un-configured Octavia deployment.

The noop or no-op drivers are there for testing. They accept requests, but don't actually provision any resources from the other services.

It might help to look at the configuration file from one of our gate tests: http://logs.openstack.org/24/604924/1...

This should give you an idea of the required settings. Note that many of the timeouts are excellently high for the gate tests and should be adjusted for your environment once you are done testing.

Michael

2018-10-04 17:20:13 -0500 commented answer openstack loadbalancer create .... give an ip address out of range of my network (198.51.100.1) !!

Oh. The no-op drivers are there for testing. They take no action against the other services. It might help to reference one of our gate test configuration files: http://logs.openstack.org/24/604924/1...

2018-10-02 13:23:08 -0500 answered a question openstack loadbalancer create .... give an ip address out of range of my network (198.51.100.1) !!

Using the command: "openstack loadbalancer create --project service --name lb1 --vip-subnet-id selfservice"

Would allocate the VIP address and ports from the "selfservice" subnet in neutron for the project "service". Since you did not ask for a specific IP address when creating the VIP, neutron allocates an IP address for us and provides it either on the port configuration or via DHCP.

If you got an IP address that is not on the neutron subnet "selfservice", please check your neutron configuration and logs to see why it issued Octavia such an address for the VIP port.

Also note, the VIP port should be visible in a port list to the project "service" you specified at load balancer creation time.

Michael

2018-10-02 13:16:54 -0500 answered a question Openstack LBaaS High Availability

LBaaS High Availability(HA) is provided by the Octavia project. Octavia can run with an HA control plane and can be configured to use an Active/Standby topology that provides highly available load balancers.

Octavia replaces the now deprecated neutron-lbaas project.

Michael

2018-10-02 13:13:42 -0500 answered a question how to configure octavia in queens

Agreed, Octavia is still in need of a detailed installation guide.

What we have at this point is an overview/quick-start guide: https://docs.openstack.org/octavia/la...

2018-09-05 11:39:29 -0500 answered a question load balancer my openstack service to AWS

Hi Rania,

If you are using Octavia for load balancing, you can specify member servers that are outside your openstack cloud. You just need to specify a subnet that has a route to those resources at member creation time.

Michael

2018-09-04 10:06:00 -0500 answered a question openstack queens octavia load balancer

Hi there,

I am not an RDO expert, so I'm going to take some guesses here.

From the look of the bottom error, it appears that the database migration did not occur. In basic devstack deployments we use a tool to do this migration:

octavia-db-manage --config-file /etc/octavia/octavia.conf upgrade head

Give that a try,

Michael

2018-07-20 13:16:31 -0500 received badge  Enthusiast
2018-07-19 11:46:53 -0500 answered a question octavia health manager bind host

Hi, Yes, if you are using the local port binding method to get access to the lb-mgmt-net for the controllers, you would do the same procedure on each controller to provide it a port on the lb-mgmt-net.

There are many ways to setup the lb-mgmt-net. Some folks use provider nets, other vxlan connections, etc. The OpenStack Ansible role uses a provider network. I will also note that you can use routes to access the lb-mgmt-net as well, it does not need to be a L2 network to function correctly.

Sorry we have not had the resource to write up the detailed installation guide with example network configurations yet, it will happen in the future.

Michael

2018-05-23 16:29:18 -0500 answered a question Problem connecting external host to vxlan

Yes, this is a common deployment topology connecting the lb-mgmt-net in neutron to your controller instances/hosts.

This does sound like a neutron / networking issue and not related to Octavia itself. Just off the top of my head here are a few things to consider: 1. Check your security groups and iptables configurations 2. Check you host's reverse path check settings in the kernel 3. Check the spanning tree configurations (if there are any)

Michael

2018-04-12 10:11:37 -0500 answered a question Is it possible to have Compute resources load balancing in Openstack ?

Octavia is a network load balancing project. With Octavia you can create a load balanacer that distributes network connections across backend servers.

More information here: https://docs.openstack.org/octavia/la...

2018-03-29 11:59:41 -0500 commented answer Octavia LBaaSv2 driver error

As we discovered today, the service account was out of quota causing the build to fail and the LB to go into error.

2018-03-26 23:01:55 -0500 answered a question Octavia LBaaSv2 driver error

Hi Huseyin,

Here are a couple of thoughts: 1. If you don't need neutron-lbaas, it is best to deploy without it and just use Octavia (Pike forward, but you can run octavia at newer versions than the rest of your cloud). 2. The error means some part of the system does not have a keystone token that allows it to access the required services. This could be because: a. The neutron API service is not configured to load the neutron-lbaas.conf. It has the config file list it loads on the command line you can see via "ps -ef", just make sure the way it is starting actually picks up the neutron-lbaas.conf. One way to test this is to just add the neutron-lbaas.conf content into your neutron.conf file. b. Make sure your octavia.conf has the v1 API enabled and the URL neutron-lbaas is using to access Octavia is via the v1 API endpoint (not advertised in keystone endpoints). This is a private API (should not be accessible from users) that neutron-lbaas uses to access Octavia. c. This appears to be the neutron client stack trace. Make sure your user credentials are enabled for the neutron client. For example, make sure you can do "neutron port-list".

Also, there are usually Octavia cores around in #openstack-lbaas on freenode IRC. Feel free to ask in the channel for help.

Michael (johnsom)

2018-03-13 10:20:52 -0500 answered a question Octavia Pike DVR HA floating ip problem

Thanks for tracking that down and opening a neutron bug for it! Others are likely to run into that as well. johnsom

2018-02-19 10:34:49 -0500 answered a question IMAP / SMTP / FTP load balancing

Hi there,

IMAP and SMTP can be configured using TCP listener, TCP Pool, and a TCP healthcheck. We do not yet have an IMAP or SMTP specific health monitor type, but it is on the roadmap.

FTP is a bit harder as it opens random ports.

2018-02-19 10:27:10 -0500 received badge  Supporter (source)
2018-02-19 10:24:57 -0500 answered a question healthmonitor multiple pools

This was a forward looking API design, but currently we do not have support for shared health monitors. It is a roadmap item.

2018-01-26 11:09:19 -0500 answered a question How to use tempest test neutron lbaas

From the top directory of neutron-lbaas, export TEMPEST_CONFIG_DIR=/opt/stack/tempest/etc (note: this should point to where your tempest config files are located. The path above assumes devstack) Then execute "tox -e apiv2" or "tox -e scenario"

Michael

2018-01-26 11:05:50 -0500 answered a question Neutron LBaaS - loadbalancer status and stats return null

I know there were some bugs in neutron-lbaas for stats/status and I'm not sure if those were resolved/impacting in Mitaka or not.

If you are using the reference driver, Octavia, I would check that the health heartbeat messages are making it back to the health manager controller process. Also check you octavia.conf file to make sure the [health_manager] event_streamer_driver is set to queue_event_streamer. This setting tells octavia to sync it's status/stats with neutron-lbaas.

Michael

2018-01-26 11:01:40 -0500 answered a question Allowed Address Pair Limit

This is a neutron quota and not an neutron-lbaas quota. I was not aware there is an AAP quota, I thought it was covered by the basic "ports" quota. My understanding is the quota is total accumulated "ports" and does not count the addresses on the port.

I would update the tags on this question and see if someone more familiar with the neutron quotas can confirm.

Michael

2018-01-26 10:56:45 -0500 answered a question modify default settings of lbaas haproxy

Hello,

For Octavia you can modify these in the jinja template at: octavia/common/jinja/haproxy/templates/base.j2 and restart your workers.

For the old namespace haproxy driver, you should be able to modify the template: neutron_lbaas/drivers/haproxy/templates/haproxy_base.j2 and restart both the neutron svc process and the lbaasv2 agents.

Michael

2017-11-29 20:18:10 -0500 answered a question lbaasv2-dashboard - TERMINATED_HTTPS greyed out

Hi there, 404 usually implies there is a mis-configuration somewhere such that a resource is not found. The dashboard uses the barbican client code to access barbican. That code is located here and pretty simple: https://github.com/openstack/octavia-...

Without more information and logs it's hard to pinpoint the issue exactly, but here are some things to investigate: 1. Check that "key-manager" is listed with the correct URL in the keystone endpoints configuration. "openstack endpoint list | grep key-manager". If that URL is incorrect It could cause the barbican client to return a 404. 2. Make sure the project you are using in the dashboard has access to barbican and that project has certificate containers and secrets. Sometimes folks will create these with a different project by accident.

Michael

2017-09-25 13:09:22 -0500 answered a question LBaaSv2 and Default Gateway

Octavia does support neutron "host-routes" if they are defined on the subnets. This might provide the functionality you are looking for with the multi-gateway subnets.

Octavia also supports "X-Forwarded-For" via the "insert-headers" capability: https://developer.openstack.org/api-r...

2017-06-01 11:28:19 -0500 answered a question volume attachment and LB

With LBaaSv2 you can have the same member server configured on multiple load balancers.