Ask Your Question

awalters's profile - activity

2020-06-01 08:18:41 -0500 received badge  Notable Question (source)
2020-05-29 16:00:49 -0500 received badge  Popular Question (source)
2020-05-29 14:26:19 -0500 answered a question Cannot launch instances anymore due to RabbitMQ errors

Well this was a fun one. This is a small test box and I wasn't running anything actively at the moment, so I didn't notice that a change that was done (not by me) to a firewall between this and the internet had broken the internet connectivity. Even though it was only trying to communicate with itself, via IP as far as I can tell, it must have been trying to do something with DNS at some point in the handshake (DNS was set to, which it couldn't reach). Restored connectivity and restarted services, and it's fine now.

2020-05-27 19:14:22 -0500 commented question Cannot launch instances anymore due to RabbitMQ errors

list_queues gives a very long list of queues, which are all at 0. Not sure if there's a specific one I should be looking at/for? scheduler.<my node="" name=""> is there. And cluster_status shows the node as running.

2020-05-27 15:08:30 -0500 asked a question Cannot launch instances anymore due to RabbitMQ errors

I used to be able to launch instances from this setup (Packstack single node install, Queens) with no problem. I haven't made changes to the network or anything within OpenStack in months. Now, for whatever reason, I can no longer launch instances - they get stuck in Scheduling. Looking in the nova-scheduler log, I can see rabbitmq errors, and the rabbitmq log is full of:

=ERROR REPORT==== 27-May-2020::15:58:26 ===

closing AMQP connection <0.17487.0> (<node's ip:47928="" -=""> <node's ip="">:5672):


I've tried restarting the service and even full on rebooting, I've reset the guest password to guest, and blown away mnesia - none of these made any difference at all.

Looking back at earlier logs, I can see these errors occasionally were in there as far back as I have logs (January), but they used to still work sometimes. But now it's just giving an error every time.

rabbitmqctl list_connections shows a number of running connections, but the last one in there shows in the logs as initiating 4 days ago. I've rebooted and restarted numerous times since then, so that seems interesting.

2020-04-09 01:14:49 -0500 received badge  Notable Question (source)
2020-01-17 15:02:27 -0500 received badge  Notable Question (source)
2020-01-17 15:02:27 -0500 received badge  Famous Question (source)
2019-12-14 08:36:18 -0500 received badge  Popular Question (source)
2019-12-13 11:39:18 -0500 commented answer Adding an iso to instance launch

Interesting...thanks! I'd wondered about doing it that way, but saw a post elsewhere that said not to directly use libvirt commands in OpenStack. Have you tried it this way personally? Any gotchas or odd potential behaviours to be aware of?

2019-12-12 16:43:33 -0500 asked a question Adding an iso to instance launch

I'm trying to launch an instance from an image, and also attach an iso to it that contains config data for that instance. I can do this via --block-device on a test box where I have Cinder installed. However, on my test box without Cinder (I don't actually need it for anything else in this test) this doesn't work. I tried changing the destination to local rather than volume and it says that combination is not supported. I tried switching to using a config-drive, and while this works with a small config file, it fails with the message "request is too large" when I include all of the info I actually need. Is there any other way to handle this short of installing Cinder? I just tried to add it on and got lots of errors (never added it after initial install before), and I don't need it for anything else, so I'm hoping to avoid going down this path if possible.


2019-11-14 23:09:29 -0500 received badge  Popular Question (source)
2019-11-13 13:36:20 -0500 asked a question Web proxy in OpenStack

I'm playing around with a web proxy in OpenStack, and it's not working when I turn IP spoofing on. I've disabled port security for the instance port, as well as ensuring that the openvswitch_agent config is set to use the NoopFirewallDriver. Iptables is disabled/inactive. This is a setup I've done outside of OpenStack with no issues, so the rest of the config should be fine. Is there anything else in OpenStack that would be preventing IP spoofing? This is in Queens, FYI.

2019-11-01 11:21:38 -0500 received badge  Famous Question (source)
2019-10-31 21:00:34 -0500 answered a question Mapping physical and virtual networking

I was able to do this with a trunk port and a single (management) IP on the CentOS box by creating two OpenStack bridges - I made br-ex for all networks other than the management network, set all of the associated OS networks up as VLAN, and attached br-ex to the physical port/team. I then made br-mgmt, put the management network on that as a flat network, and attached it to a tagged subinterface of the physical port/team. I gave br-mgmt the IP for the box, and didn't put an IP at all for br-ex. Seems to work fine, even if it was a pain to figure out. I'm curious if this is how people generally solve this (cause I didn't find instructions for this anywhere), or what else people do instead, but...what I need is up and working, anyway.

As mentioned above in the comments, I made the team in Linux rather than OVS due to a recommendation from the RHEL guide.

2019-10-18 10:53:28 -0500 commented question Timeouts when trying to create image remotely

Command is 'openstack image create --disk-format qcow2 --public --file <path> <name>' - the same command that works when I run it directly on the box. I didn't change any config for glance - it appears to have file,http,swift as the store list, and file as the default.

Queued makes sense, thanks.

2019-10-17 16:59:26 -0500 asked a question Timeouts when trying to create image remotely

I'm trying to create images in OpenStack. If I do it with a file that's directly on the box, it works fine. However, if I do it with a remote file (either through Horizon, or via command line from another box) it times out after about (but not exactly, or the same amount of time) 12 minutes. I've tried it with a small image - only about 150MB and still have the same issue. I've upped the timeouts in /etc/openstack-dashboard/local_settings and /etc/keystone/keystone.conf to 24 hours and restarted services, with no effect. I can run other commands from the remote box and do other things within Horizon, so I'm connected and authed. (Also, using main admin user.) Interestingly, it fails faster from Horizon - only about three minutes or so. The GUI just gives the message "Unable to create the image".

There are no errors or warnings in the glance.api log other than the timeout (Failed to upload image data due to internal error: error: [Errno 110] Connection timed out).

After the failed attempts from the command line, an image shows up in Horizon with just a GUID for a name, no size, format RAW, and status queued. When it fails from Horizon itself, nothing new shows there.

Are there other timeout settings I should be looking for? Or something else?

2019-10-16 11:00:26 -0500 received badge  Famous Question (source)
2019-10-16 10:59:59 -0500 commented question Mapping physical and virtual networking, what seems to be working is this - have the main trunked interface with one OVS bridge, and a VLAN tagged subinterface with a second OVS bridge (and use that one for the VMs that need to access the management VLAN that CentOS is on). Not sure if this is ideal, but only way I could find...

2019-10-16 10:45:24 -0500 received badge  Enthusiast
2019-10-15 15:11:13 -0500 received badge  Commentator
2019-10-15 15:11:13 -0500 commented question Mapping physical and virtual networking

Got this working with a trunk port with the native vlan set on the switch to be the management vlan (had to change the OpenStack network for that management vlan to flat - this made all traffic on management untagged). However, that's not ideal. How do I get it working trunked, but no native vlan?

2019-10-11 10:38:34 -0500 received badge  Notable Question (source)
2019-10-04 17:03:19 -0500 received badge  Popular Question (source)
2019-10-04 12:12:35 -0500 commented question Mapping physical and virtual networking

re: bonding, seeing in a RHEL guide a suggestion to use OVS bonding for networks that interact with OS VMs, and Linux bonding for control or storage networks "because OVS bonds carry the potential for control plane disruption that can occur when OVS or the neutron agent". Seems to make sense...

2019-10-04 12:09:52 -0500 commented question Mapping physical and virtual networking

So I've got an answer to the second question - it's a trunk port. That makes 4 irrelevant. Not seeing anything in that guide about how to implement teaming/bonding, though. Also not clear (though I'll be going through it more) about what network the CentOS install itself has an IP on, and tagging

2019-10-04 11:49:39 -0500 commented question Mapping physical and virtual networking

Hmmm....I saw that in my searches, but since it explicitly states that it's not how the manuals say to do it multiple times, I wondered how common it was. :-) And also how relevant, because of the age (as you mention). Thanks! I don't think it answers all my questions, but certainly some.

2019-10-03 15:53:43 -0500 asked a question Mapping physical and virtual networking

I'm trying to understand the best way to set up physical networking for OpenStack, and having some troubles getting my head around how to navigate the possible options. I'm just working with a single node at the moment, so not even worrying about communication between nodes yet.

So, let's assume this is running on CentOS, although obviously that shouldn't matter much to the answers, but it's easiest to call it something. For the VMs, I want them to be able to access the internet over, let's say vlan 20 (and let's say it's 192.168.1.x), and an office network on vlan 30 (10.10.10.x). There is also a server/management vlan for the office, 40 (172.16.1.x).

  • On the CentOS box, I'd assume it generally should have just one IP, in the management vlan (172.16.1.x), right?
  • Would the physical networking to the box generally be a single (presumably teamed) trunk port? Or one for each vlan (so in this case 3)?
  • If it's a trunk port, how does the CentOS box itself get its traffic tagged, so it can communicate?
  • Or if it's one for each vlan, how does that work for the gateway on CentOS, if these vlans maybe don't all have routing between them?
  • For teaming, is it best to do that in linux or with ovs tools? (Adding bonds using ovs-ofctl, or...?)

I've tried various permutations of all of these and have had various issues. I want to concentrate on solving the ones that lie in the way of my best path, not just every possibility I come across.

ETA: To further elaborate on questions 1 and 3: currently the CentOS box has an IP on the management vlan (40). I also want one of the VMs to be able to communication with this vlan. However, it is currently not working. Other external network communication is working, but not this one.

When I examined the traffic, I discovered that the issue is this - when a request from a VM goes out on vlan 20 (well, whatever OpenStack's internal id is, which the a flow in br-ex then mods to 20), I can see the packet on br-ex, then on the physical port, with vlan 20. The reply then comes back from the switch to the physical port on vlan 20, then to br-ex, then to br-int which flips it back to the internal id. This is all fine. But with vlan 40, since CentOS is on it as well, there's the extra complication of tagging on the CentOS packets as well.

When I first installed CentOS I was trying to do this with a sub-interface. This doesn't work with OpenStack since then the vlan on the subinterface doesn't match some of the VM traffic. So I then set the trunk port on the switch to have the same native vlan as ... (more)

2019-10-03 11:22:43 -0500 commented question Instance traffic not returning on external networks

Not the root, as it turns out. Matched my physical networking to theirs, and still same problem. Edited the post to reflect.

2019-10-02 12:11:32 -0500 commented question Instance traffic not returning on external networks

(Previous comment moved here from post) As a follow up, I got someone to look at a working box, and they see this same behaviour. So, weird, but not the pointer to the issue. Turns out though that their trunk port has a native vlan, and mine doesn't, so I'm guessing that's the root of the issue.

2019-10-02 12:09:57 -0500 commented question Instance traffic not returning on external networks

Figured out how best to trace the traffic flow. I can see that my requests are coming into the outer box via the qvo interface, then to the br-int, then to br-ex, then to the physical int and out to the gw. The replies are coming back from the gateway, then to br-ex, but then stopping before br-int.

2019-10-02 09:24:02 -0500 received badge  Notable Question (source)
2019-10-02 08:09:18 -0500 received badge  Popular Question (source)
2019-10-02 01:52:34 -0500 received badge  Popular Question (source)
2019-10-01 13:00:31 -0500 edited question Instance traffic not returning on external networks

I have an all-in-one Queens Packstack (aware that it's almost out of support, but trying to match something existing for testing) install that I'm trying to get networking working on. I have an internal/tenant network that works fine, but I'm also trying to configure two external networks via OVS and neither are working at all - they cannot ping or communicate out over either external network in any way.

I'm only trying to do outbound traffic, not directly connect to any instance from outside, so it's my understanding that floating IPs are not needed. I've followed the instructions at access., as they seemed the clearest and were recommended to me. As per the setup/troubleshooting sections, I've confirmed:

  • The physical network name is used consistently throughout
  • The networks are external, with the VLAN network type (and this is also in the openvswitch config file), and the correct VLAN ID br-int and br-ex are connected via patch-peer
  • "Dump-flows" shows that packets are coming in via the expected port, and the OpenStack internal VLAN ID is being modified to the correct external VLAN IDs
  • My ifcfg-br-ex and ifcfg-<physicalport> files are setup as per the instructions. Also, external connectivity from the CentOS box (the outer container where PackStack is installed) is working properly
  • My physical port shows as UP in "ip a", and is marked with master ovs-system

The only thing that is off from the troubleshooting section is that the br-ex port is showing as state UNKNOWN in "ip a", though I've seen other references to this happening and still having normal operation. Not sure what to do about it, either way...

Thought it might be firewall related, but added accept all rules in iptables as a test and still no luck. Does anyone have any other ideas about where I could look or what I could try? Been bashing my head against this for quite some time and I'm pretty out of ideas at this point. Thanks!

ETA: I've now reduced the complexity of my physical networking to match another box that I know works (I don't have access to it directly, sadly). I now have only a single physical interface, trunked, with the native vlan matching the vlan of the outer CentOS box. I deleted all networks and created just one. Still the same problem. I can see the requests on the qbr interface, then on br-ex (where it's now translated to my external vlan id), then the physical int. I can see the reply come back on the physical, then on br-ex, but it's not making it to the qbr.

2019-10-01 13:00:31 -0500 received badge  Editor (source)
2019-09-30 19:21:16 -0500 marked best answer Is it possible to generate a packstack answer file from existing install?

Title pretty much has the question. I have an existing Packstack install, which started with an answer file, and I've then made modifications afterwards. Is it possible to go backwards and generate a new answer file that would reflect the settings in the existing install? i.e. say I wanted to set up a second all-in-one install identical to the first, for testing. Or as another example, the best way to compare two different installs and diff the settings.

If not, which is looking to be the case - any thoughts on the best way to do something like this? Reproduce an existing environment or comparing two environments? I know there are conf files that could be moved/diff'd, but I'm not sure how to ensure I have all of the correct/appropriate ones, etc...