Ask Your Question

eblock's profile - activity

2020-07-23 05:25:25 -0500 commented question tripleo cluster failed

Have you any monitoring enabled to see the load on the nodes? I’ve seen a control node acting out pretty hard (OOM Killer) where openvswitch used RAM heavily. Although it could have been a bug fixed by updates I’d still take a look at that.

2020-07-23 04:45:24 -0500 commented question Cinder HTTP 503 : Keystone unavailable

Can you also share your endpoint list? I think there might be missing some v2.1 or v3 or something in the keystone url. Have you checked that it’s correct and maybe compared to other services like nova? Also the typo mentioned by @Bernd should definitely be fixed.

2020-07-22 03:25:35 -0500 commented question VM build stuck in "scheduling" task

Are the instances volume based? Is cinder responding? It also could be related to neutron when it fails to create ports on the compute node. Check all related log files.

2020-07-21 16:21:40 -0500 commented question How to Install OpenStack Keystone?

You can’t just pick one service without reading the docs and learn about the dependencies. The doc you mention tells you to install the database and messaging service after the packages. Those are the basics for all other services, since every service needs to store data and to communicate.

2020-07-18 08:12:35 -0500 commented question Instance cannot be started properly from a snapshot

I assume some network related stuff missing, e.g. the instance doesn’t receive a proper IP address and can’t start the required services. Or maybe some attached disk is missing? Can you tell at which point the recovery screen appears?

2020-07-15 04:09:57 -0500 commented question how to setup controller node HA with all services

By the way, I had posted a question 2 years ago about upgrading a non-ha environment to ha. Maybe some of the answers will help you, too.

2020-07-15 04:08:01 -0500 commented question Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'

Yeah that makes sense.

2020-07-15 02:15:49 -0500 answered a question Upgrade single-control environment to high availability

We finally got it working, thanks again @Peter Slovak for your thoughts! I'll share the key aspects of the setup:

We created a pacemaker cluster consisting of two control nodes, that's quite straight forward. The tricky part is the lacky documentation of openstack HA, I read all kinds of deployment guides and blog posts to get the main concepts.

  • Database: Galera mariadb with a third tiebreaker node (not part of pacemaker, only running garbd).
  • Neutron: We needed to switch from linuxbridge to openvswitch (we don't use dvr for now). The solution is to change the neutron table networksegments and replace/set the appropriate values for network_type, physical_network and segmentation_id to fit the actual (new) setup. For example, we changed all self-service networks from typ vlan to vxlan and remove the physical_network since it all is supposed to be handled by ovs. The provider networks got their br-provider entry (we just called it provider) and now are of type vlan instead of flat.
  • Migration: To actually migrate an instance to the new environment (luckily our storage backend is Ceph) we shut it down in the old cloud, change its host and node in nova.instances table and then run a nova reboot --hard <UUID> in the new cloud. This creates a new virsh xml configuration on a new compute node, but since the old network config is still present in nova.instance_info_caches the compute node builds a linuxbridge device (brq...) instead of an ovs interface (qbr...). In this case it can help to simply shutdown and restart the instance, but mostly that didn't work. The easiest way is to detach and re-attach the respective interface(s) of that instance. This triggers the compute node to recreate the instance's interface(s) with ovs. After restarting the VM it should now be reachable with it's designated IP.
  • Router: Be aware that you need to shutdown (disable) routers in the old environment to prevent the old control or network node to respond to requests. Otherwise this will lead to unreachable floating IPs for instances in self-service networks.

This did the trick for us, we are currently in the migration process (one by one) and have both environments active, in case anything breaks we still could move the already migrated instances back to the old cloud. But it's working for now and we're quite happy with it. We use OpenStack Train on openSUSE Leap 15.1 in the new environment.

2020-07-15 01:46:37 -0500 commented question Memory sharing concept

We use ksm only for memory sharing, it seems to work quite well for some time now. What exactly is your question? Is ksm not enough?

2020-07-15 01:45:12 -0500 commented question How to migrate openstack instance from one cluster to another?

It sounds like your services on cluster1 are not running. Check rabbitmq and glance services, and cinder services for completeness sake.

2020-07-14 05:23:43 -0500 commented question How to migrate openstack instance from one cluster to another?

There's a cli command (also an option in horizon) to upload volumes to glance: cinder upload-to-image <volume_id>.

2020-07-14 05:22:37 -0500 commented question How to migrate openstack instance from one cluster to another?

I mean the same snapshot you mean ;-) You have several options to get to your goal. Either directly upload your volume to glance and then copy it wherever you want. Or you create a snapshot from that volume, create a new volume from said snapshot and upload that new volume to glance.

2020-07-14 02:39:32 -0500 commented question Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'

Do you have ceph-common packages installed on the node? Is glance-api service running?

2020-07-14 02:38:48 -0500 edited question Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'

Using Openstack Train with Ceph Mimic

Unable to upload images.

2020-07-13 20:50:44.013 111185 ERROR glance.api.v2.image_data [req-6e200f71-0bb9-4cfc-ba32-ecd46ea2ee9f 4e8e4991ffd14330aa92bc0058              5fa196 ae1c993d8ea94236a171b1758189b24d - default default] Failed to upload image data due to internal error
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi [req-6e200f71-0bb9-4cfc-ba32-ecd46ea2ee9f 4e8e4991ffd14330aa92bc00585fa196               ae1c993d8ea94236a171b1758189b24d - default default] Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'No              neType' object has no attribute 'Rados'
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi Traceback (most recent call last):
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1474,               in __call__
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     request, **action_args)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1517,               in dispatch
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     return method(*args, **kwargs)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 417,               in wrapped
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line               299, in upload
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     self._restore(image_repo, image)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220,               in __exit__
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     self.force_reraise()
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196,               in force_reraise
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     raise value
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line               164, in upload
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/domain/proxy.py", line 198,               in set_data
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/glance/notifier.py", line 485, in s              et_data
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220,               in __exit__
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi     self.force_reraise()
2020-07-13 20:50:44.046 111185 ERROR glance.common.wsgi ...
(more)
2020-07-14 02:38:24 -0500 commented question How to migrate openstack instance from one cluster to another?

No, not that I'm aware of. You can create a volume from that snapshot, that should be flat (I believe). And that volume can be uploaded to glance so you can download it to local storage.

2020-07-13 03:29:50 -0500 commented question How to migrate openstack instance from one cluster to another?

It sounds to me that your instances are volume-based, what is your cinder backend? Cinder snapshots have a size of 0, you need to make them flat (in terms of ceph if that's your backend), you can also upload volumes to images and then download them. Please provide more information about your backend

2020-07-12 13:24:44 -0500 commented answer ERROR configuring keystone_init_tasks

You might have more luck in the openstack mailing list. Have you tried it there?

2020-07-10 08:19:05 -0500 commented question how to setup controller node HA with all services

In short, I have 2 control nodes with haproxy and virtual ip, where pacemaker is used to manage all cluster resources although that's not necessary for stateless services like nova-api, cinder-api etc. Our database is galera-mariadb with one tiebreaker node not managed by pacemaker.

2020-07-10 08:16:33 -0500 commented question how to setup controller node HA with all services

I've been building a HA environment successfully with the help of different guides, unfortunately the openstack ha guide mentioned by @Bernd is nothing more than an inspiration. This may also help as inspiration.

2020-07-09 01:59:47 -0500 commented question WARNING nova.virt.libvirt.driver

Check your neutron services and logs. I have faced the same issue just hours ago where apparmor prevented neutron from creating interfaces.

2020-07-09 01:54:55 -0500 edited question WARNING nova.virt.libvirt.driver

Unable to provision VM on Openstack Train Ubuntu with LinuxBridge

2020-07-08 14:49:12.051 24902 INFO nova.compute.claims [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Claim successful on node openstack-comp01
2020-07-08 14:49:12.483 24902 INFO nova.virt.libvirt.driver [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Creating image
2020-07-08 14:49:15.742 24902 INFO os_vif [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:88:11:61,bridge_name='brq498b4cf9-10',has_traffic_filtering=True,id=e7e794c3-aa73-4b4e-a42f-83f23618faee,network=Network(498b4cf9-1050-44cc-adb6-6c6d055f0ebd),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tape7e794c3-aa')
2020-07-08 14:49:17.012 24902 INFO nova.compute.manager [req-a370a53b-9c7e-4b97-aecc-d96d13240c00 - - - - -] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] VM Started (Lifecycle Event)
2020-07-08 14:49:17.055 24902 INFO nova.compute.manager [req-a370a53b-9c7e-4b97-aecc-d96d13240c00 - - - - -] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] VM Paused (Lifecycle Event)
2020-07-08 14:49:17.134 24902 INFO nova.compute.manager [req-a370a53b-9c7e-4b97-aecc-d96d13240c00 - - - - -] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] During sync_power_state the instance has a pending task (spawning). Skip.
2020-07-08 14:54:17.013 24902 WARNING nova.virt.libvirt.driver [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Timeout waiting for [('network-vif-plugged', 'e7e794c3-aa73-4b4e-a42f-83f23618faee')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds
2020-07-08 14:54:17.418 24902 INFO os_vif [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:88:11:61,bridge_name='brq498b4cf9-10',has_traffic_filtering=True,id=e7e794c3-aa73-4b4e-a42f-83f23618faee,network=Network(498b4cf9-1050-44cc-adb6-6c6d055f0ebd),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tape7e794c3-aa')
2020-07-08 14:54:17.419 24902 INFO nova.virt.libvirt.driver [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Deleting instance files /var/lib/nova/instances/dc68824b-fca9-4294-9f11-fea08e5001a6_del
2020-07-08 14:54:17.419 24902 INFO nova.virt.libvirt.driver [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Deletion of /var/lib/nova/instances/dc68824b-fca9-4294-9f11-fea08e5001a6_del complete
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [req-5e6f9494-5534-40fd-b8b7-5d170098b4a9 4e8e4991ffd14330aa92bc00585fa196 ae1c993d8ea94236a171b1758189b24d - default default] [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6] Traceback (most recent call last):
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6234, in _create_domain_and_network
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]     network_info)
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]     next(self.gen)
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 478, in wait_for_instance_event
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]     actual_event = event.wait()
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]   File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]     result = hub.switch()
2020-07-08 14:54:17.512 24902 ERROR nova.compute.manager [instance: dc68824b-fca9-4294-9f11-fea08e5001a6]   File "/usr/lib/python3/dist-packages/eventlet ...
(more)
2020-07-08 03:23:03 -0500 edited question Kolla zun ceph storage

Is there a way to configure zun to use ceph storage for containers instead of local disks? I don't see anything in the zun.conf to enable this nor does kolla-ansible say anything about copying ceph.conf and keys to the zun-compute or zun-api containers.

Nova, Cinder, and glance are working correctly with ceph. Running version Train on Centos 8 with Docker-ce 19.03.11

Error when creating Zun container with newly created cinder volume:

Unexpected error while running command. Command: rbd map volume-971719e3-85a9-4c80-9812-f3e61d7e8925 --pool volumes --id cinder --mon_host 172.30.0.11:6789 --mon_host 172.30.0.12:6789 --mon_host 172.30.0.13:6789 Exit code: 22 Stdout: 'In some cases useful info is found in syslog - try "dmesg | tail".\n' Stderr: "did not load config file, using default settings.\n2020-07-08 00:31:47.494 7fb221522080 -1 Errors while parsing config file!\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.494 7fb221522080 -1 Errors while parsing config file!\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.494 7fb221522080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory\n2020-07-08 00:31:47.527 7fb221522080 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\nrbd: sysfs write failed\n2020-07-08 00:31:47.535 7fb221522080 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2020-07-08 00:31:47.535 7fb221522080 -1 AuthRegistry(0x5557852c0240) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2020-07-08 00:31:47.536 7fb221522080 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory\n2020-07-08 00:31:47.536 7fb221522080 -1 AuthRegistry(0x7ffe92593908) no keyring found at /etc/ceph/ceph.client.cinder.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx\n2020-07-08 00:31:47.538 7fb221522080 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication\nrbd: couldn't connect to the cluster!\nrbd: map failed: (22) Invalid argument\n"

Thanks!

2020-07-06 08:59:36 -0500 commented answer Openstack Stein (OSA)

I'm not able to help with that, unfortunately. I've never dealt with the iptables or the NIC mode, but it couldn't hurt to share some more details. Note, you can always edit your question, you don't have to create a new answer (especially since it wouldn't be an actual answer).

2020-07-06 06:49:04 -0500 commented answer Openstack Stein (OSA)

Yes, definitely, without rabbitmq the services can't communicate with each other. But that should be visible in the logs of nova, neutron etc.

2020-07-06 04:39:40 -0500 commented answer Openstack Stein (OSA)

Could it be a security issue? If apparmor is enabled you might get errors, but I understand you already checked all logs. Anyway, it wouldn't hurt to rule that out.

2020-07-06 02:34:31 -0500 commented answer Openstack Stein (OSA)

I agree with @Bernd, before debugging metadata you should figure out why the instances don't get an address. Seeing the IP in Horizon only means that openstack assigned one, not that the instance actually got one. Is DHCP enabled on the subnet the instance is launched in? Check neutron-dhcp logs.

2020-07-04 03:10:41 -0500 commented answer Openstack Stein (OSA)

You can check out our blog article, maybe the route part is helpful. Check if your image has a route for the metadata IP: netstat -rn.

2020-07-04 03:07:54 -0500 edited answer Openstack Stein (OSA)

Hi eblock, my apologies for the lack of information, I have now updated my question and thanks for the reply. Yes I've been searching for days now and all the posts where it has been resolved I have checked my config and it was already there.

+--------------------------------------+--------------------+------------------------+-------------------+-------+-------+------------------------
---+
| ID                                   | Agent Type         | Host                   | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+------------------------+-------------------+-------+-------+---------------------------+
| 091349c2-6bae-41d5-92fb-9eb832217788 | Metering agent     | cloud2020l4-controller | None              | :-)   | UP    | neutron-metering-agent    |
| 17aed236-8e91-4dd1-9212-aa751ef6790a | Metadata agent     | cloud2020l4-controller | None              | :-)   | UP    | neutron-metadata-agent    |
| 1d108d6c-42e5-439d-8dbc-ed0b21d61cd5 | L3 agent           | cloud2020l4-controller | nova              | :-)   | UP    | neutron-l3-agent          |
| 4c5c7fa0-7be9-4561-908d-d393068a4af9 | Linux bridge agent | cloud2020l4-compute1   | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 54c79dbb-2807-4183-bfa7-1bf24121ce46 | Linux bridge agent | cloud2020l4-compute2   | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 85aa94d3-b1a8-47bd-9bc2-d5a6352ebfb3 | DHCP agent         | cloud2020l4-controller | nova              | :-)   | UP    | neutron-dhcp-agent        |
| e269d937-30b5-4883-acda-938db426dccd | Linux bridge agent | cloud2020l4-controller | None              | :-)   | UP    | neutron-linuxbridge-agent |
+--------------------------------------+--------------------+------------------------+-------------------+-------+-------+---------------------------+

DHCP

DHCP configuration for ports set([u'6a48a683-c1e3-4eba-848c-53efe1450eae']) is completed. fixed_ips=[{u'subnet_id': u'0a917da2-3446-47a0-b66d-72c572120daa', u'ip_address': u'MY_PRIVATE_IP'}],
tag:tag0,option:classless-static-route,169.254.169.254/32,MY_PRIVATE_NETWORK_FIRST_ALLOCATED_IP,0.0.0.0/0,MY_PRIVATE_NETWORK_GATEWAY tag:tag0,249,169.254.169.254/32,MY_PRIVATE_NETWORK_FIRST_ALLOCATED_IP,0.0.0.0/0,MY_PRIVATE_NETWORK_GATEWAY tag:tag0,option:router,MY_PRIVATE_NETWORK_GATEWAY

Update:

I have compared my install to a fellow colleagues install (he didn't use ansible, we are testing at the moment) in /var/lib/neutron/dhcp/ID/leases he has an entry for every instance he has spun up through horizon, my leases file is empty so maybe this might help in pointing to my issue, I will investigate a bit more in the morning but if anyone has any advice I'd appreciate it.

TCPDUMP (compute br-vlan)

09:25:04.458939 fa:16:3e:06:c4:13 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 441, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:06:c4:13, length 300, xid 0xf93c270d, secs 121, Flags [none] (0x0000)
  Client-Ethernet-Address fa:16:3e:06:c4:13
  Vendor-rfc1048 Extensions
    Magic Cookie 0x63825363
    DHCP-Message Option 53, length 1: Discover
    Client-ID Option 61, length 7: ether fa:16:3e:06:c4:13
    MSZ Option 57, length 2: 576
    Parameter-Request Option 55, length 9: 
      Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
      Domain-Name, MTU, BR, NTP
      Classless-Static-Route
    Vendor-Class Option 60, length 12: "udhcp 1.29.3"
    Hostname Option 12, length 6: "cirros"

TCPDUMP (controller lxcbr0 & nova-api-container eth0)

09:24:21.715134 fe:05:91:48:23:de > 00:16:3e:0e:fb:39, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 32644, offset 0, flags [none], proto UDP (17), length 328)
10.0.3.1.67 > 10.0.3.230.68: [bad udp cksum 0x1c2c -> 0x9d6c!] BOOTP/DHCP, Reply, length 300, xid 0x5463e7c8, secs 11797, Flags [none] (0x0000)
  Client-IP 10.0.3.230
  Your-IP 10.0.3.230
  Server-IP 10.0.3.1
  Client-Ethernet-Address 00:16:3e:0e:fb:39
  Vendor-rfc1048 Extensions
    Magic Cookie 0x63825363
    DHCP-Message Option 53, length 1: ACK
    Server-ID Option 54, length 4: 10.0.3.1
    Lease-Time Option 51, length 4: 3600
    RN Option ...
(more)
2020-07-03 07:43:39 -0500 commented question Openstack Stein (OSA)

What is the output of openstack network agent list? Did you search a little bit on your own? There's plenty of possible solutions, it would be useful to know what you have tried to resolve it yourself and where it failed.

2020-07-02 02:30:06 -0500 commented answer Cannot delete a snapshot after using it to create a volume

I'm not familiar with magnum, how would you change other config settings for different services?

2020-07-01 06:31:04 -0500 answered a question Cannot delete a snapshot after using it to create a volume

In an openstack environment with ceph backend you could let cinder flatten the volume automatically during creation, that way the volume becomes independent from the snapshot which can be deleted afterwards. The cinder config is this one, and you would need to set it to true:

# Flatten volumes created from snapshots to remove dependency from volume to
# snapshot (boolean value)
#rbd_flatten_volume_from_snapshot = false

I don't know how that works with other cinder backends, though.

2020-07-01 02:01:12 -0500 commented question Openstack Fresh Install Train - Neutron

To me it sounds as if your package versions don't match, try to make your environment consistent.

2020-06-29 04:52:11 -0500 commented question Is it possible to save a compute instance from my devstack cloud to hard drive so that I can then upload it to a production cloud?

Creating snapshots from volume based images result in 0 byte glance images, that's expected. The second approach would have been my recommendation, what is your storage backend for cinder? How did the second approach fail?

2020-06-29 02:03:54 -0500 commented answer Error: Unable to retrieve instances

As already mentioned, your neutron database seems to be corrupted. Either fix it manually or import a backup, otherwise neutron won't start. Please only manipulate the database if you really know what you're doing! And before doing so, save another backup of the current state to be able to roll back

2020-06-25 09:12:39 -0500 commented question image name while launching the instance from volume

test instance:

control1:~ #  nova show t2 | grep image
    | image                                | Attempt to boot from volume - no image supplied                  |
2020-06-25 09:12:12 -0500 commented question image name while launching the instance from volume

Can you share more information about your openstack version etc.? If I create a volume from image in Ocata the volume has volume_image_metadata: {... 'disk_format': 'raw', 'image_name': '###_SALT_...}, visible in openstack volume show. The nova instance does not have that information in my case.

2020-06-25 06:20:43 -0500 edited question Error: Unable to retrieve instances

Hi,

We're using OpenStack with just 3 nodes at the moment (1 Horizon, 1 Neutron and 1 Nova). Installation was done via Packstack.

Things were working well (although some of the errors in Horizon were already there) until a power loss left us in this state:

In Horizon, when trying to list the Instances, we get:

Error: Unable to retrieve instances.

Something "similar" happens, for example. Under Volumes:

"Error: Unable to retrieve volume/instance attachment information"

Under Overview:

"Error: Unable to retrieve attachment information." and "Error: Unable to retrieve network quota information."

This happens in several projects (all where there are some instances).

After trying to look in logs and the several servers, I've noticed that neutron-server service is not running and doesn't start:

[root@neutron-00 ~]# systemctl start neutron-server.service
Job for neutron-server.service failed because the control process exited with error code. See "systemctl status neutron-server.service" and "journalctl -xe" for details.

[root@neutron-00 ~]# systemctl status neutron-server.service
● neutron-server.service - OpenStack Neutron Server
   Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled; vendor preset: disabled)
   Active: activating (start) since Wed 2020-06-24 16:11:33 BST; 3ms ago
 Main PID: 28634 (neutron-server)
   CGroup: /system.slice/neutron-server.service
           └─28634 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutr...

I've tried starting some instances via CLI, but can't even see the instances (I know they are there, since I've queried the nova MySQL database and I can see them there):

# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

 # neutron net-list
Unable to establish connection to http://xx.xx.12.10:9696/v2.0/networks.json: HTTPConnectionPool(host='xx.xx.12.10', port=9696): Max retries exceeded with url: /v2.0/networks.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2a1cdd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

In Neutron:

 # tail /var/log/neutron/server.log 
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2020-06-24 16:15:58.667 31726 ERROR neutron     util.raise_from_cause(newraise, exc_info)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2020-06-24 16:15:58.667 31726 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2020-06-24 16:15:58.667 31726 ERROR neutron     context)
2020-06-24 16:15:58.667 31726 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2020-06-24 16:15:58.667 31726 ERROR neutron     cursor.execute(statement, parameters)
2020-06-24 16:15:58.667 31726 ERROR neutron DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations [SQL: u'SELECT ml2_geneve_allocations.geneve_vni AS ml2_geneve_allocations_geneve_vni, ml2_geneve_allocations.allocated AS ml2_geneve_allocations_allocated \nFROM ml2_geneve_allocations']
2020-06-24 16:15:58.667 31726 ERROR neutron ...
(more)
2020-06-25 00:39:23 -0500 commented question Error: Unable to retrieve instances

You can’t see instances because nova doesn’t start, same goes for the other services because they all rely on the database. You write you could query the db but the errors indicate that it’s not running or in an inconsistent state after the outage.

2020-06-24 19:12:40 -0500 received badge  Nice Answer (source)
2020-06-24 09:46:02 -0500 commented question How to disable internet in a centos VM instance created using openstack?

Depending on the exact requirements it could be enough to edit the security group and block outgoing traffic (80, 443)? Or configure the instance's firewall, or detach the floating ip (unless it's required to reach the other instances). Several ways are possible.

2020-06-23 08:06:30 -0500 commented question python-openstackclient not working 'ImportError: No module named queue'

Seems like an old version of openstackclient, can you share more information about your environment and which openstack version you're using?

2020-06-23 08:03:57 -0500 edited question python-openstackclient not working 'ImportError: No module named queue'

Tried installing python-openstackclient==3.14.3, but the python client is giving error....tried the latest version as well

Error:

[root@test-001test]# openstack
Traceback (most recent call last):
  File "/usr/bin/openstack", line 5, in <module>
    from openstackclient.shell import main
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 23, in <module>
    from osc_lib import shell
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 33, in <module>
    from osc_lib.cli import client_config as cloud_config
  File "/usr/lib/python2.7/site-packages/osc_lib/cli/client_config.py", line 18, in <module>
    from openstack.config import exceptions as sdk_exceptions
  File "/usr/lib/python2.7/site-packages/openstack/__init__.py", line 16, in <module>
    import openstack.config
  File "/usr/lib/python2.7/site-packages/openstack/config/__init__.py", line 17, in <module>
    from openstack.config.loader import OpenStackConfig  # noqa
  File "/usr/lib/python2.7/site-packages/openstack/config/loader.py", line 33, in <module>
    from openstack.config import cloud_region
  File "/usr/lib/python2.7/site-packages/openstack/config/cloud_region.py", line 44, in <module>
    from openstack import proxy
  File "/usr/lib/python2.7/site-packages/openstack/proxy.py", line 24, in <module>
    from openstack import resource
  File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 49, in <module>
    from openstack import utils
  File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>
    import queue
ImportError: No module named queue

Can someone help with this? stuck here

Thanks

2020-06-18 10:32:30 -0500 commented answer VM instance cannot be pinged or SSH

That confirms my previous suspicion, the IPs don’t match because openstack has to assign one, but the instance can’t apply it. Use config-drive to configure the network (including gateway). Provider networks work directly on compute nodes that’s why nothing is created on control node.

2020-06-18 01:48:49 -0500 commented question VM instance cannot be pinged or SSH

Do you have an external dhcp server for the provider network? It could be a different issue but I would try it this way: in our company we can only use config-drive for provider networks. So DHCP is disabled for that subnet and config-drive applies the IP address to the VM that openstack assigned.

2020-06-17 10:18:49 -0500 commented question VM instance cannot be pinged or SSH

Provider networks usually don’t serve dhcp requests, depending on how your instance is configured to obtain its IP it could be necessary to use config-drive for the boot command instead of dhcp. Can you provide more details about dhcp disabled on the provider network and the instance configuration?

2020-06-17 04:06:59 -0500 commented question Veeam rescue ISO uploaded to openstack image not bootable

I've noticed different effects with rescue mode between openstack server rescue and nova rescue commands, maybe you can also try the (probably legacy) nova command, too: nova rescue --image <IMAGE> <SERVER>

2020-06-17 03:33:34 -0500 commented answer openstack ceph concept

To use cinder you need any sort of backend, ceph is one option. You can use local file storage (e.g. LVM) if you like, it's up to you entirely. But with ceph you have a highly available and failure resilient scalable storage backend for nova, cinder and glance.