Ask Your Question

zerorobben's profile - activity

2019-03-07 09:27:26 -0500 received badge  Nice Question (source)
2018-04-09 23:59:08 -0500 received badge  Famous Question (source)
2018-01-04 14:32:04 -0500 received badge  Famous Question (source)
2018-01-04 14:32:04 -0500 received badge  Famous Question (source)
2017-12-15 17:59:19 -0500 commented question Have commands changed? I receive errors like "'port' is not an openstack command."

Interesting. A new version (plus restarting services) did the trick. I'm not sure why this worked before but had since broken. Thanks (as always)!

2017-12-15 17:57:00 -0500 received badge  Popular Question (source)
2017-12-13 10:30:39 -0500 asked a question Have commands changed? I receive errors like "'port' is not an openstack command."

Hi,

I've just updated the software on an Ubuntu 16.04 server that accesses my OpenStack configuration. This particular server only has python-openstackclient to interact with OpenStack. I was previously able to use the client without issues and I have written a number of scripts that use it to obtain information. Now, all of my scripts are broken:

$ openstack port list
openstack: 'port' is not an openstack command. See 'openstack --help'.
Did you mean one of these?
  port delete
  port show
  project create
  project delete
  project list
  project set
  project show

This has never been a problem before. One of my primary scripts used this command when allocating IP addresses. The most recent version of the documentation (for pike) says this is still a viable command: https://docs.openstack.org/python-ope...

Some commands also seem to have been reordered:

$ openstack floating ip list
openstack: 'floating' is not an openstack command. See 'openstack --help'.
Did you mean one of these?
  container create
  container delete
  container list
  container save
  container set
  container show
  container unset

no longer works, and

$ openstack ip floating list
+--------------------------------------+---------------------+------------------+--------------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 |
+--------------------------------------+---------------------+------------------+--------------------------------------+
…

seems to have taken its place. The former command is still on the documentation (https://docs.openstack.org/python-ope...).

I don't think I've broken anything. I can still use commands like openstack server create successfully on this server. The functions I am trying to use work from Horizon. My software is recent (All packages are up to date.) and I have since restarted the server.

What has happened? Has this affected anyone else? Is this platform-specific?

I would appreciate any and all information so I can get things working again (on the most recent version)!

Thanks!

2017-10-27 09:18:41 -0500 received badge  Notable Question (source)
2017-10-23 04:44:43 -0500 received badge  Popular Question (source)
2017-10-20 12:50:16 -0500 received badge  Teacher (source)
2017-10-19 17:09:03 -0500 answered a question Is there a way to see the kolla-build output after the build process ends?

I have not used kolla-build before, but your logs should either be in your home directory or /var/log. Use a program like more or less to view the full log file if you are unable to scroll in your terminal emulator.

2017-10-19 10:59:18 -0500 commented question kolla-build fails at start due to AtrributeError please help

Now it looks as though you've not installed the Docker SDK. The Python script has import docker and this line attempts to access docker.APIClient, but that fails because it doesn't exist. Have you installed the prerequisites for this software?

2017-10-19 10:37:31 -0500 commented question kolla-build fails at start due to AtrributeError please help

I've not used kolla-build before, but it looks like your version of pyOpenSSL is incompatible (or not present). Try to reinstall it?

2017-10-19 10:20:06 -0500 commented question nova launch instance fails with "Unknown auth type: None"

Can you post what you have in /etc/neutron/neutron.conf? I suspect this has not yet been configured, as your compute service configuration file seems fine.

2017-10-19 10:07:04 -0500 asked a question Can't reach instances by floating IPs; can ping router

Hi,

I have an interesting problem I've not encountered before. I have recently created a new OpenStack environment that is functional in every respect but networking. I can instantiate instances and associate floating IPs, but when I attempt to ping or ssh, I receive no response from the nodes. I can ping the router from my external network.

I have configured my external network as a flat network with network address 10.247.1.0/24 and gateway 10.247.1.1, according to my physical network. I then created a local internal network with address 10.10.0.0/24 and gateway 10.10.0.1. I created a router and added the two networks (10.247.1.5 and 10.10.0.1 as interfaces).

I can ping the router:

$ ping -q -c 3 10.247.1.5
PING 10.247.1.5 (10.247.1.5) 56(84) bytes of data.

--- 10.247.1.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.153/0.171/0.200/0.023 ms

I can also associate floating IPs with my instances. However, I cannot reach the instances themselves:

$ ping -q -c 3 10.247.1.7
PING 10.247.1.7 (10.247.1.7) 56(84) bytes of data.

--- 10.247.1.7 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2015ms
pipe 3

If I try to add instances to the external network itself, I cannot ping them, which leads me to believe there is some issue with my network configuration.

Is there some configuration file or any more information I could provide that could help with this?

I appreciate any information.

Thanks!

2017-10-12 10:18:47 -0500 received badge  Notable Question (source)
2017-10-12 10:15:23 -0500 received badge  Scholar (source)
2017-10-08 01:15:12 -0500 received badge  Nice Question (source)
2017-10-08 01:15:09 -0500 received badge  Popular Question (source)
2017-10-05 19:38:47 -0500 received badge  Student (source)
2017-10-05 11:04:21 -0500 received badge  Notable Question (source)
2017-10-05 10:54:51 -0500 asked a question Integration of Ironic and existing VM configuration

Hello,

I've been looking for information on a potential project for some time and have been unsuccessful in finding relevant information elsewhere. I thought to turn to the community to see if anyone has done something similar; if this is not the place for this type of question, please let me know where I might be able to ask.

I am working on a project that is looking to utilize OpenStack to dynamically allocate and provision computational resources in a high-performance computing environment. I have a functional configuration for virtual machines and would like to incorporate Ironic (the performance of bare metal makes it ideal for computational research, but it's largely inflexible in our current setup, so we're exploring alternatives). From what I have read, however, this is not an easy task: the documentation for Ironic does not mention an existing configuration (https://docs.openstack.org/ironic/latest/install/configure-compute.html (https://docs.openstack.org/ironic/lat...)) and most case studies don't seem to use both virtual machines and bare metal in one environment. From what I understand, such a system would require multiple instances of Nova.

Can someone with experience deploying Ironic provide some guidance so I know whether this is feasible? I have a few questions:

  • How will running multiple Nova services work? What needs to be done to configure OpenStack such that all other services integrate with both, if this is at all possible? This project would also need to be able to scale (up to hundreds of servers); will this complicate things?
  • Is it possible to integrate networking for virtual machines and bare metal? I would like to be able to connect the two with some degree of isolation from the rest of the world. It looks like the default configuration for Ironic is a flat network (https://docs.openstack.org/ironic/latest/install/configure-networking.html (https://docs.openstack.org/ironic/lat...) uses a flat network for provisioning), so my existing network configuration (with OpenVSwitch) seems incompatible. I have access to the switch and can adjust the physical networks (with automation) if this is necessary.
  • Can the same hardware be used for virtual machines and bare metal? We are already using another platform that can do this (developed by a research group at our university and lacking commercial or open source support), but we haven't seen any information about it in the OpenStack community. This doesn't seem easy because the existing compute nodes are running a host operating system that manages the other services. I cannot predict the type of work that will be done, so limiting the number of servers by specifying whether each is a bare metal or virtual machine host seems counterproductive.

I can, of course, run multiple OpenStack instances (one for virtual machines and one for bare metal), but doing so would require a middle-man, render resources incompatible, and require manual configuration and considerations for architecture. The most feasible option seems to be running multiple instances ... (more)

2017-10-05 09:15:42 -0500 received badge  Enthusiast
2017-10-04 10:20:57 -0500 asked a question Can't SSH from one instance to another

Hi,

I can't SSH between my instances (which, of course, have the same SSH key). This is quite puzzling, as I am able to SSH to the instances from the controller node. The "Key Pair" is the same on all of the instances I've created thus far and they can see each other:

# ping -c 3 10.10.0.7
PING 10.10.0.7 (10.10.0.7) 56(84) bytes of data.
64 bytes from 10.10.0.7: icmp_seq=1 ttl=64 time=0.624 ms
64 bytes from 10.10.0.7: icmp_seq=2 ttl=64 time=0.571 ms
64 bytes from 10.10.0.7: icmp_seq=3 ttl=64 time=0.518 ms

--- 10.10.0.7 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.518/0.571/0.624/0.043 ms

This is true of instances running both CentOS 7 and Ubuntu 16.04. I've tried to ssh -i with all of the keys in /etc/ssh. Notably, the ~/.ssh directory contains no keys.

I receive the following error regardless of my arguments to the ssh program:

# ssh centos@10.10.0.7
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

This works, of course, if I add the private SSH key to the server. However, much of what I plan to do is automated, and copying the key to the server after it has been initialized does not seem very efficient and seems a possible avenue for failure. Is there a way to automatically add the private SSH key from the key pair to the virtual machines themselves? Is it possible to use a different key, then? Some similar questions suggest this would require manual configuration, but I'm not quite sure how to approach this: what would I do to copy the key? SSH without a password and copy the file?

Thanks in advance!

2017-09-11 08:39:16 -0500 received badge  Popular Question (source)
2017-09-07 10:53:39 -0500 answered a question Ironic API: Address already in use?

I was able to resolve this. I disabled and stopped ironic-api, but Ironic had spawned a server listening on port 6385. This is a python process and is not killed when Ironic is. As a result, stopping Ironic, reconfiguring Apache, and restarting the apache2 service does not work because Apache attempts to listen on the (now blocked) port 6385 when it is first started and fails immediately.

This was as simple as running netstat -anop | grep 6385 and killing the rogue process with kill <pid> before restarting apache2.

2017-09-06 11:21:07 -0500 received badge  Famous Question (source)
2017-09-06 11:10:48 -0500 received badge  Editor (source)
2017-09-06 10:53:06 -0500 asked a question Ironic API: Address already in use?

Hi,

I'm just starting to install Ironic on my existing OpenStack (Ocata, Ubuntu 16) environment. I am presently https://docs.openstack.org/project-install-guide/baremetal/ocata/configure-integration.html (trying to integrate my compute and baremetal resources), but the Ironic API service is shutting down as soon as it starts. The compute service is still functional and my virtual machines are still running, so I believe this is relevant to the Ironic setup.

The same error message is repeated many thousands of times in /var/log/ironic/ironic-api.log:

ERROR oslo.service.wsgi [-] Could not bind to 0.0.0.0:6385
CRITICAL ironic [-] error: [Errno 98] Address already in use
ERROR ironic Traceback (most recent call last):
ERROR ironic   File "/usr/bin/ironic-api", line 10, in <module>
ERROR ironic     sys.exit(main())
ERROR ironic   File "/usr/lib/python2.7/dist-packages/ironic/cmd/api.py", line 42, in main
ERROR ironic     server = wsgi_service.WSGIService('ironic_api', CONF.api.enable_ssl_api)
ERROR ironic   File "/usr/lib/python2.7/dist-packages/ironic/common/wsgi_service.py", line 48, in __init__
ERROR ironic     use_ssl=use_ssl)
ERROR ironic   File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 115, in __init__
ERROR ironic     self.socket = self._get_socket(host, port, backlog)
ERROR ironic   File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 143, in _get_socket
ERROR ironic     sock = eventlet.listen(bind_addr, family, backlog=backlog)
ERROR ironic   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 44, in listen
ERROR ironic     sock.listen(backlog)
ERROR ironic   File "/usr/lib/python2.7/socket.py", line 228, in meth
ERROR ironic     return getattr(self._sock,name)(*args)
ERROR ironic error: [Errno 98] Address already in use

I don't see the port 6385 repeated multiple times in any of my configuration files, so I'm not sure what could be causing this issue. Perhaps it is the IP address? I'm not sure what the convention is. The only other mention of this (with the same port and message) is in a https://bugs.launchpad.net/kolla/+bug/1518336 (2015 issue with Kolla) that is said to have been patched around that time (so I don't think it's related to my issue).

Perhaps it is related to the Apache server, as the openstack baremetal node create command returns "Forbidden (HTTP 403)." It is also visible multiple times with the lsof command:

# lsof -i | grep 6385
apache2    6699            root    8u  IPv6     65122      0t0  TCP *:6385 (LISTEN)
apache2   18825        www-data    8u  IPv6     65122      0t0  TCP *:6385 (LISTEN)
apache2   18835        www-data    8u  IPv6     65122      0t0  TCP *:6385 (LISTEN)

Ironic is the only OpenStack service configured to use this port:

cat /etc/apache2/sites-enabled/* | grep 6385
Listen 6385
<VirtualHost *:6385>

Has anyone encountered a similar error? Is there anything that can be checked? I must admit, I'm not familiar with Ironic, so some of my configurations might not be entirely correct.

I'd be happy to provide more information if it's needed. Please let me know if there's anything that ... (more)

2017-08-28 08:34:46 -0500 received badge  Famous Question (source)
2017-07-22 18:13:00 -0500 received badge  Notable Question (source)
2017-07-17 10:20:31 -0500 received badge  Notable Question (source)
2017-07-12 12:01:38 -0500 commented answer Linux bridge agent functioning on controller, but not compute node; both nodes visible to each other, services running

Bernd,

I re-installed everything (on an Ubuntu Server base this time) on different nodes with the same configuration and I am now able to see both Linux bridge agents without issue. I'll try to see if my configuration files differ for the CentOS installation, but I consider this resolved.

Thanks!

2017-07-12 11:22:34 -0500 received badge  Popular Question (source)
2017-07-12 09:12:09 -0500 commented answer Linux bridge agent functioning on controller, but not compute node; both nodes visible to each other, services running

Hi Bernd,

Thanks for the comment. It is set to the management IP address of the controller (10.247.1.11), as is recommended by the error message.

[vxlan]

enable_vxlan = true

local_ip = 10.247.1.11

l2_population = true

Thanks!

2017-07-11 11:38:39 -0500 asked a question Linux bridge agent functioning on controller, but not compute node; both nodes visible to each other, services running

Hi,

I'm struggling with my network configuration (or Neutron configuration). I am following the https://docs.openstack.org/ocata/install-guide-rdo/neutron-controller-install-option2.html (RDO installation guide) with "Networking Option 2." Both nodes are running a recent version of CentOS 7.

[root@controller ~]# openstack network agent list # output has been modified to fit on small screens

+----+--------------------+------------+------------+-------+-------+---------------------------+
| ID | Agent Type         | Host       | Avail Zone | Alive | State | Binary                    |
+----+--------------------+------------+------------+-------+-------+---------------------------+
| ID | Linux bridge agent | controller | None       | True  | UP    | neutron-linuxbridge-agent |
| ID | L3 agent           | controller | nova       | True  | UP    | neutron-l3-agent          |
| ID | DHCP agent         | controller | nova       | True  | UP    | neutron-dhcp-agent        |
| ID | Metadata agent     | controller | None       | True  | UP    | neutron-metadata-agent    |
+----+--------------------+------------+------------+-------+-------+---------------------------+

According to https://docs.openstack.org/ocata/install-guide-rdo/neutron-verify-option2.html (the documentation), I should also have a "Linux bridge agent" on the compute node (compute1). I had issues with the controller bridge agent not appearing on this list either, but this was resolved by removing the Infiniband card on my server (apparently older versions of OpenStack don't like the non-standard MAC address). I also removed the card from the compute node, but this issue persists, which leads me to believe the problem is related to my network configuration.

The neutron-linuxbridge-agent service is running on the compute node, but it is restarting repeatedly in short intervals.

[root@compute1 ~]# systemctl | grep neutron

neutron-linuxbridge-agent.service                                                                                              loaded active running   OpenStack Neutron Linux Bridge Agent

[root@compute1 ~]# journalctl -xel

Jul 11 10:12:41 compute1 neutron-linuxbridge-agent[30559]: Could not load neutron.openstack.common.notifier.rpc_notifier
Jul 11 10:12:41 compute1 systemd[1]: neutron-linuxbridge-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 11 10:12:41 compute1 systemd[1]: Unit neutron-linuxbridge-agent.service entered failed state.
Jul 11 10:12:41 compute1 systemd[1]: neutron-linuxbridge-agent.service failed.
Jul 11 10:12:41 compute1 systemd[1]: neutron-linuxbridge-agent.service holdoff time over, scheduling restart.
Jul 11 10:12:41 compute1 systemd[1]: Starting OpenStack Neutron Linux Bridge Agent...

The error message is different on the compute node. The IP 10.247.1.11 is the management IP of the controller and is associated with the device "eno1."

[root@compute1 ~]# tail -40 /var/log/neutron/linuxbridge-agent.log

2017-07-11 10:33:53.806 10798 INFO neutron.common.config [-] Logging enabled!
2017-07-11 10:33:53.806 10798 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 10.0.1
2017-07-11 10:33:53.806 10798 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'eno2'}
2017-07-11 10:33:53.807 10798 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {}
2017-07-11 10:33:53.827 10798 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 10.247.1.11 on the host interface to be used for tunneling and restart the agent.

The compute node is visible to the controller.

[root@controller ~]# openstack host list

+------------+-------------+----------+
| Host Name  | Service     | Zone     |
+------------+-------------+----------+
| compute1   | compute     | nova     |
| controller | consoleauth | internal |
| controller | conductor   | internal |
| controller | scheduler   | internal |
+------------+-------------+----------+

[root@controller ~]# openstack hypervisor list

+----+---------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP ...
(more)
2017-07-11 10:58:08 -0500 received badge  Popular Question (source)
2017-07-07 12:58:38 -0500 answered a question Changed host name on controller; all compute services are now "down"

I was able to fix this problem on my own. When I changed my hostname, RabbitMQ stopped working—in fact, the "openstack" account disappeared entirely. I followed the https://docs.openstack.org/ocata/install-guide-rdo/environment-messaging.html (instructions) on configuring RabbitMQ a second time and everything resolved itself.

2017-07-07 03:06:01 -0500 asked a question Changed host name on controller; all compute services are now "down"

Hi,

I'm starting to install OpenStack on a controller and compute node (both running CentOS 7) with the official installation guide. I've just changed the hostname of my controller server from "localhost.localdomain" to "controller"; I had forgotten to do so earlier. Everything has just broken. When I run the command openstack compute service list, all the services are "down."

+----+------------------+------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host       | Zone     | Status  | State | Updated At                 |
+----+------------------+------------+----------+---------+-------+----------------------------+
| 11 | nova-compute     | compute1   | nova     | enabled | down  | 2017-07-06T15:52:01.000000 |
| 12 | nova-consoleauth | controller | internal | enabled | down  | None                       |
| 13 | nova-conductor   | controller | internal | enabled | down  | None                       |
| 14 | nova-scheduler   | controller | internal | enabled | down  | None                       |
+----+------------------+------------+----------+---------+-------+----------------------------+

The error messages on the controller are all very similar.

tail -n 50 /var/log/nova/nova-scheduler.log

…

2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     self.wait()
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     return self.dispatch_method(method_sig, args, content)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     return amqp_method(self, args)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     (class_id, method_id), ConnectionError)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service

The compute node cannot start the service openstack-nova-compute.service. This was working properly before the controller hostname was changed and was "up" on openstack compute service list.

systemctl start openstack-nova-compute.service
Job for openstack-nova-compute.service failed because the control process exited with error code. See "systemctl status openstack-nova-compute.service" and "journalctl -xe" for details.

The commands it recommends are not too helpful.

journalctl -xe
-- Unit openstack-nova-compute.service has begun starting up.
Jul 06 10:35:38 compute1 systemd[1]: openstack-nova-compute.service: main proces
Jul 06 10:35:38 compute1 systemd[1]: Failed to start OpenStack Nova Compute Serv
-- Subject: Unit openstack-nova-compute.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-compute.service has failed.
-- 
-- The result is failed.

The error messages are generic.

systemctl status openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: activating (start) since Thu 2017-07-06 10:39:25 MDT; 767ms ago
 Main PID: 8461 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─8461 /usr/bin/python2 /usr/bin/nova-compute

Jul 06 10:39:25 compute1 systemd[1]: Starting OpenStack Nova Compute Server...

The services stay in the "starting" state, which leads me to believe they are not ... (more)