Ask Your Question

Krist's profile - activity

2019-12-09 03:03:39 -0600 received badge  Famous Question (source)
2019-12-09 03:03:39 -0600 received badge  Notable Question (source)
2019-11-21 03:56:58 -0600 received badge  Nice Question (source)
2019-04-17 08:25:34 -0600 received badge  Famous Question (source)
2018-03-23 01:39:57 -0600 received badge  Famous Question (source)
2017-03-10 13:55:15 -0600 received badge  Famous Question (source)
2015-09-24 08:53:31 -0600 received badge  Famous Question (source)
2015-06-29 07:44:09 -0600 received badge  Famous Question (source)
2015-05-24 17:29:46 -0600 received badge  Famous Question (source)
2015-05-12 03:54:31 -0600 received badge  Notable Question (source)
2015-05-12 03:54:31 -0600 received badge  Popular Question (source)
2015-05-12 03:54:31 -0600 received badge  Famous Question (source)
2015-04-27 05:06:01 -0600 received badge  Famous Question (source)
2015-04-27 05:06:01 -0600 received badge  Popular Question (source)
2015-04-27 05:06:01 -0600 received badge  Notable Question (source)
2015-04-23 12:53:01 -0600 received badge  Famous Question (source)
2015-04-23 12:53:01 -0600 received badge  Notable Question (source)
2015-04-13 00:44:25 -0600 received badge  Popular Question (source)
2015-04-08 06:15:29 -0600 asked a question How do I use rally with a openstack installation that uses a self signed certificate?

We have a openstack installation where the apis are protected using ssl. At the moment we still are using a self signed certificate. When I try to run tests against the apis using rally it (understandably) complains that it can't verifiy the SSL certificate:

2015-04-08 13:11:48.378 8653 TRACE rally SSLError: SSL exception connecting to https://api/keystone/v2.0: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I though that I could fix this by setting "https_insecure = true" in rally.conf, but this had no effect.

So how do I go about testing my open stack with rally, using a self signed certificate?

2015-04-03 03:53:39 -0600 received badge  Famous Question (source)
2015-02-27 10:02:56 -0600 received badge  Notable Question (source)
2015-02-11 00:26:08 -0600 received badge  Notable Question (source)
2015-02-05 01:06:10 -0600 answered a question Nova compute fails: RemoteError: Remote error: NoSuchOptError no such option: memcached_servers

The cause was that after adding the memcached_servers setting to nova.conf and restarting the nova services a few processes were not properly stopped and restarted. As a restult there were nova conductor workers running with an outdated config. When they consumed messages from the queues they of course threw errors, which got passed back through the queues. It's only after I started tracing the queues that I discovered this.

So the moral of the story: When you do a restart, make sure that you do indeed restart everything...

2015-02-03 09:54:44 -0600 commented question Nova compute fails: RemoteError: Remote error: NoSuchOptError no such option: memcached_servers

Another observation: When we do a test starting lots of instances about 5% are succesful, and are started. The rest fail. This appears to be a random error with a high probability... But why?

2015-02-03 09:29:29 -0600 received badge  Popular Question (source)
2015-02-03 01:07:21 -0600 commented question Nova compute fails: RemoteError: Remote error: NoSuchOptError no such option: memcached_servers

This is havana. Commenting out the line makes no difference. It hinkg however that the error is thrown by the request for CONF.memcached_servers. So it expects this parameter. Maybe I just need to add it to every section of the config...

2015-02-02 08:13:50 -0600 asked a question Nova compute fails: RemoteError: Remote error: NoSuchOptError no such option: memcached_servers

When I try to start an instance this fails. The message I see in compute.log is:

2015-02-02 14:55:46.903 13368 TRACE nova.compute.manager [instance: 7265cacc-8adb-4aed-9d33-076733ff0a12] RemoteError: Remote error: NoSuchOptError no such option: memcached_servers
2015-02-02 14:55:46.903 13368 TRACE nova.compute.manager [instance: 7265cacc-8adb-4aed-9d33-076733ff0a12] [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data\n    **args)\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 513, in get_ec2_ids\n    ec2_ids[\'instance-id\'] = ec2utils.id_to_ec2_inst_id(instance[\'uuid\'])\n', u'  File "/usr/lib/python2.6/site-packages/nova/api/ec2/ec2utils.py", line 189, in id_to_ec2_inst_id\n    int_id = get_int_id_from_instance_uuid(ctxt, instance_id)\n', u'  File "/usr/lib/python2.6/site-packages/nova/api/ec2/ec2utils.py", line 45, in memoizer\n    _CACHE = memorycache.get_client()\n', u'  File "/usr/lib/python2.6/site-packages/nova/openstack/common/memorycache.py", line 39, in get_client\n    memcached_servers = CONF.memcached_servers\n', u'  File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1652, in __getattr__\n    raise NoSuchOptError(name)\n', u'NoSuchOptError: no such option: memcached_servers\n'].

I see a similar message in scheduler.log.

What does this message mean?

As far as I know "memcached_servers" is a valid config parameter. In my config this is set to "none".

What is going on here?

2015-01-31 00:34:35 -0600 marked best answer Error message in nova/api.log with empty token.

Hello,

In nova/api.log I see a lot of messages like this:

WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token

I seem to recall however that in this error messages the token should be mentioned, ie. the after "token" a token should be printed. So this looks like something using an empty token.

Is my suspicion correct? And if this is the case, how do I find out what it is that is causing these warnings?

2015-01-31 00:33:52 -0600 received badge  Famous Question (source)
2015-01-26 23:29:58 -0600 received badge  Notable Question (source)
2015-01-26 23:29:58 -0600 received badge  Popular Question (source)
2015-01-21 15:39:53 -0600 received badge  Famous Question (source)
2015-01-20 00:28:10 -0600 commented question How do I create a Swift only user

My problem is that it's not just swift that allows access. It's nova as well. I removed _member_ as a role from a user, and this user could still log in to Horizon, and could still stop and start instances. Also this user could use the nova command line tool. I want a swift only user.

2015-01-19 02:02:47 -0600 commented question How do I create a Swift only user

I tested this now, and it is not working as expected. I created a user with only the SwiftOperator role, (removed _member_) but this user has full access to everything on the tenant. What have I overlooked?

2015-01-15 05:24:48 -0600 asked a question How do I create a Swift only user

I want to backup stuff to swift from within an instance. Therefore I need to enter credentials for a user that can write objects to a swift container in the backup script. For security reasons I want to limit what this user can as much as possible.

So given a tenant, I want to create a user in the tenant that can not log in to horizon, cannot use the API to spawn instances or even just list/get stuff. I want this user to be able to do only one thing: Write objects in a specified swift container.

How should I go about this?

I am thinking along the lines of:

  • Create another role "_swift_", and assigning that to this user, but not the "_member_" role. I would also assign this _swift_ role to all other users.

  • Modifying proxy-server.conf so that it contains this role. like this:

[filter:keystone]
use = egg:swift#keystoneauth
operator_roles = admin, SwiftOperator, _swift_
  • Then add some ACLS maybe?

Would this work, or would this break thing?

2014-12-22 00:44:07 -0600 received badge  Notable Question (source)
2014-12-14 18:25:14 -0600 received badge  Famous Question (source)
2014-12-12 17:52:44 -0600 received badge  Notable Question (source)
2014-12-12 17:52:44 -0600 received badge  Popular Question (source)
2014-12-11 06:25:32 -0600 asked a question How to repair a missing container in Swift

I have a Swift install, part of a RHEL OS4 install. On this install I have a problem...

My problem is that for some objects the container for those objects is missing. I want to recreate those containers. This ought to be possible, as the information in the metadata for each opject contains the container.

What I did so far, on a storage node...

# find . -name "*.data" > /tmp/object
# while read f ; do swift-object-info $f ; done > /tmp/objectinfo< /tmp/objects 
# grep Path /tmp/objectinfo  | sed -e 's/Path: //' | sort  > /tmp/objectpaths

This gave me a list of all exiting objects on our swift stack (not a lot at the moment). I then fed this in to swift-acount-audit:

# swift-account-audit -c1 -e /tmp/problempaths < /tmp/objectpaths 
...
  Accounts checked: 22

Containers checked: 35
  Missing Replicas: 8

   Objects checked: 218

Now this is a test environment, with only one device and replica count 1. I know this is not how it should be, right now, this is how it is. As you can see there are 8 containers missing.

Now, looking at hte swif-objectinfo output I see that the container name is stored in the metadata. So the information needed to recreate the containers. (and the .db files in them) does exist. But how do I do this? Is there some swift tool that I can use that basically crawls a directory structure with .data files and crecreates any missing container and account files?

2014-11-26 04:35:31 -0600 received badge  Famous Question
2014-11-25 17:07:37 -0600 received badge  Popular Question
2014-11-25 17:07:37 -0600 received badge  Notable Question
2014-11-19 01:26:27 -0600 commented answer Improving neutron openvswitch performance

I am aware of this. However the MTU is not our problem... What I need is either tuning our oVswitch, or a better way to detect misbehaving hosts....

2014-11-17 09:00:30 -0600 edited question Improving neutron openvswitch performance

Hello all,

We run openstack havana, with neutron/openvswitch. Our networking node is dual socket XEON machine with 6 sockets (12 threads) per CPU. 32 GB ram. This network node and the compute nodes sit on a 10Gb network. We have a 1Gb connection with the internet, that is burstable to 10Gb.

What I notice is that as soon as traffic hits about 180Mb/s the network starts feeling saturated. When I log in to a node for example the prompt responds sluggish. Connections fail etc.

On the neutron node the load increased to about 8, (which for a 12 core machine should not be a problem) and openvswitch usage in top sat at about 500%. Which means it was using about 5 cores. Again, that ought, in my iponion, not be a problem.

We're running openvSwitch 2.1.3, multi threaded, and about 16 threads are started.

My questions:

  • I suspect that fragmentation might be going on. Is there a way to find out if an image has an incorrect MTU set, without needing access to the image?
  • Is there a way to make openvswitch even more performant? I really want to fill the pipe.

Update:

I managed to get network traffic of up to 1Gb or more, over our line without the network node seeing it's load going over 0.7. I did this by starting a bittorrent session in a VM. I noticed that the openvswitch had no problem coping.

My current working hypothesis is that the problem is that a customers VM got hacked, and started working as part of a botnet, and as such executed a lot of network scans. This means (I think) that a lot of packets hit the switch for which no flows exist. This means that the userspace component has a lot of work to do, and that the flow tables get large.

Googling I do find references to the possibility to tweak things, however I do not find any information on the openvswitch site on which parameters I can set, and what the consequences are. Can anyone point me in the right direction?

2014-11-17 08:56:23 -0600 commented answer Improving neutron openvswitch performance

Would this help us? We have broadcom network cards.

2014-11-17 03:06:05 -0600 received badge  Famous Question (source)
2014-11-16 10:06:11 -0600 received badge  Famous Question (source)
2014-11-14 19:06:19 -0600 received badge  Famous Question (source)
2014-11-14 13:06:58 -0600 received badge  Notable Question (source)