briancline's profile - activity

HI,

We have configured SWIFT as an object storage with HAVANA. All the services is working fine but we are not able to authenticate is with keystone. It is showing the below error. We have checked all the configuration but not able to figure out the issue.

Please help in providing the solution.

root@Storage:/etc/swift# swift stat Account HEAD failed: http://Storage.abc.com:8080/v1/AUTH_c.. . 401 Unauthorized

hi I installed openstack using devstack with the following localrc file.

####enable swift
enable_service s-proxy s-object s-container s-account
SWIFT_REPLICAS=3
SWIFT_HASH=A1231312312
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-metering
enable_service q-lbaas
enable_service tempest

when i upload an object using swift it creates 3 copies of the file, but when i delete one copy the object is not replicated.

Given your use case, this sounds like a simple high-activity versus low-IOPS capacity problem. Essentially you're asking a relatively few number of drives to do quite a bit of work; it may not seem like a lot, but factor in writes of all replicas, account and container database retrieval/maintenance involved in each request, replication runs for account DBs/container DBs/objects, object-expirer scans, auditor runs, etc.).

Probably the least-hassle solution may simply be to add more raw IOPS capacity by adding more drives to the cluster (note that this does not necessarily mean more physical servers; only drives), as well as set up some monitoring and alerting to check these metrics frequently so you can anticipate the problem much sooner in the future.

You might counter this by arguing that you're nowhere near your total disk capacity and you don't need more disks, but more disk capacity isn't really the answer here since you're hitting a ceiling on the IOPS your existing disks provide. More drives and therefore more IOPS is going to be your best bet, unless you want to replace out these drives with expensive 10K or even 15K drives (although, assuming your activity continues to increase as it always does, there will be a point where you'll still hit the same raw IOPS limitations and have to scale out onto more drives anyway).

EDIT: I missed the tail end of your post the first time I wrote this. I know you mention SSDs are too expensive, however it sounds like you were ruling them out (rightly so) for storage of all your objects. You might reconsider using SSDs if you were to use them only for account and container database storage (essentially using only those SSDs in your account and container rings, rather than the same disks as your object ring uses). Using SSDs in this manner is a very common and battle-tested way of taking some load off the spindle-based disks storing your actual objects while realizing major performance gains in account- and container-related operations.

Using SSDs only for account and container databases won't require anywhere near as much space as your objects will, so may not be as cost-prohibitive as you thought; if you wanted to know your absolute minimum space requirement for SSDs for this purpose, you could run a bit of a cache-killing command du -sh /srv/node/*/{account,container} on each of your storage nodes to determine how much space those currently use.

hi all,

according to my understanding of openstack swift object put request flow path, the request first goes to the proxy server, then proxy server forward the request to ObjectController defined in swift.proxy.controllers.obj.py. Here is some portion of lines of code of ObjectController.PUT() in swift.proxy.controllers.obj.py

def PUT(self, req):
        """HTTP PUT request handler."""
-----------
# do a HEAD request for container sync and checking object versions
if 'x-timestamp' in req.headers or \
                (object_versions and not
                 req.environ.get('swift_versioned_copy')):
            hreq = Request.blank(req.path_info, headers={'X-Newest': 'True'},
                                 environ={'REQUEST_METHOD': 'HEAD'})
            hresp = self.GETorHEAD_base(
                hreq, _('Object'), self.app.object_ring, partition,
                hreq.swift_entity_path)
-----------

in the following lines of code, self.GETorHEAD_base() create an HTTPConnection to object server for a HEAD request which is defined in ObjectController.HEAD() of swift.obj.server.py HEAD request retrieve the metadata of the object

My question is:

AT these lines of code, we are in process of writting an object, object is not written till yet on the node, then what will it return to hresp?

thanks in advance Pragya Jain

The nova-api host needs to route packets to/from which network, exactly?

As a clarification, I *am* using cloud-init in my environment.

I'm seeing this as well, except most of the metadata requests come back in 0.3 secs, whereas 3 or 4 of all the ones made by a single boot from a single instance take as long as the above (~8-17 secs). Also noticed that the quicker ones usually only hit once every ~2 seconds, rather than rapid-fire.

I've noticed with both Ubuntu cloud images and CirrOS images that, even though I provide multiple vnics when creating a VM in Nova, they only bring up the first interface.

After booting, I'm able to confirm by checking the /etc/network/interfaces file, where only lo and eth0 are defined. This seems to be hard-coded for some reason.

When I add eth1, I'm able to bring it up, but this seems counterintuitive to the idea of spinning up ready-to-use VMs.

Is there any way whatsoever to get these to bring up each NIC instead of just one?

D'oh. Apologies. I didn't realize that the new volume_dd_blocksize option didn't make it into Grizzly and is currently only available in trunk. I'm using Grizzly packages.

Nothing to see here...

If I understand correctly, it sounds like you want to limit the throughput rate of only the replication between storage nodes, and not the data from proxy nodes.

If so, you can either use tc with multiple classes, placing proxy nodes into a specific class, and storage nodes into a different class with a lower priority and its own set of throughput limits. Depending on how you construct your tc ruleset, proxy-to-storage traffic can then be prioritized over storage-to-storage traffic (replication in this case). You could also put the aforementioned rate/throughput-limiting rules in place only for storage node IPs, so that traffic to/from proxies won't be limited.

The configuration value you're thinking of is probably replica count -- when you set up your rings, you can specify the number of replicas that are created. Each replica has throughput cost, as you mention, however you'll want to consider whether reducing the redundancy of data stored within the cluster is worth reducing the bandwidth spent to persist it. Personally, I'd recommend keeping replica count at 3 replicas (or more), and limiting/deprioritizing the throughput between storage nodes separately.

Hope this helps!

I've got Cinder running atop an LVM volume group as described in the install guide. Everything works -- creating, attaching, detaching, deleting, etc.

However, upon deleting a 100GB Cinder volume, I noticed it took quite a long time due to zeroing out the bits on disk to prevent data leak once an overlapping volume gets created. The first test took about ~11 minutes for this step to complete.

I'd definitely like to speed this up a bit, if possible. I've tried adjusting the volume_dd_blocksize option in /etc/cinder/cinder.conf from the default 1M to 4M, and restarted all Cinder services. However, creating then deleting a volume still yields the same block size being used when it gets wiped (dd ... bs=1M).

Is there some other place this option needs to go? My configuration currently consists of:

[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
debug = True
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder

# MySQL Connection #
sql_connection=mysql://cinder:111p455w0rd@10.1.2.3/cinder

# RabbitMQ #
rabbit_password=111p455w0rd

# iSCSI and Volume Management #
iscsi_helper = tgtadm
iscsi_ip_address = 10.90.18.152
volume_name_template = volume-%s
volume_group = cinder-volumes
volume_dd_blocksize = 4M

Yes, this is correct behavior. It gets substituted accordingly at run-time when a tenant requests a service/endpoint list.

This formatting makes it easier for services that require additional segments after the tenant ID, as well as for deployers who need to change the URL schemes a bit.

Are there any errors or warnings in the /var/log/nova/nova-conductor.log and nova-compute.log files?

In addition to @obuisson's answer, here's a good checklist to run through to diagnose problems with attaching Cinder volumes:

• Ensure the sysfsutils package is installed, which provides the systool command
• Ensure the cinder-volumes LVM volume group is active if you're using LVM: vgchange -ay cinder-volumes
• Determine whether you have both tgt and iscsitarget installed and running. If so, remove one, and use the --purge flag with apt-get to remove the other (i.e., apt-get remove --purge iscsitarget). Then, start or restart the one you wish to keep, if it's not already running.
• Ensure iscsid is started with service open-iscsi status
• Verify you get something back from iscsiadm -m discovery
• If the iscsiadm discovery returned a result, take note of the IP address and and verify you don't get any errors from iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260 (assuming 127.0.0.1:3260 is what you got from iscsiadm's discovery)
• Check your /etc/cinder/cinder.conf file to ensure:
  • iscsi_helper is set to tgtadm if you're using tgt, or ietadm if you're using iscsitarget
  • iscsi_ip_address is set to the management-network (private) IP where you are running Cinder and tgt/iscsitarget
  • volume_group and volume_name_template are set correctly if using LVM
• Check your /etc/nova/nova.conf file to ensure:
  • volume_api_class is set to nova.volume.cinder.API
  • iscsi_helper is set the same as it is in cinder.conf
  • iscsi_ip_address is set the same as it is in cinder.conf
  • volume_group and volume_name_template are the same as they are in cinder.conf, if using LVM
• Restart all Cinder services: for ii in /etc/init.d/cinder-*; do restart $(basename $ii); done
• Restart all Nova services: for ii in /etc/init.d/nova-*; do restart $(basename $ii); done
• Test again with a brand new Cinder volume, since some of the info attached to volumes from previous tests may be wrong (it gets set at create time; so if the config is bad when a volume is created, the info in Cinder's DB will be bad for that volume even after you correct the config files).

I was receiving precisely the same error on a dev environment deploy yesterday and this seemed to be the most exhaustive set of things I could think of to run through to diagnose the issue. Ultimately I was able to get it working.

As always, be sure to take note of any change in the log messages each time you test a change. Good luck!

As I understand the issue, there's not a lot you can do in a non-crude way here, other than modifying the code. That being said, two of the easier (crude) options available to you are:

1) Use iptables to limit the packets/second rate (see the limit and hashlimit module documentation in iptables). Something like this on your storage nodes may do the trick:

iptables -A OUTPUT -p tcp --dport 6000 -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 100/second -j ACCEPT

This effectively puts in place a rule that reduces the throughput rate of a connection to 50 packets/sec only after it sees that connection hit a rate of 100 packets/sec. Note that 50 and 100 here are sample values -- you'll have to determine what packets/sec rate is acceptable between object servers in your environment, and what the trigger threshold should be, but you get the idea.

2) Use tc (traffic control) to limit the actual throughput rate.

This is a bit more involved, but you'd essentially use iptables with -j MARK and --set-xmark to mark outbound packets on port 6000 with a specific class, then set up your tc rules to tell the kernel the maximum throughput rate for an interface, and the maximum throughput rate acceptable for packets marked with a specific mask. You can optionally get relatively fancy depending on the tc classes to make them adaptive based on current usage in other classes, and so forth.

Here's a few good links on tc if you wish to go this route: http://lartc.org/howto/ http://www.cyberciti.biz/faq/linux-tr... http://shearer.org/Linux_Shaping_Temp...

Can you post a link to the specific cookbook you're using, and the network related config info you're supplying to it?

To whoever downvoted, it's not constructive to downvote a valid question. What's obvious to one may not be obvious to another...the whole idea of this site is to share that sort of knowledge and help others. Just add a comment if you need more info.

What release are you running--Essex, Folsom, or Grizzly?

Are any of your other services having the same problem keeping a MySQL connection open?

System-level accounts on instances aren't affected by Keystone. Are you using SSH keypairs on your instances, or password-based authentication?

Also, to make it easier for others to read the configurations you posted, try editing your post, highlighting the config blocks, and clicking the "101 010" button in the editor to prefix each of the lines with 4 spaces. Once you save, it causes the site to present these in a readable code block.

Are you able to ping 172.16.1.201 from an instance?