Ask Your Question

jaypipes's profile - activity

2016-11-02 15:02:35 -0500 received badge  Famous Question (source)
2016-11-02 15:02:35 -0500 received badge  Notable Question (source)
2014-11-18 08:58:23 -0500 received badge  Popular Question (source)
2014-02-21 15:18:50 -0500 received badge  Good Answer (source)
2014-02-05 09:50:03 -0500 received badge  Nice Answer (source)
2014-01-09 11:46:42 -0500 answered a question Best practice to achieve High Availability and scalability for Neutron

The Neutron L3 agent is the only OpenStack service that is not stateless, and therefore you cannot use traditional load-balancing across a set of identical nodes.

That said, there's nothing wrong with running multiple L3 nodes, with routers for different tenants hosted on different L3 agents. We do this successfully in our deployment using a custom Neutron scheduler that my colleague Alan Meadows wrote and a Python script (also written by Alan) that runs in cron looking for failures on an L3 agent and if found, moves the routers from the failed node to a working one.

The advantage to this vs. something like pacemaker is that you spread the L3 agent workload across many nodes -- accomplishing a sort of poor-man's load balancing/sharding for L3 agent requests.

To summarize, if your installation is running Grizzly or Havana and you don't want use Pacemaker (so that you can spread L3 agent load across multiple nodes):

  1. Apply this patch to Neutron: https://gist.github.com/jaypipes/8135839
  2. Set the router_scheduler_driver in nova.conf to neutron.scheduler.l3_agent_scheduler.LeastUtilizedScheduler Reference: https://github.com/stackforge/cookbook-openstack-network/blob/master/templates/default/neutron.conf.erb#L232
  3. Put this script into a cron job: https://github.com/stackforge/cookbook-openstack-network/blob/master/files/default/neutron-ha-tool.py

If you're on Icehouse, simply do:

  1. Set the router_scheduler_driver in nova.conf to neutron.scheduler.l3_agent_scheduler.LeastRoutersScheduler Reference: https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L223
  2. Put this script into a cron job: https://github.com/stackforge/cookbook-openstack-network/blob/master/files/default/neutron-ha-tool.py

Things may change, so make sure you also check the High Availability documentation in the sections:

  • Network Controller Cluster Stack
    • Highly available Neutron L3 Agent
    • Highly available Neutron DHCP Agent
    • Highly available Neutron Metadata Agent
    • Manage network resources
2013-06-02 09:24:45 -0500 received badge  Good Answer (source)
2013-05-21 10:42:59 -0500 received badge  Supporter (source)
2013-05-09 09:59:46 -0500 received badge  Nice Answer (source)
2013-05-08 14:43:30 -0500 answered a question How to enable HTTPs in Grizzly Quantum

Generally, you should use something like nginx or a load balancer to offload and terminate SSL. Python OpenSSL package is very slow compared to having a separate service do SSL termination.

2013-04-23 02:54:27 -0500 received badge  Nice Answer (source)
2013-04-05 13:17:29 -0500 answered a question Is Horizon thread safe?

Horizon is run inside one of more processes within a web server container (like mod_wsgi in Apache). It doesn't really have much state of its own, besides session management, but if you are asking whether Horizon, the web application, can handle lots of concurrent users without blocking other users, the answer is yes. But it's not strictly multi-threaded, per-se. It's multi-process -- via the containing servlet.

2013-03-22 19:02:00 -0500 received badge  Nice Answer (source)
2013-03-22 16:43:27 -0500 received badge  Teacher (source)
2013-03-22 16:35:52 -0500 answered a question Why marker instead of SQL OFFSET in oslo paginate-query?

Hi David,

Here's the thread (from nearly 2 years ago) where Justin Santa Barbara and I discussed the reasons around using marker/limit instead of LIMIT x OFFSET y:

http://lists.launchpad.net/openstack/msg02548.html

Best, -jay

2013-03-18 15:18:02 -0500 received badge  Autobiographer
2013-02-27 16:43:20 -0500 answered a question glance-manage db_sync running problem

What does your glance-registry.conf look like?

2013-02-27 16:32:19 -0500 answered a question glance-manage db_sync running problem

sudo glance-manage db_sync

2013-02-24 19:33:21 -0500 answered a question Endpoint not found

You need a /v1 at the end of your image URIs.

2012-10-23 14:32:44 -0500 answered a question "glance-manage db_sync " problem

This is merely a warning being produced from SQLAlchemy. You can safely ignore this.

2012-08-23 08:33:33 -0500 answered a question upload a kernel error

Hi DigitalWonk,

The inconsistencies with the parameters are addressed in the next-generation client ( http://github.com/openstack/python-gl... ). The old glance client will be deprecated in the Grizzly release cycle.

Best, -jay

2012-08-07 19:53:23 -0500 answered a question Can't ping or ssh an instance on anything but control node

Looks like this was solved with multi_host=True... please close if so.

2012-08-06 22:08:27 -0500 answered a question Can't ping or ssh an instance on anything but control node

What are the rules for the security group that you start the instances with?

2012-08-06 17:02:51 -0500 answered a question Glance development environment

Joao, you can create a config file and paste INI file and put them in your ~ directory. Make bind_port something that is not used on your system already (for example, don't use 9292 or 9191 since Glance uses those ports by default. Make sure that the bind_port option in your glance-registry.conf file and the registry_port option in your glance-api.conf file match.

The start the servers using tools/with_venv.sh glance-api --config-file=~/glance-api.conf &

Do similar for the glance-registry daemon.

Alternately, you can use devstack to spin up a whole OpenStack environment, and have devstack's stackrc branch file point to your local development repo. Then just run stack.sh and you will have glance-api and glance-registry servers in screen sesssions.

-jay

2012-07-25 11:50:53 -0500 answered a question Snapshots stay in queued status

This seems like a bug in Horizon. It is not supplying the container_format required parameter to the image_add() method of the client. I will turn this into a bug on Horizon for you.

Best. -jay

2012-07-23 17:59:46 -0500 answered a question glance backend with swift upload image problem

You do not have SSL set up correctly on your Swift proxy server. Thus, the error about SSL in the return from Glance...

2012-07-23 16:58:29 -0500 answered a question glance 401 unauththorized

Joeu, try adding -S keystone to your glance CLI tool calls. That's the only thing I can think of.. everything else seems to be correct!

2012-07-20 14:44:52 -0500 answered a question glance 401 unauththorized

It looks from the log output that 012345SECRET99TOKEN012345 is not a valid token. If you grab a token using a curl command to Keystone:

curl -d '{"auth":{"passwordCredentials":{"username": "adminUser", "password": "secretword"}}}' -H "Content-type: application/json" http://localhost:35357/v2.0/tokens

The token will be returned in that request. Try using that token with the -A option to the glance client.

2012-07-20 14:39:50 -0500 answered a question glance 401 unauththorized

Joeu, did you also use the CentOS/yum install method?

2012-07-02 22:16:57 -0500 answered a question Glance development environment

On 07/02/2012 05:21 PM, João Pereira wrote:

Hi,

I'm starting with OpenStack and most precisely with Glance, and although I've very few experience with Python, I've a very strong Ruby background so I think I can learn Python while experimenting with Glance directly.

Hi! We always appreciate new contributors. Welcome :)

However, I dont know what's the recommended development environment if wanting to experiment with glance in order to contribute with code improvements...

Is it better to install it with the package manager and go experimenting by editing the code in the packages installation location (I guess this is not the best option)

No, absolutely not... the best way to begin contributing is to follow the contributor guidelines and get a local Git repository set up for developing Glance.

Start here:

http://wiki.openstack.org/GerritWorkflow

Do the following:

  • Clone the Glance repo to your local workstation

git clone git://github.com/openstack/glance cd glance

  • Install all the necessary dependencies for doing development and testing

Depends on your workstation. For Ubuntu, do:

apt-get install -y gcc libxslt-dev libxml2-dev python-dev

  • Run the test suite in a virtual environment

./run_tests.sh -V

The above will create a virtualenv ( http://www.virtualenv.org/en/latest/i... ) and install all the development stuff for Glance into the virtual environment and then run the Glance test suite.

After that, hop onto Freenode.net #openstack-dev and chat with the PTL, Brian Waldon (bcwaldon) and ask about starter bugs or tasks you might be able to take a hack at.

All the best, -jay

or is it recommended to fetch the source code from git and patch and run the code/servers manually (./bin/glance ...)? If yes, how do we handle the .ini and .conf files?

I'm seeking for best practice advices from the programmers already integrated in the project. If anyone can point me some guidelines/tools/tutorials,etc I would be grateful. I've already read the "how to contribute" wikis but cant find any guidelines about the code development environment.

2012-07-02 17:39:48 -0500 answered a question Glance with S3 backend store

You are probably getting hit by the leap second issue... try refreshing your NTP date on the host that is running the Glance code.

2012-06-12 15:41:23 -0500 answered a question Centos Cloud Install

Hi again, Peter, sorry for delayed response....

You can grep through the nova-compute logs looking for "libvirtError".

The rule of thumb I'm using is that /tmp should just have enough space to be able to complete a resize and/or snapshot operation for the largest flavors that you plan on using on the box. The resize and snapshot operations in KVM uses the qemu-img and resize2fs command line tools, and these tools (AFAIK) set aside an area in /tmp by default to build new images. The space needed depends on how big the images being snapshotted or resized are (in RAM)

Best, -jay

2012-06-12 05:21:50 -0500 answered a question Centos Cloud Install

Make sure you have enough space allocated to the partition/volume that you are using for /tmp. resize2fs and libvirt will error out with Unable to write errors if there is not enough space in /tmp to deal with the resize operation.

Do an:

sudo ln -s /some/directory/with/space /tmp

if you need to...

-jay

2012-06-11 14:34:47 -0500 answered a question Glance images on S3 storage - shared

Peter, you can add the image (as shown in previous post) to Glance, and then on the second and N glance registry servers, you can add the:

glance add id=<uuid_of_first_image> ...

to set the ID at time of adding. However, that said, your images will still have separate URIs, although the UUID will be the same, of course, as the glance API endpoints will be different, I assume :)

Best, -jay

2012-06-11 13:46:09 -0500 answered a question Glance images on S3 storage - shared

You can upload your images to S3 and then add those image locations to Glance using the location field. For instance:

glance add location=http+s3://AWSACCOUNTID:AWSACCOUNTKEY@s3.amazonaws.com/BUCKET/OBJECTNAME name="My S3 Image" disk_format=ami container_format=ami is_public=True

You will get a different image ID in each of your glance registry servers, however.

Best, -jay

2012-04-27 18:16:29 -0500 answered a question essex: glance and swift intergration question.

Hi! That last error means that the Swift client is old -- it does not include the new auth_version parameter in its constructor. The solution is to install a new version of python-swift package

All the best, -jay

2012-04-13 18:29:10 -0500 answered a question incorrect result when executed glance index with keystone

Turning this into a packaging bug, Rain..

2012-04-12 14:06:11 -0500 answered a question incorrect result when executed glance index with keystone

Anything in the glance-api log, Rain?

Best, -jay

2012-04-12 14:01:41 -0500 answered a question glance image upload fails during installation

Hi Sumit,

What does this return? Please pastebin it. Thanks!

curl -v -X POST -H "X-Auth-User: <service_username>" -H "X-Auth-Key: <service_password>" http://<keystone_host>:5000/v2.0/tokens

2012-04-09 19:59:21 -0500 answered a question glance image upload fails during installation

If you do:

curl -v -H "X-Auth-User: <service_username>" -H "X-Auth-Key: <service_password>" http://<keystone_host>:35357/v2.0/tokens/03fbad35089b4f5684667b237421f76e

and replace with appropriate service user/password credentials, what is the response?

-jay

2012-04-09 17:56:13 -0500 answered a question HTTPNotFound

Hi Rain,

It seems to me the error message is self-explanatory. It means the image file "70" does not exist in your /srv/glusterfs/glance/images directory...

As for encrypt/decrypt, Glance does not support encryption/decryption of image files. While Glance supports servers in SSL mode, I'd recommend using something like Pound to do SSL termination instead...

Best, -jay

2012-04-02 15:22:18 -0500 answered a question Error uploading image: (SSLError): [Errno 1] _ssl.c:499: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

antiError, please also open a new Question. This one was marked closed quite some time ago.

Best, -jay

2012-04-02 15:21:47 -0500 answered a question Error uploading image: (SSLError): [Errno 1] _ssl.c:499: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

anitError, could you provide specifics of the error you are getting please? Thanks, -jay

2012-03-27 16:20:26 -0500 answered a question Swift backend with Keystone v1 auth

Nicolas, you will have to wait until the bug fix for the v2 auth in the Glance Swift driver to hit.

Here is the code review happening right now on it:

https://review.openstack.org/#change,...

2012-03-22 15:34:49 -0500 answered a question glance add problem:NotAuthorized: None

What version of Keystone and Glance are you using? Thanks!

2012-03-20 00:42:14 -0500 answered a question Keystone integration in glance

Sanjaya, please ask your question on the Keystone forum:

https://answers.launchpad.net/keyston...

Thanks! jay

2012-02-20 16:00:12 -0500 answered a question 2011.3 upgrade to 2012.1~e3

Well, to be safe, you will want to back up your image files and Glance registry database before doing anything.

After that, you should just be able to do a standard upgrade (whatever method you use in your operating system) and then do a glance-manage db_sync

Best,

-jay

2012-01-27 17:58:21 -0500 answered a question Run a Instance from Glance

You can use the Nova API to launch an instance (or you can use OpenStack Dashboard/Horizon to do the same in a GUI).

Please see here for more information:

http://docs.openstack.org/api/opensta...

Cheers! -jay

2012-01-17 18:37:52 -0500 answered a question What is the plan for uploading tarballs?

Hmm, interesting... I did not know that... alright, lemme look further into this and see what the deal is. I don't really understand how this is the case since you do not need to supply a kernel when starting things other than an AMI-style image.

2012-01-13 21:08:49 -0500 answered a question What is the plan for uploading tarballs?

David, devstack isn't uploading tarballs to Glance... it is manually untar'ing a tarball and adding the kernel and ramdisk images included in the UEC-style tarball, and then manually uncompressing the disk image contained in the tarball and adding that to glance along with the kernel_id and ramdisk_id custom properties set to the UUIDs generated by glance for the kernel and ramdisk images.

Technically, adding the kernel and ramdisk is entirely optional and AFAICT, is only done because Horizon's "edit image" form includes fields for kernel and ramdisk IDs because Horizon was originally an EC2 API only thing and kernel and ramdisk only really makes sense in the EC2 world. When you call euca-describe-images, the kernel and ramdisk custom properties that are stored in Glance are displayed.

-jay