Ask Your Question
2

machine creation really slow

asked 2014-04-10 10:32:06 -0500

Itxaka gravatar image

Hello,

we have a 3 node setup of openstack havanna, using the automated rackspace cookbooks.

Everything work correctly but I have noticed that when creating new mahcines with a flavour that has a big HDD (400Gb for example) it will take like 20 minutes too boot up the machine.

Looking at the /var/lib/nova/<instance_id> I can see the "disk" file growing but it's incredibly slow!! it does about 3Mb/s.

I played with the image configuration in nova but the machines are still slow as hell.

The nodes have 4 2Tb HDD with LVM.

This is an iotop of one of the nodes using about 7Mb/s, as you can see it's only doing writes because it's creating the HDD files for the machines:

  778 be/3 root        0.00 B/s    7.88 K/s  0.00 % 21.97 % [jbd2/dm-0-8]
24194 be/4 libvirt-    0.00 B/s 1613.74 B/s  0.00 %  6.77 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000266 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24105 be/4 libvirt-    0.00 B/s 1613.74 B/s  0.00 %  5.73 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000266 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
  449 be/4 libvirt-    0.00 B/s   14.97 K/s  0.00 %  4.89 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-0000027e -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
  655 be/4 libvirt-    0.00 B/s   14.97 K/s  0.00 %  4.61 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-0000027e -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
  574 be/4 libvirt-    0.00 B/s   10.24 K/s  0.00 %  4.43 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000274 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24084 be/4 libvirt-    0.00 B/s 1613.74 B/s  0.00 %  4.19 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000266 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24085 be/4 libvirt-    0.00 B/s 1613.74 B/s  0.00 %  4.09 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000266 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24074 be/4 libvirt-    0.00 B/s 1613.74 B/s  0.00 %  3.93 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000264 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24258 be/4 libvirt-    0.00 B/s   14.38 K/s  0.00 %  3.83 % qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000264 -S -machi~en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
24228 be/4 libvirt-    0.00 B/s  404.22 K/s  0.00 %  3.67 % qemu-system-x86_64 -machine accel=kvm ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-04-16 06:49:38 -0500

Hello,

When you say the flavour has a 400GB disk, how much disk space does the image really use in Glance ?

In summary, what happens on boot:

Nova checks to see if the image exists locally (in cache) if not, it will download it from Glance. Assuming Glance is not running under any web server like Apache, this will eat some CPU on the server running Glance and will take some time if the instance is big enough.

If the image is cached locally (after you first create an instance, depending on your image cache configuration), then a copy is created from cache. This should be much faster, but it depends on the size of the actual image. If you have an image that's 100GB (the actual image, not the filesystem inside the image), then copying 100GB of data would take a long time.

If you have a single LVM for the entire OS (compute node) and ephemeral storage, when you create an instance from a cached image, the image needs to be copied locally from one place to another, this will also slow down instance creation.

Check what processes are running and using CPU/I/O on the compute node while you create an instance and if the disk I/O is maxed out (most likely).

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-04-10 10:32:06 -0500

Seen: 854 times

Last updated: Apr 16 '14