Ask Your Question
9

how instance storage work?

asked 2013-12-19 03:10:13 -0600

PO gravatar image

updated 2013-12-19 09:00:30 -0600

larsks gravatar image

kvm-img create ubuntu_12.04_server.img 80G

On the compute node:

root@compute1:/# ls -lh /var/lib/nova/instances/183ee6d0-a363-46c9-952b-2fe38d3960c1(instance_id)**
total 7.0G
-rw-rw---- 1 libvirt-qemu kvm     1 Dec 17 17:17 console.log
-rw-r--r-- 1 libvirt-qemu kvm  7.0G Dec 19 16:51 disk
-rw-r--r-- 1 libvirt-qemu kvm  193K Dec 17 16:23 disk.local
-rw-r--r-- 1 libvirt-qemu kvm  193K Dec 17 16:23 disk.swap
-rw-r--r-- 1 nova         nova 2.0K Dec 17 17:16 libvirt.xml

root@compute1:/# ls -lh /var/lib/nova/instances/_base
total 14G
-rw-r--r-- 1 nova         kvm  79G Dec 19 16:34 5f9399500420f4a42ca62ae246660b311913c708
-rw-r--r-- 1 libvirt-qemu kvm 5.0G Dec 17 16:23 ephemeral_5_default
-rw-r--r-- 1 libvirt-qemu kvm 2.0G Dec 17 16:23 swap_2048

this the flavor:

Flavors-OpenStack-Dashboard 2013-12-19 16-54-22.png

I was confused by the directory _base, especially the 5f9399500420f4a42ca62ae246660b311913c708 (79G).Is it the img file from glance? In the instance directory /var/lib/nova/instances/183ee6d0-a363-46c9-952b-2fe38d3960c1,is the 'disk' file the device /dev/vda based on in the vm and what about the 'disk.local' file?

edit retag flag offensive close merge delete
add a comment

2 answers

Sort by ยป oldest newest most voted
15

answered 2013-12-19 09:17:40 -0600

larsks gravatar image

updated 2013-12-19 20:49:14 -0600

jtopjian gravatar image

An OpenStack environment will usually have many compute nodes. In order to avoid unnecessary network traffic, the first time an image is used on a compute node that image is fetched from Glance into the /var/lib/nova/instances/_base directory. The boot disk for your instance is then created as a qcow2 disk using the image in _base as a backing file. This allows even large disks to be provisioned very quickly once the image has been downloaded to the compute node, and it means the image only needs to be fetched once per compute node rather than once per instance.

You can use the qemu-img info command to examine a disk file and see what's using for a backing file. For example, on my system:

# qemu-img info /var/lib/nova/instances/0debe180-ac32-49a4-9caa-5eebe834eef5/disk 
image: /var/lib/nova/instances/0debe180-ac32-49a4-9caa-5eebe834eef5/disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 218M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/7d9817593f5a6f87fafc930737d8a55eb6932b7e

The disk file is the boot image for the instance, and disk.local is the "ephemeral" disk for the instance.

If you're running OpenStack in a KVM environment, you can see the devices actually attached to your instance with the virsh domblklist command. Run virsh list to get a list of instances:

# virsh list
 Id    Name                           State
----------------------------------------------------
 12    instance-0000003b              running
 13    instance-0000003c              running

And run domblkinfo on a specific instance:

# virsh domblklist 12
Target     Source
------------------------------------------------
vda        /var/lib/nova/instances/0debe180-ac32-49a4-9caa-5eebe834eef5/disk

This confirms that the disk file is used as the boot image for that instance.

edit flag offensive delete link more

Comments

Hi,larsks.Thanks for the detailed answer and I have a clearly understanding.But I also have a question. the 'disk' file size only 10M,but the instance's filesystem usage such as '/','/boot' is much more than 10M. Where are the '/','/boot' stored.Are they stored in the 'disk' file?

PO gravatar imagePO ( 2013-12-22 21:26:17 -0600 )edit

The `disk` file is a copy-on-write image. It only contains data that differs from that in the backing file.

larsks gravatar imagelarsks ( 2013-12-23 09:17:47 -0600 )edit

hi larsks, is there also a reference link between the file in _base and an image when there isn't an instance running anymore.

I want to link them to an image file so I could get an overview of which images are available (copied over) on which compute node..

visibilityspots gravatar imagevisibilityspots ( 2016-11-22 04:51:33 -0600 )edit

hi larsks

So you say Backing file is a shared image base file and all instances in compute node mapped to it for booting, any changes and installation after booting save in disk.local file, in this way, having one base image that bind to all vms can be point of failure, IOps is important and ...

p.bagherpour gravatar imagep.bagherpour ( 2017-01-25 06:42:01 -0600 )edit

So it is better to use a better backend for images, for ex. Ceph, but where? I can set Ceph for glance backend also should use ceph for compute node? because as you said images cache in compute node not glance node

p.bagherpour gravatar imagep.bagherpour ( 2017-01-25 06:44:39 -0600 )edit
see more comments
0

answered 2017-06-14 21:33:00 -0600

ducpx gravatar image

I'm using OpenStack Ocata. In /var/lib/nova/instances/0fa667ca-afd7-4f0e-9bde-2bd1c6c27841 directory does not have disk.local file. It has only console.log, disk and disk.info. Why not have disk.local file?

edit flag offensive delete link more
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

6 followers

subscribe to rss feed

Stats

Asked: 2013-12-19 03:10:13 -0600

Seen: 14,448 times

Last updated: Jun 14 '17

Related questions

New instance stuck on scheduling after lunch

While launching an instance, I get the error, "ERROR(BadRequest) : invalid key_name provided(http 400). Please help me with this

resize instance to smaller flavor

Cannot ping/ssh to CirrOS/Ubuntu instance

nova boot instance failing

Openstack instance ip_address

fuel storage node deploy with ceph and 2 fiber channel volumes

how to get instance delete time

san storage

High Availability of compute node's instances