Ask Your Question
0

Help resizing bootstack VM kvm which is 100% full

asked 2015-09-22 12:19:48 -0600

LouisinLondon gravatar image

Hi

Background:

I have inherrited sole ownership of an 14 node openstack which was setup via Canonical "jumpstart" which was working fine for the mostpart but is now broken completely..

cant juju or anything

Error / warning:

@maas:~$ juju status
WARNING ignoring invalid API endpoint environment UUID

but can luckily still ssh to VMs from maas, and ssh to other VMs that are part of the infrastructure from there.

Problem:

bootstack vm on the main hypervisor is at 100% disk usage.

@bootstrap:/home$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       9.8G  9.8G     0 100% /

What I've done: ps –faux | grep bootstrap -> shows the drive file is under /dev/mapper/system-bootstrap

and

@hypervisor:~$ ls -ahlti /dev/mapper

total 0
1440 lrwxrwxrwx  1 root root       8 Sep 22 15:50 system-bootstrap -> ../dm-11

I've done a extention of the lvm

@hypervisor:
sudo lvextend --size +10G /dev/system/bootstrap
  Extending logical volume bootstrap to 20.00 GiB
  Logical volume bootstrap successfully resized

but I'm not very use to openstack style virtualization (mostly work with VSphere )

PS: We're getting a Canonical support contract in place but that might take ages and researchers want the data out of some VMs which were happily working even though the dashboard was inaccessible.

Confusion:

I know that after resizing the LVM partitions would then usually need to be resized but super confused with the way this was setup...

hypervisor:~$ ls -ahlti /dev/mapper | grep bootstrap
1440 lrwxrwxrwx  1 root root       8 Sep 22 15:50 system-bootstrap -> ../dm-11

@hypervisor:~$ ls -ahlti /dev/ | grep dm-11
 9267 brw-rw----  1 libvirt-qemu kvm     252,  11 Sep 22 17:13 dm-11

Question:

Can I just do a

virsh shutdown bootstrap

and then resize using qemu-img ?

http://serverfault.com/questions/324281/how-do-you-increase-a-kvm-guests-disk-space

1.  stop the VM 
2.  run qemu-img resize vmdisk.img +10G to increase image size by 10Gb
3.  start the VM, resize the partitions and LVM structure within it normally

Help *sigh

edit retag flag offensive close merge delete

Comments

qemu-img won't work on a device file. It only works on disk files of type qcow2 and similar. If there is a filesystem on /dev/dm-11, you need to increase it as well. If it's ext[234], use resize2fs, or if XFS, xfs_growfs.

Bernd Bausch gravatar imageBernd Bausch ( 2015-09-24 06:21:43 -0600 )edit

After that, I guess the VM must be told that the disk size has changed. If it's a SCSI disk, it would be echo > /sys/block/sda/device/rescan. You seem to have virtio though, and I don't know what to do there. Based on http://linux-kvm.com/content/how-resi..., reboot the VM.

Bernd Bausch gravatar imageBernd Bausch ( 2015-09-24 06:24:42 -0600 )edit

Note I know nothing about Juju and Jumpstart and how it's implemented

Bernd Bausch gravatar imageBernd Bausch ( 2015-09-24 06:25:25 -0600 )edit

Thanks Bernd. Yeah openstack is a beast (powerful but a beast)

I've tried resizefs inside the bootstrap VM and no luck.

  1. @bootstrap:~$ sudo resize2fs /dev/vda1 resize2fs 1.42.9 (4-Feb-2014) The filesystem is already 2621184 blocks long. Nothing to do!
LouisinLondon gravatar imageLouisinLondon ( 2015-09-24 09:14:20 -0600 )edit

I've also tried growing the filesystem from the Hypervisor: @hypervisor:~$ sudo resize2fs /dev/system/bootstrap resize2fs 1.42.9 (4-Feb-2014) resize2fs: Bad magic number in super-block while trying to open /dev/system/bootstrap Couldn't find valid filesystem superblock.

LouisinLondon gravatar imageLouisinLondon ( 2015-09-24 09:14:52 -0600 )edit

2 answers

Sort by » oldest newest most voted
0

answered 2016-03-14 10:41:46 -0600

LouisinLondon gravatar image

updated 2016-03-14 10:46:56 -0600

OK. I figured it out in Sept 2015 but forgot to add the solution here... Today we had to do it again and with the help of the Canonical support my memory was jogged. Here's the solution to resizing VMs running under Virsh using LVM's on the Hypervisor..

Firstly increase the Logical Volume: (in this case we are trying to resize the controller VM

sudo lvextend --size +10G /dev/system/controller

After it has resized the LVM to make space for a larger VM disk, we need to use "virt-" commands to rename the current disk to .backup, make new vm disks, and basically copy and expand the data from the old disks onto the new disks.

You can do a

man virt-resize

before you begin.

TO DO : Increase Controller from 10Gb to 20Gb
=====================
virsh shutdown controller
sudo virt-filesystems --long --parts --blkdevs -h -a /dev/system/controller 
sudo lvrename /dev/system/controller /dev/system/controller.backup
sudo lvcreate system -n controller -L 20GB 
sudo virt-resize /dev/system/controller.backup /dev/system/controller --expand /dev/vda1

Then after that

virsh start controller

@hypervisor: ssh maas

from maas

ssh controller

df -h to make sure the controller VM (or whichever VM you were increasing it's size did get it's Filesystem resized)

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  8.5G   11G  46% /

Then when you are happy your Openstack is working again Remove the Temporary backups of the FileSystems you made

sudo lvremove /dev/system/controller.backup
edit flag offensive delete link more
0

answered 2015-09-24 09:43:23 -0600

LouisinLondon gravatar image

updated 2015-09-24 09:44:10 -0600

NOT the answer but an update:

I read up about virsh blockresize and tried on the VM... http://serverfault.com/questions/604876/online-resizing-of-kvm-guest-root-filesystem (http://serverfault.com/questions/6048...)

@hypervisor:~$ sudo virsh blockresize bootstrap --path /dev/system/bootstrap --size 20G error: Failed to resize block device '/dev/system/bootstrap' error: Requested operation is not valid: domain is not running

PS: this also has the wrong path for the file... /dev/system/bootstrap

so a) sudo virsh start bootstrap b) find the path using ps -faux | grep bootstrap which will show file=/dev/mapper/system-bootstrap

Then:

@hypervisor:~$ sudo virsh blockresize bootstrap --path /dev/mapper/system-bootstrap --size 20G Block device '/dev/mapper/system-bootstrap' is resized

except it doesn't seem to make any difference:

Using fdisk shows (I think similarly to previously after the first lvextend that there is now 21.5Gb hdd avail..) but the OS just wont resize... at this point I wish there was just a way to boot the VM using "gparted live" and resize...

@bootstrap:~$ sudo fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes 16 heads, 63 sectors/track, 41610 cylinders, total 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

Device Boot Start End Blocks Id System /dev/vda1 * 2048 20971519 10484736 83 Linux

edit flag offensive delete link more

Comments

Now that the guest knows that its device has grown to 21Gb, its filesystem must be resized as well, probably with resize2fs.

Bernd Bausch gravatar imageBernd Bausch ( 2015-09-24 23:44:47 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-09-22 12:19:48 -0600

Seen: 841 times

Last updated: Mar 14 '16