Magnum kubernetes cluster stucks in create in progress state (exactly on kube-master))

asked 2018-10-03 03:55:25 -0600

Am Ab gravatar image

updated 2018-10-03 05:58:50 -0600

Hello,

the kube master has been created and I am able to ssh. but I've found the following in cloud-init-output log. the cluster will be Timed Out afterwards, I've tried that 3~4 times.



Cloud-init v. 17.1 running 'init-local' at Wed, 03 Oct 2018 07:55:46 +0000. Up 37.98 seconds.
Cloud-init v. 17.1 running 'init' at Wed, 03 Oct 2018 08:09:09 +0000. Up 841.08 seconds.
ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | eth0:  | True |  10.0.0.5 | 255.255.255.0 |   .   | fa:16:3e:b1:e6:48 |
ci-info: | eth0:  | True |     .     |       .       |   d   | fa:16:3e:b1:e6:48 |
ci-info: |  lo:   | True | 127.0.0.1 |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |     .     |       .       |   d   |         .         |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | Route |   Destination   | Gateway  |     Genmask     | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: |   0   |     0.0.0.0     | 10.0.0.1 |     0.0.0.0     |    eth0   |   UG  |
ci-info: |   1   |     10.0.0.0    | 0.0.0.0  |  255.255.255.0  |    eth0   |   U   |
ci-info: |   2   | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 |    eth0   |  UGH  |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
Cloud-init v. 17.1 running 'modules:config' at Wed, 03 Oct 2018 08:09:25 +0000. Up 857.28 seconds.
+ CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem
+ '[' -n '' ']'

/var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No such file or directory
/var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No such file or directory
/var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No such file or directory
Cloud-init v. 17.1 running 'modules:final' at Wed, 03 Oct 2018 08:09:27 +0000. Up 859.32 seconds.
2018-10-03 08:09:40,001 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-007 [1]
Removed /etc/systemd/system/multi-user.target.wants/docker-storage-setup.service.
  New size given (1280 extents) not larger than existing size (4863 extents)
ERROR: There is not enough free space in volume group atomicos to create data volume of size MIN_DATA_SIZE=2G.
2018-10-03 08:09:40,699 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-009 [1]
configuring kubernetes (master)




sed: can't read /etc/kubernetes/config: No such file or directory
sed: can't read /etc/kubernetes/apiserver: No such file or directory
sed: can't read /etc/kubernetes/controller-manager: No such file or directory
sed: can't read /etc/kubernetes/scheduler: No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1439  100  1439    0     0    162      0  0:00:08  0:00:08 --:--:--   393
Generating RSA private key, 4096 bit long modulus
.............................................................................++++
......++++
e is 65537 (0x010001)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5749  100  3822  100  1927    343    173  0:00:11  0:00:11 --:--:--   977
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload ...
(more)
edit retag flag offensive close merge delete

Comments

This is unreadable. Can you edit your question and format the cloudiniy log as code?

Interesting: “ERROR: There is not enough free space in volume group atomicos to create data volume of size MIN_DATA_SIZE=2G”. Is the flavor large enough? Can you check the master’s disk size?

Bernd Bausch gravatar imageBernd Bausch ( 2018-10-03 05:14:18 -0600 )edit

I've edited that, if it not readable yet I can paste it somewhere else. Also it is the default m1.small flavor with 2G disk. When I ssh to fedora vm, the atomicos-root has 18G free.

/dev/mapper/atomicos-root 19G 1.3G 18G 7% /sysroot

Am Ab gravatar imageAm Ab ( 2018-10-03 06:04:32 -0600 )edit
1

Great for my eyes, thanks!

Something went horribly wrong provisioning this instance. I don’t understand why there is no room for a 2G volume in atomicos?

I would start with the Heat stacks. Troubleshooting tips.

Bernd Bausch gravatar imageBernd Bausch ( 2018-10-03 09:23:09 -0600 )edit