Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Magnum kubernetes cluster stucks in create in progress state (exactly on kube-master))

Hello,

the kube master has been created and I am able to ssh. but I've found the following in cloud-init-output log. the cluster will be Timed Out afterwards, I've tried that 3~4 times.

Cloud-init v. 17.1 running 'init-local' at Wed, 03 Oct 2018 07:55:46 +0000. Up 37.98 seconds. Cloud-init v. 17.1 running 'init' at Wed, 03 Oct 2018 08:09:09 +0000. Up 841.08 seconds. ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++ ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: | eth0: | True | 10.0.0.5 | 255.255.255.0 | . | fa:16:3e:b1:e6:48 | ci-info: | eth0: | True | . | . | d | fa:16:3e:b1:e6:48 | ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . | ci-info: | lo: | True | . | . | d | . | ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++ ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG | ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U | ci-info: | 2 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH | ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ Cloud-init v. 17.1 running 'modules:config' at Wed, 03 Oct 2018 08:09:25 +0000. Up 857.28 seconds. + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem + '[' -n '' ']'

/var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No such file or directory Cloud-init v. 17.1 running 'modules:final' at Wed, 03 Oct 2018 08:09:27 +0000. Up 859.32 seconds. 2018-10-03 08:09:40,001 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-007 [1] Removed /etc/systemd/system/multi-user.target.wants/docker-storage-setup.service. New size given (1280 extents) not larger than existing size (4863 extents) ERROR: There is not enough free space in volume group atomicos to create data volume of size MIN_DATA_SIZE=2G. 2018-10-03 08:09:40,699 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-009 [1] configuring kubernetes (master)

sed: can't read /etc/kubernetes/config: No such file or directory sed: can't read /etc/kubernetes/apiserver: No such file or directory sed: can't read /etc/kubernetes/controller-manager: No such file or directory sed: can't read /etc/kubernetes/scheduler: No such file or directory % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1439 100 1439 0 0 162 0 0:00:08 0:00:08 --:--:-- 393 Generating RSA private key, 4096 bit long modulus .............................................................................++++ ......++++ e is 65537 (0x010001) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5749 100 3822 100 1927 343 173 0:00:11 0:00:11 --:--:-- 977 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1439 100 1439 0 0 143 0 0:00:10 0:00:10 --:--:-- 376 Generating RSA private key, 4096 bit long modulus ............++++ ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++++ e is 65537 (0x010001) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6379 100 4242 100 2137 500 251 0:00:08 0:00:08 --:--:-- 1294 + _prefix=docker.io/openstackmagnum/ + atomic install --storage ostree --system --system-package no --set REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt --name heat-container-agent docker.io/openstackmagnum/heat-container-agent:rawhide

  • systemctl start heat-container-agent Failed to start heat-container-agent.service: Unit heat-container-agent.service not found. 2018-10-03 08:10:59,526 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-013 [5] starting services activating service etcd Failed to enable unit: Unit file etcd.service does not exist. Failed to start etcd.service: Unit etcd.service not found. activating service docker activating service kube-apiserver Failed to enable unit: Unit file kube-apiserver.service does not exist. Failed to start kube-apiserver.service: Unit kube-apiserver.service not found. activating service kube-controller-manager Failed to enable unit: Unit file kube-controller-manager.service does not exist. Failed to start kube-controller-manager.service: Unit kube-controller-manager.service not found. activating service kube-scheduler Failed to enable unit: Unit file kube-scheduler.service does not exist. Failed to start kube-scheduler.service: Unit kube-scheduler.service not found. creating /usr/local/bin/flannel-config Created symlink /etc/systemd/system/multi-user.target.wants/flannel-config.service → /etc/systemd/system/flannel-config.service. Failed to start flannel-config.service: Unit etcd.service not found. activating service flanneld Failed to enable unit: Unit file flanneld.service does not exist. Failed to start flanneld.service: Unit flanneld.service not found. 2018-10-03 08:11:00,681 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-016 [5]
  • . /etc/sysconfig/heat-params ++ PROMETHEUS_MONITORING=False ++ KUBE_API_PUBLIC_ADDRESS=192.168.41.143 ++ KUBE_API_PRIVATE_ADDRESS=10.0.0.5 ++ KUBE_API_PORT=6443 ++ KUBE_NODE_PUBLIC_IP=192.168.41.143 ++ KUBE_NODE_IP=10.0.0.5 ++ KUBE_ALLOW_PRIV=true ++ ENABLE_CINDER= ++ ETCD_VOLUME=d5b74d83-e8a2-4b96-9dd9-b5bf1b360176 ++ ETCD_VOLUME_SIZE=0 ++ DOCKER_VOLUME=7d496030-2845-4d7e-a8a0-01bb354bb047 ++ DOCKER_VOLUME_SIZE=0 ++ DOCKER_STORAGE_DRIVER=devicemapper ++ NETWORK_DRIVER=flannel ++ FLANNEL_NETWORK_CIDR=10.100.0.0/16 ++ FLANNEL_NETWORK_SUBNETLEN=24 ++ FLANNEL_BACKEND=udp ++ PODS_NETWORK_CIDR=10.100.0.0/16 ++ PORTAL_NETWORK_CIDR=10.254.0.0/16 ++ ADMISSION_CONTROL_LIST=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota ++ ETCD_DISCOVERY_URL=https://discovery.etcd.io/7dae7a905b725a5273cb38bd1cf735df ++ USERNAME=admin ++ PASSWORD=ChangeMe ++ CLUSTER_SUBNET=0ae61531-fcc3-46bf-b18a-3df682182db3 ++ TLS_DISABLED=False ++ KUBE_DASHBOARD_ENABLED=True ++ INFLUX_GRAFANA_DASHBOARD_ENABLED=False ++ VERIFY_CA=True ++ CLUSTER_UUID=55276ad7-2c18-4b65-8c13-9849f21ff2d2 ++ MAGNUM_URL=http://192.168.41.146:9511/v1 ++ VOLUME_DRIVER= ++ HTTP_PROXY=http://xx.xx.xx.xx:80 ++ HTTPS_PROXY=http://xx.xx.xx.xx:80 ++ NO_PROXY= ++ WAIT_CURL='curl -i -X POST -H '\''X-Auth-Token: gAAAAABbtHOCrXV6lEkrpJ_JK4HW1ieMSOqufovUCzKhDroRipKJBeia7VpRqJgj7XSaok2EPy-fx5HBt3rra4XQ9vEugANcNfgDU3N8FQ_jKZoX-mHb0DUY8Cs2crkuVAhTPPxFRQwcXY33hNmz7-nZJWrBQ4hBkrWK09k6yijpzL-VhK6ZwKE'\'' -H '\''Content-Type: application/json'\'' -H '\''Accept: application/json'\'' http://192.168.41.146:8004/v1/2452e0a099c74afc94ebcde35be10a1f/stacks/kubernetes-cluster-ofga4pkibuog-kube_masters-7d5h4nj3he2s-0-qpqfcpg34kib/73402937-3adb-40a9-be8c-09f216c8d525/resources/master_wait_handle/signal' ++ KUBE_TAG=v1.9.3 ++ ETCD_TAG=v3.2.7 ++ FLANNEL_TAG=v0.9.0 ++ KUBE_VERSION=v1.9.3 ++ KUBE_DASHBOARD_VERSION=v1.8.3 ++ TRUSTEE_USER_ID=26886307c1444f4fa4c32bc1985d6619 ++ TRUSTEE_PASSWORD=p6jodu8tchwNGoHyn5 ++ TRUST_ID= ++ AUTH_URL=http://192.168.41.146:5000/v3 ++ INSECURE_REGISTRY_URL= ++ CONTAINER_INFRA_PREFIX= ++ SYSTEM_PODS_INITIAL_DELAY=30 ++ SYSTEM_PODS_TIMEOUT=5 ++ ETCD_LB_VIP= ++ DNS_SERVICE_IP=10.254.0.10 ++ DNS_CLUSTER_DOMAIN=cluster.local ++ CERT_MANAGER_API=False ++ CA_KEY= ++ CALICO_TAG=v2.6.7 ++ CALICO_CNI_TAG=v1.11.2 ++ CALICO_KUBE_CONTROLLERS_TAG=v1.0.3 ++ CALICO_IPV4POOL=192.168.0.0/16 ++ INGRESS_CONTROLLER= ++ INGRESS_CONTROLLER_ROLE=ingress ++ KUBELET_OPTIONS= ++ KUBECONTROLLER_OPTIONS= ++ KUBEAPI_OPTIONS= ++ KUBEPROXY_OPTIONS= ++ KUBESCHEDULER_OPTIONS=
  • echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API...
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5
  • curl --silent http://127.0.0.1:8080/version
  • sleep 5

Magnum kubernetes cluster stucks in create in progress state (exactly on kube-master))

Hello,

the kube master has been created and I am able to ssh. but I've found the following in cloud-init-output log. the cluster will be Timed Out afterwards, I've tried that 3~4 times.

times.

Cloud-init v. 17.1 running 'init-local' at Wed, 03 Oct 2018 07:55:46 +0000. Up 37.98 seconds. Cloud-init v. 17.1 running 'init' at Wed, 03 Oct 2018 08:09:09 +0000. Up 841.08 seconds. ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++ ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: | eth0: | True | 10.0.0.5 | 255.255.255.0 | . | fa:16:3e:b1:e6:48 | ci-info: | eth0: | True | . | . | d | fa:16:3e:b1:e6:48 | ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . | ci-info: | lo: | True | . | . | d | . | ci-info: +--------+------+-----------+---------------+-------+-------------------+ ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++ ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG | ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U | ci-info: | 2 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH | ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ Cloud-init v. 17.1 running 'modules:config' at Wed, 03 Oct 2018 08:09:25 +0000. Up 857.28 seconds. + CA_FILE=/etc/pki/ca-trust/source/anchors/openstack-ca.pem + '[' -n '' ']'

']'

/var/lib/cloud/instance/scripts/part-007: line 57: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 70: /etc/etcd/etcd.conf: No such file or directory /var/lib/cloud/instance/scripts/part-007: line 86: /etc/etcd/etcd.conf: No such file or directory Cloud-init v. 17.1 running 'modules:final' at Wed, 03 Oct 2018 08:09:27 +0000. Up 859.32 seconds. 2018-10-03 08:09:40,001 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-007 [1] Removed /etc/systemd/system/multi-user.target.wants/docker-storage-setup.service. New size given (1280 extents) not larger than existing size (4863 extents) ERROR: There is not enough free space in volume group atomicos to create data volume of size MIN_DATA_SIZE=2G. 2018-10-03 08:09:40,699 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-009 [1] configuring kubernetes (master)

(master)

sed: can't read /etc/kubernetes/config: No such file or directory sed: can't read /etc/kubernetes/apiserver: No such file or directory sed: can't read /etc/kubernetes/controller-manager: No such file or directory sed: can't read /etc/kubernetes/scheduler: No such file or directory % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1439 100 1439 0 0 162 0 0:00:08 0:00:08 --:--:-- 393 Generating RSA private key, 4096 bit long modulus .............................................................................++++ ......++++ e is 65537 (0x010001) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5749 100 3822 100 1927 343 173 0:00:11 0:00:11 --:--:-- 977 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1439 100 1439 0 0 143 0 0:00:10 0:00:10 --:--:-- 376 Generating RSA private key, 4096 bit long modulus ............++++ ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++++ e is 65537 (0x010001) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6379 100 4242 100 2137 500 251 0:00:08 0:00:08 --:--:-- 1294 + _prefix=docker.io/openstackmagnum/ + atomic install --storage ostree --system --system-package no --set REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt --name heat-container-agent docker.io/openstackmagnum/heat-container-agent:rawhide

docker.io/openstackmagnum/heat-container-agent:rawhide
  • + systemctl start heat-container-agent Failed to start heat-container-agent.service: Unit heat-container-agent.service not found. 2018-10-03 08:10:59,526 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-013 [5] starting services activating service etcd Failed to enable unit: Unit file etcd.service does not exist. Failed to start etcd.service: Unit etcd.service not found. activating service docker activating service kube-apiserver Failed to enable unit: Unit file kube-apiserver.service does not exist. Failed to start kube-apiserver.service: Unit kube-apiserver.service not found. activating service kube-controller-manager Failed to enable unit: Unit file kube-controller-manager.service does not exist. Failed to start kube-controller-manager.service: Unit kube-controller-manager.service not found. activating service kube-scheduler Failed to enable unit: Unit file kube-scheduler.service does not exist. Failed to start kube-scheduler.service: Unit kube-scheduler.service not found. creating /usr/local/bin/flannel-config Created symlink /etc/systemd/system/multi-user.target.wants/flannel-config.service → /etc/systemd/system/flannel-config.service. Failed to start flannel-config.service: Unit etcd.service not found. activating service flanneld Failed to enable unit: Unit file flanneld.service does not exist. Failed to start flanneld.service: Unit flanneld.service not found. 2018-10-03 08:11:00,681 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-016 [5]
  • [5] + . /etc/sysconfig/heat-params ++ PROMETHEUS_MONITORING=False ++ KUBE_API_PUBLIC_ADDRESS=192.168.41.143 ++ KUBE_API_PRIVATE_ADDRESS=10.0.0.5 ++ KUBE_API_PORT=6443 ++ KUBE_NODE_PUBLIC_IP=192.168.41.143 ++ KUBE_NODE_IP=10.0.0.5 ++ KUBE_ALLOW_PRIV=true ++ ENABLE_CINDER= ++ ETCD_VOLUME=d5b74d83-e8a2-4b96-9dd9-b5bf1b360176 ++ ETCD_VOLUME_SIZE=0 ++ DOCKER_VOLUME=7d496030-2845-4d7e-a8a0-01bb354bb047 ++ DOCKER_VOLUME_SIZE=0 ++ DOCKER_STORAGE_DRIVER=devicemapper ++ NETWORK_DRIVER=flannel ++ FLANNEL_NETWORK_CIDR=10.100.0.0/16 ++ FLANNEL_NETWORK_SUBNETLEN=24 ++ FLANNEL_BACKEND=udp ++ PODS_NETWORK_CIDR=10.100.0.0/16 ++ PORTAL_NETWORK_CIDR=10.254.0.0/16 ++ ADMISSION_CONTROL_LIST=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota ++ ETCD_DISCOVERY_URL=https://discovery.etcd.io/7dae7a905b725a5273cb38bd1cf735df ++ USERNAME=admin ++ PASSWORD=ChangeMe ++ CLUSTER_SUBNET=0ae61531-fcc3-46bf-b18a-3df682182db3 ++ TLS_DISABLED=False ++ KUBE_DASHBOARD_ENABLED=True ++ INFLUX_GRAFANA_DASHBOARD_ENABLED=False ++ VERIFY_CA=True ++ CLUSTER_UUID=55276ad7-2c18-4b65-8c13-9849f21ff2d2 ++ MAGNUM_URL=http://192.168.41.146:9511/v1 ++ VOLUME_DRIVER= ++ HTTP_PROXY=http://xx.xx.xx.xx:80 ++ HTTPS_PROXY=http://xx.xx.xx.xx:80 ++ NO_PROXY= ++ WAIT_CURL='curl -i -X POST -H '\''X-Auth-Token: gAAAAABbtHOCrXV6lEkrpJ_JK4HW1ieMSOqufovUCzKhDroRipKJBeia7VpRqJgj7XSaok2EPy-fx5HBt3rra4XQ9vEugANcNfgDU3N8FQ_jKZoX-mHb0DUY8Cs2crkuVAhTPPxFRQwcXY33hNmz7-nZJWrBQ4hBkrWK09k6yijpzL-VhK6ZwKE'\'' -H '\''Content-Type: application/json'\'' -H '\''Accept: application/json'\'' http://192.168.41.146:8004/v1/2452e0a099c74afc94ebcde35be10a1f/stacks/kubernetes-cluster-ofga4pkibuog-kube_masters-7d5h4nj3he2s-0-qpqfcpg34kib/73402937-3adb-40a9-be8c-09f216c8d525/resources/master_wait_handle/signal' ++ KUBE_TAG=v1.9.3 ++ ETCD_TAG=v3.2.7 ++ FLANNEL_TAG=v0.9.0 ++ KUBE_VERSION=v1.9.3 ++ KUBE_DASHBOARD_VERSION=v1.8.3 ++ TRUSTEE_USER_ID=26886307c1444f4fa4c32bc1985d6619 ++ TRUSTEE_PASSWORD=p6jodu8tchwNGoHyn5 ++ TRUST_ID= ++ AUTH_URL=http://192.168.41.146:5000/v3 ++ INSECURE_REGISTRY_URL= ++ CONTAINER_INFRA_PREFIX= ++ SYSTEM_PODS_INITIAL_DELAY=30 ++ SYSTEM_PODS_TIMEOUT=5 ++ ETCD_LB_VIP= ++ DNS_SERVICE_IP=10.254.0.10 ++ DNS_CLUSTER_DOMAIN=cluster.local ++ CERT_MANAGER_API=False ++ CA_KEY= ++ CALICO_TAG=v2.6.7 ++ CALICO_CNI_TAG=v1.11.2 ++ CALICO_KUBE_CONTROLLERS_TAG=v1.0.3 ++ CALICO_IPV4POOL=192.168.0.0/16 ++ INGRESS_CONTROLLER= ++ INGRESS_CONTROLLER_ROLE=ingress ++ KUBELET_OPTIONS= ++ KUBECONTROLLER_OPTIONS= ++ KUBEAPI_OPTIONS= ++ KUBEPROXY_OPTIONS= ++ KUBESCHEDULER_OPTIONS=
  • KUBESCHEDULER_OPTIONS= + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API...
  • API... + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
  • 5 + curl --silent http://127.0.0.1:8080/version
  • http://127.0.0.1:8080/version + sleep 5
5`enter code here`