Unable to scale Kubernetes cluster. Invalid discovery URL

asked 2019-07-05 10:29:35 -0500

dohlsson gravatar image

updated 2019-07-08 08:30:58 -0500

I am trying to scale up a Kubernetes cluster created with Openstack Magnum.

The creation of the cluster went fine. And the cluster is working as expected.

However. When i try to scale up by changing node_count the status becomes UPDATE_FAILED. The status_reason is 'Invalid discovery URL'.

This seems to be an issue caused by discovery.etcd.io changing its behaviour and only keeping a token alive for 12 hours.

If I create a new cluster I can scale it as much as I want during this 12 hour period while the token still exists.

How can i work around this?

$ openstack cluster show kubernetes_cluster

+---------------------+-------------------------------------------------------------------------------------+
| Field               | Value                                                                               |
+---------------------+-------------------------------------------------------------------------------------+
| status              | UPDATE_FAILED                                                                       |
| cluster_template_id | 38c6687b-4e8c-4fd2-9153-dc2dcb8c492f                                                |
| node_addresses      | ['188.95.231.55', '188.95.231.94']                                                  |
| uuid                | 51abfdb0-ede9-449e-a3bd-68df37f64249                                                |
| stack_id            | 0f3b894e-dc45-425e-ad4f-495a8a5addef                                                |
| status_reason       | Invalid discovery URL 'https://discovery.etcd.io/54d9ff763ebee369bcf87d36b45ad600'. |
| created_at          | 2019-06-26T12:54:43+00:00                                                           |
| updated_at          | 2019-07-01T08:26:55+00:00                                                           |
| coe_version         | v1.11.1                                                                             |
| labels              | {'kube_tag': 'v1.14.0'}                                                             |
| faults              | {}                                                                                  |
| keypair             | mykeys                                                                              |
| api_address         | https://188.95.231.74:6443                                                          |
| master_addresses    | ['188.95.231.74']                                                                   |
| create_timeout      | 60                                                                                  |
| node_count          | 3                                                                                   |
| discovery_url       | https://discovery.etcd.io/54d9ff763ebee369bcf87d36b45ad600                          |
| master_count        | 1                                                                                   |
| container_version   | 1.12.6                                                                              |
| name                | kubernetes_cluster                                                                  |
| master_flavor_id    | 2C-4GB-50GB                                                                         |
| flavor_id           | 2C-4GB-50GB                                                                         |
+---------------------+-------------------------------------------------------------------------------------+

and

$ openstack coe cluster template show kubernetes_cluster_template
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| insecure_registry     | -                                    |
| labels                | {'kube_tag': 'v1.14.0'}              |
| updated_at            | -                                    |
| floating_ip_enabled   | True                                 |
| fixed_subnet          | -                                    |
| master_flavor_id      | 2C-4GB-50GB                          |
| uuid                  | 38c6687b-4e8c-4fd2-9153-dc2dcb8c492f |
| no_proxy              | -                                    |
| https_proxy           | -                                    |
| tls_disabled          | False                                |
| keypair_id            | -                                    |
| public                | False                                |
| http_proxy            | -                                    |
| docker_volume_size    | 5                                    |
| server_type           | vm                                   |
| external_network_id   | ext-net                              |
| cluster_distro        | fedora-atomic                        |
| image_id              | Fedora Atomic 27                     |
| volume_driver         | cinder                               |
| registry_enabled      | False                                |
| docker_storage_driver | overlay2                             |
| apiserver_port        | -                                    |
| name                  | kubernetes_cluster_template          |
| created_at            | 2019-06-25T12:36:58+00:00            |
| network_driver        | flannel                              |
| fixed_network         | -                                    |
| coe                   | kubernetes                           |
| flavor_id             | 2C-4GB-50GB                          |
| master_lb_enabled     | False                                |
| dns_nameserver        | 8.8.8.8                              |
+-----------------------+--------------------------------------+

Thank you

Edit: Additional Information

The hosting company I am using are running Openstack Rocky. This is a service I am using so I do not have access to the Openstack setup directly, only through the API's.

I do not know what Barbican is. It is not something I think I have interacted with.

The OS image for Kubernetes is Fedora Atomic 27.

$ openstack versions show
+-------------+-------------------------------------+---------+------------+-------------------------------------------------+------------------+------------------+
| Region Name | Service Type                        | Version | Status     | Endpoint                                        | Min Microversion | Max Microversion |
+-------------+-------------------------------------+---------+------------+-------------------------------------------------+------------------+------------------+
[...]
| Kna1        | placement                           | None    | CURRENT    | https://kna1.citycloud.com:8780/                | None             | None             |
| Kna1        | block-storage                       | 2.0     | DEPRECATED | https://kna1.citycloud.com:8776/v2/             | None             | None             |
| Kna1        | block-storage                       | 3.0     | CURRENT    | https://kna1.citycloud.com:8776/v3/             | 3.0              | 3.55             |
| Kna1        | orchestration                       | 1.0     | CURRENT    | https://kna1.citycloud.com:8004/v1/             | None             | None             |
| Kna1        | object-store                        | 1.0     | CURRENT    | https://swift-kna1.citycloud.com:8080/swift/v1/ | None             | None             |
| Kna1        | meter                               | None    | CURRENT    | http://10.101.0.2:8777/                         | None             | None             |
| Kna1        | identity                            | 3.10    | CURRENT    | https://kna1.citycloud.com:5000/v3/             | None             | None             |
| Kna1        | cloudformation                      | 1.0     | CURRENT    | https://kna1.citycloud.com:8000/v1/             | None             | None             |
| Kna1        | network                             | 2.0     | CURRENT    | https://kna1.citycloud.com:9696/v2.0/           | None             | None             |
| Kna1        | key-manager                         | 1.0     | CURRENT    | https://kna1.citycloud.com:9311/v1/             | None             | None             |
| Kna1        | ec2                                 | None    | CURRENT    | https://kna1.citycloud.com:8773/services/Cloud/ | None             | None             |
| Kna1        | container-infrastructure-management | 1.0     | CURRENT    | https://kna1.citycloud.com:9511/v1/             | 1.1              | 1.6 ...
(more)
edit retag flag offensive close merge delete

Comments

what version of openstack did you install?

trove gravatar imagetrove ( 2019-07-05 15:51:52 -0500 )edit

what is your kubernetes os?fedora?

trove gravatar imagetrove ( 2019-07-05 15:52:10 -0500 )edit

what is your openstack OS?

trove gravatar imagetrove ( 2019-07-05 15:53:28 -0500 )edit

do you install Barbican and use it for magnum?

trove gravatar imagetrove ( 2019-07-05 15:54:05 -0500 )edit

I have updated the question with answers to your questions.

dohlsson gravatar imagedohlsson ( 2019-07-08 08:31:57 -0500 )edit