kolla ansible ceilometer bootstrap fails/hangs with multinode

asked 2018-03-31 03:31:54 -0500

theque42 gravatar image

updated 2018-04-04 10:11:08 -0500

I am trying to install Openstack with kolla ansible, and the all-in-one worked fine on a single host.

But trying to use the the multinode-setup, the deploy hangs on :

"TASK [ceilometer : Running Ceilometer bootstrap container] "

I am running kolla-ansible 6.0.0 (from pip) and ansible-2.4.2.0-2.el7.noarch, and using updated CentOS 7 nodes.

Checking the logs for the container on my controller node, just says:

[root@ctrl ~()]# docker logs bootstrap_ceilometer
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf
INFO:__main__:Setting permission for /etc/ceilometer/ceilometer.conf
INFO:__main__:Copying /var/lib/kolla/config_files/event_definitions.yaml to /etc/ceilometer/event_definitions.yaml
INFO:__main__:Setting permission for /etc/ceilometer/event_definitions.yaml
INFO:__main__:Copying /var/lib/kolla/config_files/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml
INFO:__main__:Setting permission for /etc/ceilometer/event_pipeline.yaml
INFO:__main__:Copying /var/lib/kolla/config_files/pipeline.yaml to /etc/ceilometer/pipeline.yaml
INFO:__main__:Setting permission for /etc/ceilometer/pipeline.yaml
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/kolla/ceilometer
INFO:__main__:Setting permission for /var/log/kolla/ceilometer/ceilometer-agent-notification.log
Running command: 'ceilometer-agent-notification'

Configs:

multinode:(Just listing minor changes in the beginning)

[control]
ctrl
[network]
neutron1
neutron2
[inner-compute]
compute1
compute2
[external-compute]
[compute:children]
inner-compute
external-compute
[monitoring]
ctrl
[storage]
storage1
storage2
[deployment]
localhost       ansible_connection=local
[baremetal:children]
control
network
compute
storage
monitoring

globals.yml: (all non-commented lines)

kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "pike"
kolla_enable_sanity_checks: "yes"
kolla_internal_vip_address: "172.16.104.111"
kolla_external_vip_address: "10.10.104.111"
docker_registry: "10.10.10.64:5000"
network_interface: "eth1"
kolla_external_vip_interface: "eth0"
api_interface: "eth1"
storage_interface: "eth3"
tunnel_interface: "eth2"
neutron_external_interface: "eth4"
neutron_plugin_agent: "openvswitch"
keepalived_virtual_router_id: "54"
kolla_enable_tls_external: "no"
enable_aodh: "yes"
enable_ceilometer: "yes"
enable_cinder: "yes"
enable_cinder_backup: "no" ###TQ Need swift!
enable_cinder_backend_lvm: "yes"
enable_gnocchi: "yes"
enable_horizon_neutron_lbaas: "yes"
enable_neutron_lbaas: "yes"
enable_trove: "yes"
keystone_token_provider: 'fernet' ###TQ UUID
fernet_token_expiry: 86400
glance_backend_file: "yes"
nova_compute_virt_type: "kvm"
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

Attaching a shell to the controller node and the bootstrap-ceilometer container shows tons of output like:

2018-04-04 16:21:45.772 30 INFO ceilometer.declarative [-] Definitions: {'resources': [{'metrics': ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failure', 'identity.user.created', 'identity.user.deleted', 'identity.user.updated', 'identity.group.created', 'identity.group.deleted', 'identity.group.updated', 'identity.role.created', 'identity.role.deleted', 'identity.role.updated', 'identity.project.created', 'identity.project.deleted', 'identity.project.updated', 'identity.trust.created', 'identity.trust.deleted', 'identity.role_assignment.created', 'identity.role_assignment.deleted'], 'resource_type': 'identity'}, {'metrics': ['radosgw.objects', 'radosgw.objects.size', 'radosgw.objects.containers', 'radosgw.api.request', 'radosgw.containers.objects', 'radosgw.containers.objects.size'], 'resource_type': 'ceph_account'}, {'metrics': ['memory', 'memory.usage', 'memory.resident', 'memory.swap.in', 'memory.swap.out', 'memory.bandwidth.total', 'memory.bandwidth.local', 'vcpus', 'cpu', 'cpu.delta', 'cpu_util', 'cpu_l3_cache', 'disk.root.size', 'disk.ephemeral.size', 'disk.read.requests', 'disk.read.requests.rate', 'disk.write.requests', 'disk.write.requests.rate', 'disk ...
(more)
edit retag flag offensive close merge delete

Comments

1

please, check logs at /var/lib/docker/volumes/kolla_log/_data/ceilometer

Eduardo Gonzalez gravatar imageEduardo Gonzalez ( 2018-04-03 04:30:16 -0500 )edit

Thanks!

It seems there is a gnocci problem, but there are some complaints earlier on keystone, so...

"2018-04-02 23:19:38.548 30 WARNING ceilometer.dispatcher.gnocchi [-] gnocchi project not found in keystone,

theque42 gravatar imagetheque42 ( 2018-04-03 09:16:08 -0500 )edit

Btw Eduardo, do you know what I should trust...the documentation that says that "if I use kolla-ansible from pip I must use the latest release" of openstack.....and then says PIKE....

theque42 gravatar imagetheque42 ( 2018-04-04 09:10:19 -0500 )edit