compute services goes down while spawning instance

asked 2018-02-07 06:37:26 -0500

Rashmi gravatar image

I have Openstack liberty on my environment with 1 controller and 7 compute nodes. After reboot of all nodes, nova and neutron services on 2 compute nodes out of 7 are went down . Now , NOVA and NEUTRON running on computes are UP but whenever I launch an instance on these two specific computes the services goes down. On checking the /var/logs/nova/nova-compute.log and /var/logs/neutron/openvswitch-agent.log it shows AMQP server on controller:5672 is unreachable and when the instance goes in error state after spawning , the AMQP is connected and the services are UP again. Rabbitmqctl status shows: Status of node rabbit@controller1 ... [{pid,23071}, {running_applications,[{rabbit,"RabbitMQ","3.5.4"}, {os_mon,"CPO CXC 138 46","2.2.14"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {xmerl,"XML parser","1.3.5"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, {kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true]\n"}, {memory,[{total,480688504}, {connection_readers,3308304}, {connection_writers,675304}, {connection_channels,2871584}, {connection_other,6732472}, {queue_procs,5254048}, {queue_slave_procs,0}, {plugins,0}, {other_proc,13817552}, {mnesia,710232}, {mgmt_db,0}, {msg_index,200680}, {other_ets,1155176}, {binary,422900312}, {code,16627389}, {atom,602729}, {other_system,5832722}]}, {alarms,[]}, {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,10074528153}, {disk_free_limit,50000000}, {disk_free,286755672064}, {file_descriptors,[{total_limit,924}, {total_used,268}, {sockets_limit,829}, {sockets_used,266}]}, {processes,[{limit,1048576},{used,3460}]}, {run_queue,0}, {uptime,7725}]

PLEASE HELP on what should be done.
edit retag flag offensive close merge delete