Ocata Openstack controller node on Ubuntu 16.04 Crashed

asked 2018-07-05 00:55:00 -0500

TijoV gravatar image

updated 2018-07-05 04:05:40 -0500

Hello, Our controller node on Ubuntu 16.04.4 LTS 4.4.0-130-generic #156-Ubuntu server crashed twice this week, Can someone help us to find the root cause of this crash and what fix needs to approach.

Below are few entries from syslog file,

Jul  5 00:00:01 U16-Server CRON[27075]: (root) CMD (/usr/bin/cinder-volume-usage-audit)
Jul  5 00:00:20 U16-Server CRON[27074]: (CRON) info (No MTA installed, discarding output)
Jul  5 00:03:01 U16-Server cron[2968]: (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:08:01 U16-Server cron[2968]: (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:13:01 U16-Server cron[2968]: (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:17:01 U16-Server CRON[3269]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  5 00:18:01 U16-Server cron[2968]: (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Cloud-init target.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Timers.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Authenticate and Authorize Users to Run Privileged Tasks...
Jul  5 00:22:57 U16-Server systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Daily apt upgrade and clean activities.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped QEMU KVM preparation - module, ksm, hugepages.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Stop ureadahead data collection 45s after completed startup.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping ACPI event daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Graphical Interface.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Accounts Service...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Execute cloud user/final scripts.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Multi-User System.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping memcached daemon...
Jul  5 00:22:57 U16-Server systemd-memcached-wrapper[3016]: Signal handled: Terminated.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Unattended Upgrades Shutdown...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Wait until snapd is fully seeded.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Backup...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Login Service...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: Apache2 web server...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping juju agent for machine-0...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Regular background program processing daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Suspend/Resume Running libvirt Guests...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Volume...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Snappy daemon...
Jul  5 00:22:57 U16-Server snapd[3018]: 2018/07/05 00:22:57.937117 main.go:79: Exiting on terminated signal.
Jul ...
(more)
edit retag flag offensive close merge delete

Comments

It doesn't look like a crash, but rather an orderly shutdown of your server.

Since you use Ubuntu 16, you should have a systemd journal. Ensure that it's permanent (mkdir /var/log/journal). It may contain more information.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-05 04:07:30 -0500 )edit

Thanks Bernd I have created the Journel directory, Orderly shutdown but why, we have not initiated any shutdown. The system again got powered off in the last 30 mins. In the Kern.logs its showing to update BIOS version, so its updated now, and running on the latest BIOS version.

TijoV gravatar imageTijoV ( 2018-07-05 05:29:17 -0500 )edit

Check the journal. To see only the messages before the shutdown, journalctl --boot -1 if I am not wrong. journalctl also allows you to filter for certain times, using the --since and --until options.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-05 06:52:39 -0500 )edit

https://files.fm/u/s58hzvj9@Bernd Bausch, Kindly take a look at the logs attached on above link

TijoV gravatar imageTijoV ( 2018-07-05 09:08:30 -0500 )edit

The journal output stops before the shutdown. Perhaps you need to look at the full journal.

syslog:

Jul  5 12:24:35 U16-Server systemd[1]: Stopping User Manager for UID 1000...
Jul  5 12:24:35 U16-Server systemd[6358]: Reached target Shutdown

I can't tell what initiated the shutdown.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-06 20:22:22 -0500 )edit