Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Ocata Openstack controller node on Ubuntu 16.04 Crashed

Hello, Our controller node on Ubuntu 16.04.4 LTS 4.4.0-130-generic #156-Ubuntu server crashed twice this week, Can someone help us to find the root cause of this crash and what fix needs to approach.

Below are few entries from syslog file,

Jul 5 00:00:01 U16-Server CRON[27075]: (root) CMD (/usr/bin/cinder-volume-usage-audit) Jul 5 00:00:20 U16-Server CRON[27074]: (CRON) info (No MTA installed, discarding output) Jul 5 00:03:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit) Jul 5 00:08:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit) Jul 5 00:13:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit) Jul 5 00:17:01 U16-Server CRON[3269]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jul 5 00:18:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit) Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Cloud-init target. Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Timers. Jul 5 00:22:57 U16-Server systemd[1]: Stopping Authenticate and Authorize Users to Run Privileged Tasks... Jul 5 00:22:57 U16-Server systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Daily Cleanup of Temporary Directories. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Daily apt upgrade and clean activities. Jul 5 00:22:57 U16-Server systemd[1]: Stopped QEMU KVM preparation - module, ksm, hugepages. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Stop ureadahead data collection 45s after completed startup. Jul 5 00:22:57 U16-Server systemd[1]: Stopping ACPI event daemon... Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Graphical Interface. Jul 5 00:22:57 U16-Server systemd[1]: Stopping Accounts Service... Jul 5 00:22:57 U16-Server systemd[1]: Stopped Execute cloud user/final scripts. Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Multi-User System. Jul 5 00:22:57 U16-Server systemd[1]: Stopping memcached daemon... Jul 5 00:22:57 U16-Server systemd-memcached-wrapper[3016]: Signal handled: Terminated. Jul 5 00:22:57 U16-Server systemd[1]: Stopping Unattended Upgrades Shutdown... Jul 5 00:22:57 U16-Server systemd[1]: Stopped Wait until snapd is fully seeded. Jul 5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Backup... Jul 5 00:22:57 U16-Server systemd[1]: Stopping Login Service... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: Apache2 web server... Jul 5 00:22:57 U16-Server systemd[1]: Stopping juju agent for machine-0... Jul 5 00:22:57 U16-Server systemd[1]: Stopping Regular background program processing daemon... Jul 5 00:22:57 U16-Server systemd[1]: Stopping Suspend/Resume Running libvirt Guests... Jul 5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Volume... Jul 5 00:22:57 U16-Server systemd[1]: Stopping Snappy daemon... Jul 5 00:22:57 U16-Server snapd[3018]: 2018/07/05 00:22:57.937117 main.go:79: Exiting on terminated signal. Jul 5 00:22:57 U16-Server systemd[1]: Stopping juju unit agent for nrpe/2... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"... Jul 5 00:22:57 U16-Server systemd[1]: Stopping Deferred execution scheduler... Jul 5 00:22:57 U16-Server systemd[1]: Stopping juju unit agent for cinder/0... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: Record successful boot for GRUB... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: Start/Stop the Nagios remote plugin execution daemon... Jul 5 00:22:57 U16-Server systemd[1]: Stopping HAProxy Load Balancer... Jul 5 00:22:57 U16-Server systemd[1]: Stopped Apply the settings specified in cloud-config. Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Cloud-config availability. Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: MD monitoring daemon... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LXD - container startup/shutdown... Jul 5 00:22:57 U16-Server systemd[1]: Stopped Daily apt download activities. Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: automatic crash report generation... Jul 5 00:22:57 U16-Server systemd[1]: Stopping OpenBSD Secure Shell server... Jul 5 00:22:57 U16-Server systemd[1]: Stopped target Login Prompts. Jul 5 00:22:57 U16-Server systemd[1]: Stopping Getty on tty1... Jul 5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Scheduler... Jul 5 00:22:57 U16-Server systemd[1]: Stopping (i)SCSI target daemon... Jul 5 00:22:57 U16-Server systemd[1]: Stopping LSB: daemon to balance interrupts for SMP systems... Jul 5 00:22:57 U16-Server systemd[1]: Stopped Regular background program processing daemon. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Accounts Service. Jul 5 00:22:57 U16-Server systemd[1]: Stopped OpenBSD Secure Shell server. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Deferred execution scheduler. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Login Service. Jul 5 00:22:57 U16-Server systemd[1]: Stopped memcached daemon. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Snappy daemon. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Authenticate and Authorize Users to Run Privileged Tasks. Jul 5 00:22:57 U16-Server systemd[1]: Stopped juju agent for machine-0. Jul 5 00:22:57 U16-Server systemd[1]: Stopped juju unit agent for nrpe/2. Jul 5 00:22:57 U16-Server systemd[1]: Stopped Getty on tty1. Jul 5 00:22:57 U16-Server systemd[1]: Stopped juju unit agent for cinder/0. Jul 5 00:22:57 U16-Server systemd[1]: Stopped HAProxy Load Balancer. Jul 5 00:22:58 U16-Server apache2[30845]: * Stopping Apache httpd web server apache2 Jul 5 00:22:58 U16-Server systemd[1]: Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand". Jul 5 00:22:58 U16-Server nagios-nrpe-server[30864]: * Stopping nagios-nrpe nagios-nrpe Jul 5 00:22:58 U16-Server nrpe[30860]: Caught SIGTERM - shutting down... Jul 5 00:22:58 U16-Server nrpe[30860]: Daemon shutdown Jul 5 00:22:58 U16-Server nagios-nrpe-server[30864]: ...done. Jul 5 00:22:58 U16-Server systemd[1]: Stopped ACPI event daemon. Jul 5 00:22:58 U16-Server systemd[1]: Stopped LSB: Start/Stop the Nagios remote plugin execution daemon. Jul 5 00:22:58 U16-Server mdadm[30874]: * Stopping MD monitoring service mdadm --monitor Jul 5 00:22:58 U16-Server mdadm[30874]: ...done. Jul 5 00:22:58 U16-Server systemd[1]: Stopped LSB: MD monitoring daemon. Jul 5 00:22:58 U16-Server systemd[1]: Stopped LSB: Record successful boot for GRUB. Jul 5 00:22:58 U16-Server systemd[1]: Stopped Unattended Upgrades Shutdown. Jul 5 00:22:58 U16-Server apport[30881]: * Stopping automatic crash report generation: apport Jul 5 00:22:58 U16-Server apport[30881]: ...done. Jul 5 00:22:58 U16-Server systemd[1]: Stopped LSB: automatic crash report generation. Jul 4 06:25:02 U16-Server rsyslogd: message repeated 5 times: [ [origin software="rsyslogd" swVersion="8.16.0" x-pid="2987" x-info= "http://www.rsyslog.com"] rsyslogd was HUPed] Jul 5 00:22:58 U16-Server rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="2987" x-info="http://www.rsyslog.com"] ex iting on signal 15. Jul 5 04:02:40 U16-Server rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="3014" x-info="http://www.rsyslog.com"] st art Jul 5 04:02:40 U16-Server rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ] Jul 5 04:02:40 U16-Server rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.16.0 try http://www.rsyslog.com/e/2307 ]

Thanks

Ocata Openstack controller node on Ubuntu 16.04 Crashed

Hello, Our controller node on Ubuntu 16.04.4 LTS 4.4.0-130-generic #156-Ubuntu server crashed twice this week, Can someone help us to find the root cause of this crash and what fix needs to approach.

Below are few entries from syslog file,

Jul  5 00:00:01 U16-Server CRON[27075]: (root) CMD (/usr/bin/cinder-volume-usage-audit)
Jul  5 00:00:20 U16-Server CRON[27074]: (CRON) info (No MTA installed, discarding output)
Jul  5 00:03:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:08:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:13:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:17:01 U16-Server CRON[3269]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  5 00:18:01 U16-Server cron[2968]: (systemcinder-volume-usage-audit) (*system*cinder-volume-usage-audit) RELOAD (/etc/cron.d/cinder-volume-usage-audit)
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Cloud-init target.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Timers.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Authenticate and Authorize Users to Run Privileged Tasks...
Jul  5 00:22:57 U16-Server systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Daily apt upgrade and clean activities.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped QEMU KVM preparation - module, ksm, hugepages.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Stop ureadahead data collection 45s after completed startup.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping ACPI event daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Graphical Interface.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Accounts Service...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Execute cloud user/final scripts.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Multi-User System.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping memcached daemon...
Jul  5 00:22:57 U16-Server systemd-memcached-wrapper[3016]: Signal handled: Terminated.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Unattended Upgrades Shutdown...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Wait until snapd is fully seeded.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Backup...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Login Service...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: Apache2 web server...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping juju agent for machine-0...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Regular background program processing daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Suspend/Resume Running libvirt Guests...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Volume...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Snappy daemon...
Jul  5 00:22:57 U16-Server snapd[3018]: 2018/07/05 00:22:57.937117 main.go:79: Exiting on terminated signal.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping juju unit agent for nrpe/2...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Deferred execution scheduler...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping juju unit agent for cinder/0...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: Record successful boot for GRUB...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: Start/Stop the Nagios remote plugin execution daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping HAProxy Load Balancer...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Apply the settings specified in cloud-config.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Cloud-config availability.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: MD monitoring daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LXD - container startup/shutdown...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Daily apt download activities.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: automatic crash report generation...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenBSD Secure Shell server...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped target Login Prompts.
Jul  5 00:22:57 U16-Server systemd[1]: Stopping Getty on tty1...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping OpenStack Cinder Scheduler...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping (i)SCSI target daemon...
Jul  5 00:22:57 U16-Server systemd[1]: Stopping LSB: daemon to balance interrupts for SMP systems...
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Regular background program processing daemon.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Accounts Service.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped OpenBSD Secure Shell server.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Deferred execution scheduler.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Login Service.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped memcached daemon.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Snappy daemon.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Authenticate and Authorize Users to Run Privileged Tasks.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped juju agent for machine-0.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped juju unit agent for nrpe/2.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped Getty on tty1.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped juju unit agent for cinder/0.
Jul  5 00:22:57 U16-Server systemd[1]: Stopped HAProxy Load Balancer.
Jul  5 00:22:58 U16-Server apache2[30845]:  * Stopping Apache httpd web server apache2
Jul  5 00:22:58 U16-Server systemd[1]: Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
Jul  5 00:22:58 U16-Server nagios-nrpe-server[30864]:  * Stopping nagios-nrpe nagios-nrpe
Jul  5 00:22:58 U16-Server nrpe[30860]: Caught SIGTERM - shutting down...
Jul  5 00:22:58 U16-Server nrpe[30860]: Daemon shutdown
Jul  5 00:22:58 U16-Server nagios-nrpe-server[30864]:    ...done.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped ACPI event daemon.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped LSB: Start/Stop the Nagios remote plugin execution daemon.
Jul  5 00:22:58 U16-Server mdadm[30874]:  * Stopping MD monitoring service mdadm --monitor
Jul  5 00:22:58 U16-Server mdadm[30874]:    ...done.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped LSB: MD monitoring daemon.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped LSB: Record successful boot for GRUB.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped Unattended Upgrades Shutdown.
Jul  5 00:22:58 U16-Server apport[30881]:  * Stopping automatic crash report generation: apport
Jul  5 00:22:58 U16-Server apport[30881]:    ...done.
Jul  5 00:22:58 U16-Server systemd[1]: Stopped LSB: automatic crash report generation.
Jul  4 06:25:02 U16-Server rsyslogd: message repeated 5 times: [ [origin software="rsyslogd" swVersion="8.16.0" x-pid="2987" x-info=
"http://www.rsyslog.com"] rsyslogd was HUPed]
Jul  5 00:22:58 U16-Server rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="2987" x-info="http://www.rsyslog.com"] ex
iting on signal 15.
Jul  5 04:02:40 U16-Server rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="3014" x-info="http://www.rsyslog.com"] st
art
Jul  5 04:02:40 U16-Server rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it
via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
Jul  5 04:02:40 U16-Server rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.16.0 try
http://www.rsyslog.com/e/2307 ]

]

Thanks