neutron heartbeat in one hour late

asked 2019-01-17 10:16:08 -0500

Johnny gravatar image

updated 2019-01-18 02:40:46 -0500

Hello!

All the network agents in "XXX" alive state.

root@mantis(controller)~ # openstack network agent list
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                  | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| 3248e20f-cbed-4bd5-85f2-4248a54ebd38 | DHCP agent         | gamora.remiroda.local | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 4352592b-0d66-4a68-ba2c-9750f4239e11 | DHCP agent         | groot.remiroda.local  | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 673061d1-f8e3-4441-aa40-afb9b3c0aaf0 | L3 agent           | gamora.remiroda.local | nova              | XXX   | UP    | neutron-l3-agent          |
| 83fe4bfd-b68e-489f-9de1-d24e06b034ea | L3 agent           | groot.remiroda.local  | nova              | XXX   | UP    | neutron-l3-agent          |
| 88ddabaf-74ac-435b-aacd-0a9e0c5144a1 | Linux bridge agent | gamora.remiroda.local | None              | XXX   | UP    | neutron-linuxbridge-agent |
| e20f5127-0dc7-4e7f-9a53-bfc40e81a920 | Linux bridge agent | groot.remiroda.local  | None              | XXX   | UP    | neutron-linuxbridge-agent |
| ffa33e26-c87c-49d9-af56-4aab3033f241 | Metadata agent     | mantis.remiroda.local | None              | XXX   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+

But all the agents running without error on nodes.

in server.log I found that the heartbeat in one hour late.

2019-01-17 17:06:03.252 3705 WARNING neutron.db.agents_db [req-283c2844-74d5-4f6e-bce1-75e9fb8ef0db - - - - -] Agent healthcheck: found 7 dead agents out of 7:
                Type       Last heartbeat host
          DHCP agent  2019-01-17 16:05:57 gamora.remiroda.local
          DHCP agent  2019-01-17 16:05:55 groot.remiroda.local
            L3 agent  2019-01-17 16:05:40 gamora.remiroda.local
            L3 agent  2019-01-17 16:05:56 groot.remiroda.local
  Linux bridge agent  2019-01-17 16:05:40 gamora.remiroda.local
  Linux bridge agent  2019-01-17 16:05:56 groot.remiroda.local
      Metadata agent  2019-01-17 16:05:57 mantis.remiroda.local

root@gamora(compute1)/var/log/neutron # systemctl status neutron-dhcp-agent.service
● neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-01-17 16:48:25 CET; 26min ago
...
root@gamora(compute1)/var/log/neutron # ip netns
qdhcp-9565e868-a21e-4ea9-b347-e02685056e91 (id: 3)
qdhcp-165c112f-7e3c-4b52-a2e2-dbd013defd10 (id: 1)
qdhcp-3896460d-3a82-4b16-a75a-1f1693c410e8 (id: 0)
qrouter-ac25de8f-f4a9-4911-b726-33d9544a33aa (id: 5)
qrouter-263c80ea-b5fa-4a93-a922-87b49337999a (id: 4)

root@mantis(controller)~ # timedatectl
      Local time: Fri 2019-01-18 09:39:18 CET
  Universal time: Fri 2019-01-18 08:39:18 UTC
        RTC time: Fri 2019-01-18 08:39:18
       Time zone: Europe/Budapest (CET, +0100)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: no
 Last DST change: DST ended at
                  Sun 2018-10-28 02:59:59 CEST
                  Sun 2018-10-28 02:00:00 CET
 Next DST change: DST begins (the clock jumps one hour forward) at
                  Sun 2019-03-31 01:59:59 CET
                  Sun 2019-03-31 03:00:00 CEST

All the nodes the ntp is configured properly and all in same timezone (Europe/Budapest).

what is the problem? Thank you the help.

Johnny

edit retag flag offensive close merge delete