Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

neutron heartbeat in one hour late

Hello!

All the network agents in "XXX" alive state.

root@mantis(controller)~ # openstack network agent list
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                  | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| 3248e20f-cbed-4bd5-85f2-4248a54ebd38 | DHCP agent         | gamora.remiroda.local | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 4352592b-0d66-4a68-ba2c-9750f4239e11 | DHCP agent         | groot.remiroda.local  | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 673061d1-f8e3-4441-aa40-afb9b3c0aaf0 | L3 agent           | gamora.remiroda.local | nova              | XXX   | UP    | neutron-l3-agent          |
| 83fe4bfd-b68e-489f-9de1-d24e06b034ea | L3 agent           | groot.remiroda.local  | nova              | XXX   | UP    | neutron-l3-agent          |
| 88ddabaf-74ac-435b-aacd-0a9e0c5144a1 | Linux bridge agent | gamora.remiroda.local | None              | XXX   | UP    | neutron-linuxbridge-agent |
| e20f5127-0dc7-4e7f-9a53-bfc40e81a920 | Linux bridge agent | groot.remiroda.local  | None              | XXX   | UP    | neutron-linuxbridge-agent |
| ffa33e26-c87c-49d9-af56-4aab3033f241 | Metadata agent     | mantis.remiroda.local | None              | XXX   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+

But all the agents running without error on nodes.

in server.log I found that the heartbeat in one hour late.

2019-01-17 17:06:03.252 3705 WARNING neutron.db.agents_db [req-283c2844-74d5-4f6e-bce1-75e9fb8ef0db - - - - -] Agent healthcheck: found 7 dead agents out of 7:
                Type       Last heartbeat host
          DHCP agent  2019-01-17 16:05:57 gamora.remiroda.local
          DHCP agent  2019-01-17 16:05:55 groot.remiroda.local
            L3 agent  2019-01-17 16:05:40 gamora.remiroda.local
            L3 agent  2019-01-17 16:05:56 groot.remiroda.local
  Linux bridge agent  2019-01-17 16:05:40 gamora.remiroda.local
  Linux bridge agent  2019-01-17 16:05:56 groot.remiroda.local
      Metadata agent  2019-01-17 16:05:57 mantis.remiroda.local

root@gamora(compute1)/var/log/neutron # systemctl status neutron-dhcp-agent.service
● neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-01-17 16:48:25 CET; 26min ago
...
root@gamora(compute1)/var/log/neutron # ip netns
qdhcp-9565e868-a21e-4ea9-b347-e02685056e91 (id: 3)
qdhcp-165c112f-7e3c-4b52-a2e2-dbd013defd10 (id: 1)
qdhcp-3896460d-3a82-4b16-a75a-1f1693c410e8 (id: 0)
qrouter-ac25de8f-f4a9-4911-b726-33d9544a33aa (id: 5)
qrouter-263c80ea-b5fa-4a93-a922-87b49337999a (id: 4)

All the nodes the ntp is configured properly and all in same timezone.

what is the problem? Thank you the help.

Johnny

neutron heartbeat in one hour late

Hello!

All the network agents in "XXX" alive state.

root@mantis(controller)~ # openstack network agent list
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                  | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| 3248e20f-cbed-4bd5-85f2-4248a54ebd38 | DHCP agent         | gamora.remiroda.local | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 4352592b-0d66-4a68-ba2c-9750f4239e11 | DHCP agent         | groot.remiroda.local  | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 673061d1-f8e3-4441-aa40-afb9b3c0aaf0 | L3 agent           | gamora.remiroda.local | nova              | XXX   | UP    | neutron-l3-agent          |
| 83fe4bfd-b68e-489f-9de1-d24e06b034ea | L3 agent           | groot.remiroda.local  | nova              | XXX   | UP    | neutron-l3-agent          |
| 88ddabaf-74ac-435b-aacd-0a9e0c5144a1 | Linux bridge agent | gamora.remiroda.local | None              | XXX   | UP    | neutron-linuxbridge-agent |
| e20f5127-0dc7-4e7f-9a53-bfc40e81a920 | Linux bridge agent | groot.remiroda.local  | None              | XXX   | UP    | neutron-linuxbridge-agent |
| ffa33e26-c87c-49d9-af56-4aab3033f241 | Metadata agent     | mantis.remiroda.local | None              | XXX   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+

But all the agents running without error on nodes.

in server.log I found that the heartbeat in one hour late.

2019-01-17 17:06:03.252 3705 WARNING neutron.db.agents_db [req-283c2844-74d5-4f6e-bce1-75e9fb8ef0db - - - - -] Agent healthcheck: found 7 dead agents out of 7:
                Type       Last heartbeat host
          DHCP agent  2019-01-17 16:05:57 gamora.remiroda.local
          DHCP agent  2019-01-17 16:05:55 groot.remiroda.local
            L3 agent  2019-01-17 16:05:40 gamora.remiroda.local
            L3 agent  2019-01-17 16:05:56 groot.remiroda.local
  Linux bridge agent  2019-01-17 16:05:40 gamora.remiroda.local
  Linux bridge agent  2019-01-17 16:05:56 groot.remiroda.local
      Metadata agent  2019-01-17 16:05:57 mantis.remiroda.local

root@gamora(compute1)/var/log/neutron # systemctl status neutron-dhcp-agent.service
● neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-01-17 16:48:25 CET; 26min ago
...
root@gamora(compute1)/var/log/neutron # ip netns
qdhcp-9565e868-a21e-4ea9-b347-e02685056e91 (id: 3)
qdhcp-165c112f-7e3c-4b52-a2e2-dbd013defd10 (id: 1)
qdhcp-3896460d-3a82-4b16-a75a-1f1693c410e8 (id: 0)
qrouter-ac25de8f-f4a9-4911-b726-33d9544a33aa (id: 5)
qrouter-263c80ea-b5fa-4a93-a922-87b49337999a (id: 4)

All the nodes the ntp is configured properly and all in same timezone.timezone (Europe/Budapest).

what is the problem? Thank you the help.

Johnny

neutron heartbeat in one hour late

Hello!

All the network agents in "XXX" alive state.

root@mantis(controller)~ # openstack network agent list
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                  | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+
| 3248e20f-cbed-4bd5-85f2-4248a54ebd38 | DHCP agent         | gamora.remiroda.local | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 4352592b-0d66-4a68-ba2c-9750f4239e11 | DHCP agent         | groot.remiroda.local  | nova              | XXX   | UP    | neutron-dhcp-agent        |
| 673061d1-f8e3-4441-aa40-afb9b3c0aaf0 | L3 agent           | gamora.remiroda.local | nova              | XXX   | UP    | neutron-l3-agent          |
| 83fe4bfd-b68e-489f-9de1-d24e06b034ea | L3 agent           | groot.remiroda.local  | nova              | XXX   | UP    | neutron-l3-agent          |
| 88ddabaf-74ac-435b-aacd-0a9e0c5144a1 | Linux bridge agent | gamora.remiroda.local | None              | XXX   | UP    | neutron-linuxbridge-agent |
| e20f5127-0dc7-4e7f-9a53-bfc40e81a920 | Linux bridge agent | groot.remiroda.local  | None              | XXX   | UP    | neutron-linuxbridge-agent |
| ffa33e26-c87c-49d9-af56-4aab3033f241 | Metadata agent     | mantis.remiroda.local | None              | XXX   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+-----------------------+-------------------+-------+-------+---------------------------+

But all the agents running without error on nodes.

in server.log I found that the heartbeat in one hour late.

2019-01-17 17:06:03.252 3705 WARNING neutron.db.agents_db [req-283c2844-74d5-4f6e-bce1-75e9fb8ef0db - - - - -] Agent healthcheck: found 7 dead agents out of 7:
                Type       Last heartbeat host
          DHCP agent  2019-01-17 16:05:57 gamora.remiroda.local
          DHCP agent  2019-01-17 16:05:55 groot.remiroda.local
            L3 agent  2019-01-17 16:05:40 gamora.remiroda.local
            L3 agent  2019-01-17 16:05:56 groot.remiroda.local
  Linux bridge agent  2019-01-17 16:05:40 gamora.remiroda.local
  Linux bridge agent  2019-01-17 16:05:56 groot.remiroda.local
      Metadata agent  2019-01-17 16:05:57 mantis.remiroda.local

root@gamora(compute1)/var/log/neutron # systemctl status neutron-dhcp-agent.service
● neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-01-17 16:48:25 CET; 26min ago
...
root@gamora(compute1)/var/log/neutron # ip netns
qdhcp-9565e868-a21e-4ea9-b347-e02685056e91 (id: 3)
qdhcp-165c112f-7e3c-4b52-a2e2-dbd013defd10 (id: 1)
qdhcp-3896460d-3a82-4b16-a75a-1f1693c410e8 (id: 0)
qrouter-ac25de8f-f4a9-4911-b726-33d9544a33aa (id: 5)
qrouter-263c80ea-b5fa-4a93-a922-87b49337999a (id: 4)

root@mantis(controller)~ # timedatectl
      Local time: Fri 2019-01-18 09:39:18 CET
  Universal time: Fri 2019-01-18 08:39:18 UTC
        RTC time: Fri 2019-01-18 08:39:18
       Time zone: Europe/Budapest (CET, +0100)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: no
 Last DST change: DST ended at
                  Sun 2018-10-28 02:59:59 CEST
                  Sun 2018-10-28 02:00:00 CET
 Next DST change: DST begins (the clock jumps one hour forward) at
                  Sun 2019-03-31 01:59:59 CET
                  Sun 2019-03-31 03:00:00 CEST

All the nodes the ntp is configured properly and all in same timezone (Europe/Budapest).

what is the problem? Thank you the help.

Johnny