Saturday, June 20, 2020

Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)' in OEM 13c

Agent 13.2 home was restored from a backup after failed OS patching maintenance. On the Enterprise Manager 13.2 console, the agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)', and targets monitored by the agent show 'Diagnose for Status Pending (Post Blackout)'.

Check agent status on agent host
[oracle@host01]$ emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/product/em/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/product/em/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0
Core JAR Location      : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/jlib
Agent Process ID       : 63177128
Parent Process ID      : 1573530
Agent URL              : https://host01.lab.dbaplus.ca:3872/emd/main/
Local Agent URL in NAT : https://host01.lab.dbaplus.ca:3872/emd/main/
Repository URL         : https://oms1.lab.dbaplus.ca:4903/empbs/upload
Started at             : 2020-06-20 11:06:45
Started by user        : oracle
Operating System       : AIX version 7.1 (ppc64)
Number of Targets      : 42
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : 2020-06-20 11:07:16
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 738
Size of XML files pending upload(MB)         : 1.95
Available disk space on upload filesystem    : 32.81%
Collection Status                            : Collections enabled
Heartbeat Status                             : Agent is blocked
Blocked Reason         : Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console.
Last attempted heartbeat to OMS              : 2020-06-20 13:31:20
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2020-06-20 13:34:20

---------------------------------------------------------------
Agent is Running and Ready

It can be found that "Heartbeat Status" shows "Agent is blocked", and "Blocked Reason" is "Agent is out-of-sync with repository" because "most likely" "the agent was reinstalled or recovered" and the proposed solution is "unblock the agent by performing an agent resync from the console". It is smart enough and knows the agent was restored/recovered.

Try the proposed solution to resynchronize the agent with emcli command on OMS server,
[oracle@oms1]$emcli login -username=sysman
Login successful

[oracle@oms1]$emcli sync
Synchronized successfully

[oracle@oms1]$emcli resyncAgent -agent=host01.lab.dbaplus.ca:3872
Resync job RESYNC_20200620135608271 successfully submitted

The Resync job RESYNC_20200620135608271 is created, and it may take a few minutes to complete. 

Note: The resync job can be created by emcli on any host where emcli is installed. If you are not familiar with emcli, it can also be done in OEM GUI console by select menu as folowing,

EM Console -> Setup -> Manage Cloud Control -> Agents -> click on the Agent name ->Expand the Agent menu -> Resynchronization -> Continue -> ok -> click on the job name and verify the job runs successful.

After resync job is completed, "Heartbeat Status" shows "Ok" as following
[oracle@host01]$ emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/product/em/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/product/em/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0
Core JAR Location      : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/jlib
Agent Process ID       : 63177128
Parent Process ID      : 1573530
Agent URL              : https://host01.lab.dbaplus.ca:3872/emd/main/
Local Agent URL in NAT : https://host01.lab.dbaplus.ca:3872/emd/main/
Repository URL         : https://oms1.lab.dbaplus.ca:4903/empbs/upload
Started at             : 2020-06-20 11:06:45
Started by user        : oracle
Operating System       : AIX version 7.1 (ppc64)
Number of Targets      : 42
Last Reload            : (none)
Last successful upload                       : 2020-06-20 15:30:08
Last attempted upload                        : 2020-06-20 15:30:08
Total Megabytes of XML files uploaded so far : 2.33
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 32.81%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2020-06-20 15:30:28
Last successful heartbeat to OMS             : 2020-06-20 15:30:28
Next scheduled heartbeat to OMS              : 2020-06-20 15:31:28

---------------------------------------------------------------
Agent is Running and Ready

You have to wait longer when you get following message while running emctl status agent
--> Compute Dynamic Properties (total operations: 41, active: 2, finished: 37)

Current target operations in progress
-------------------------------------
        rac_database.CTHEISPD - ADD_TARGET running for 18 seconds
        cluster.cluster01 - ADD_TARGET running for 18 seconds

Dynamic property executor tasks running
------------------------------
        TaskName = CDProps:cluster.cluster01:MgmtDBName submitted at 2020-06-20 13:56:09 EDT
        TaskName = CDProps:rac_database.CTHEISPD:GetAdrBase submitted at 2020-06-20 13:56:09 EDT

No comments: