OEM 13c shows database and ASM instance DOWN when TCPS (TLS/SSL) enabled

Database server is configured with TCPS only, ASM and database instance can only be accessed with TCPS.

Both ASM and database instances are configured successfully in OEM. Test Connection succeeded in Monitoring Configuration page.

On database and ASM home page, OEM can connect to target and manage objects. However, the target status shows DOWN.

On database server, following command returns errors

   emctl getmetric agent <target_name>,<target_type>,Response

The output as following,

[oracle@host01]$ emctl getmetric agent DB01,oracle_database,Response
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Status,State,oraerr,Archiver,DatabaseStatus,ActiveState
0,UNKNOWN,Failed to connect: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection,UNKNOWN,UNKNOWN,UNKNOWN
[oracle@host01]$ 
[oracle@host01]$ emctl getmetric agent +ASM_host01.lab.dbaplus.ca,osm_instance,Response
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Status,oraerr
0,Failed to connect: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Turn on agent debug mode on database server with commands,

  emctl getproperty agent -name 'Logger.log.level'

  emctl setproperty agent -name 'Logger.log.level' -value 'DEBUG'

  emctl reload agent

[oracle@host01]$ emctl getproperty agent -name 'Logger.log.level'
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Logger.log.level=INFO
[oracle@host01]$ 
[oracle@host01]$ emctl setproperty agent -name 'Logger.log.level' -value 'DEBUG'
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
EMD setproperty succeeded
[oracle@host01]$ 
[oracle@host01]$ emctl reload agent
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully
[oracle@host01]$ 
[oracle@host01]$ emctl getproperty agent -name 'Logger.log.level'
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Logger.log.level=DEBUG

Re-run following commands to reproduce error,

  emctl getmetric agent <target_name>,<target_type>,Response

Turn off agent debug mode on database server with commands,

  emctl setproperty agent -name 'Logger.log.level' -value <original_level>

  emctl reload agent

[oracle@host01]$ emctl setproperty agent -name 'Logger.log.level' -value 'INFO'
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
EMD setproperty succeeded
[oracle@host01]$ 
[oracle@host01]$ emctl reload agent
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully

Check <agent_instance_home>/sysman/log/gcagent.log, will find following error,

  No class found for OBJECT IDENTIFIER {1 2 840 113549 2 9}

Log file example,

Caused by: oracle.security.crypto.asn1.ASN1FormatException: oracle.security.crypto.core.CipherException: oracle.security.crypto.core.InvalidKeyException: oracle.security.crypto.core.AlgorithmIdentifierException:
No class found for OBJECT IDENTIFIER {1 2 840 113549 2 9}
        at oracle.security.crypto.cert.PKCS12Safe.input(PKCS12Safe.java:222)
        at oracle.security.crypto.cert.PKCS12Safe.<init>(PKCS12Safe.java:129)
        at oracle.security.crypto.cert.PKCS12.input(PKCS12.java:179)
        at oracle.security.crypto.cert.PKCS12.<init>(PKCS12.java:122)
        at oracle.security.pki.OracleKeyStoreSpi.load(Unknown Source)
        at oracle.security.pki.OracleKeyStoreSpi.engineLoad(Unknown Source)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at oracle.net.nt.CustomSSLSocketFactory.loadFileBasedKeyStore(CustomSSLSocketFactory.java:1066)
        at oracle.net.nt.CustomSSLSocketFactory.loadKeyStore(CustomSSLSocketFactory.java:1038)
        at oracle.net.nt.CustomSSLSocketFactory.getTrustManagerArray(CustomSSLSocketFactory.java:669)
        ... 50 more
2026-01-29 11:58:41,454 [4898:E9A2CC82:GC.Executor.53 (oracle_database:DB01:Response)] DEBUG - End task oracle_database:DB01:Response:Response

The error was thrown by agent when agent accesses trusted certificate wallet. It can be proven by run orapki shipped with agent software,

   <AGENT_HOME>/oracle_common/bin/orapki wallet display -wallet <WALLET_PATH>/ewallet.p12

For example,

[oracle@host01]$ ./orapki wallet display -wallet /u01/wallet/ewallet.p12
Oracle PKI Tool : Version 12.2.1.4.0
Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:   
oracle.security.crypto.core.CipherException: oracle.security.crypto.core.InvalidKeyException: oracle.security.crypto.core.AlgorithmIdentifierException: 
No class found for OBJECT IDENTIFIER {1 2 840 113549 2 9}

Cause

The wallet was generated with the latest database 19c orapki tool, but the agent still has the older orapki, and not able to read the certificates.

Solution

Convert wallet to lower version format which agent orapki can read. The conversion can be done by database 19c orapki

   <DATABASE_19c_HOME>/bin/orapki wallet convert -wallet <wallet_path>/ewallet.p12 -compat_v11

Note: agent orapki cannot be used for conversion.

$ orapki wallet convert -wallet /u01/wallet/ewallet.p12 -compat_v11
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2025, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:   
Operation is successfully completed.