Wednesday, April 22, 2026

OEM 13c WebTier Could Not Be Started Due To The Default OHS Keystore Certificate Expired

OMS start failed with error "WebTier Could Not Be Started."

$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Could Not Be Started.
Error Occurred: WebTier Could Not Be Started.
Please check /u01/app/oracle/em13.5/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

emctl.log shows "Failed to start the server ohs1"

2026-04-19 08:06:38,272 [Thread-1] INFO  commands.BaseCommand - <OUT>NMProcess: SEVERE: Failed to start the server ohs1
2026-04-19 08:06:38,275 [Thread-1] INFO  commands.BaseCommand - <OUT>NMProcess: <Apr 19, 2026 8:06:38 AM EDT> <WARNING> <Server start command for OHS server 'ohs1' failed due to: [Failed to start the server ohs1
2026-04-19 08:06:38,275 [Thread-1] INFO  commands.BaseCommand - <OUT>NMProcess: Check log file /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/system_components/OHS/ohs_nm.log
2026-04-19 08:06:38,275 [Thread-1] INFO  commands.BaseCommand - <OUT>NMProcess: Check log file /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/servers/ohs1/logs/ohs1.log]. Please check Node Manager log and/or server 'ohs1' log for detailed information.>
  ... ...
2026-04-19 08:06:38,754 [main] ERROR commands.BaseCommand - WebTier Could Not Be Started.
2026-04-19 08:06:38,754 [main] ERROR wls.OMSController - OMSController failed for start oms
2026-04-19 08:06:38,755 [main] ERROR wls.OMSController - OMSController Error: WebTier Could Not Be Started.
java.lang.Exception: WebTier Could Not Be Started.
        at oracle.sysman.emctl.commands.StartCommand.startOMS(StartCommand.java:431) ~[emCoreSDKImpl.jar:?]
        at oracle.sysman.emctl.commands.StartCommand.execute(StartCommand.java:281) ~[emCoreSDKImpl.jar:?]
        at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:246) [emCoreSDKImpl.jar:?]
ohs1.log shows "OHS:2057 Init: (localhost:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791"
[2026-04-19T08:06:37.1384-04:00] [OHS] [ERROR:32] [OH99999] [ossl] [host_id: host01] [host_addr: 10.10.28.25] [pid: 3028810] [tid: 140246783358336] [user: oracle] [VirtualHost: localhost:0] OHS:2057 Init: (localhost:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791
[2026-04-19T08:06:37.1384-04:00] [OHS] [ERROR:32] [OH99999] [ossl] [host_id: host01] [host_addr: 10.10.28.25] [pid: 3028810] [tid: 140246783358336] [user: oracle] [VirtualHost: localhost:0] OHS:2171 NZ Library Error: Unknown error
The error message shows OHS failed to communicate with locallhost using SSL, it usually means invalid certificate.

OHS has four keystores/wallets (default, console, proxy, upload) to store certificates for different purposes. The default wallet is used for SSL communication to localhost. There wallets are located under directory 
 <GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores
 <GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/keystores

Verify if certificate in default wallet expired, following How to Check Validity of Server Certificate from Oracle Wallet in Command Line.
$ cd /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/default

$ orapki wallet display -wallet ./
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY
Trusted Certificates:
Subject:        CN=CertGenCA,OU=FOR TESTING ONLY,O=MyOrganization,L=MyTown,ST=MyState,C=US

$ orapki wallet export -wallet ./ -dn 'CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY' -cert /tmp/export.cert
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ orapki cert display -cert /tmp/export.cert -summary
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
Copyright (c) 2004, 2019, Oracle and/or its affiliates. All rights reserved.

Subject:        CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY
Issuer:         CN=CertGenCA,OU=FOR TESTING ONLY,O=MyOrganization,L=MyTown,ST=MyState,C=US
Valid Until:    Sat Apr 18 21:54:35 EDT 2026
The certificate has expired on Sat Apr 18 21:54:35 EDT 2026.

Solution

Since the certificate is self signed, we can create a new one with new expiration date as following,
* orapki should run from the location "<OMS_HOME>/oracle_common/bin" not from the "<OMS_HOME>/bin"
* Make sure performing these steps to the OHS stage location not under the instance location:
Stage location:
<GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/default
Instance location:
<GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/keystores/default/
1. Rename existing wallet file
$ cd /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/default
$ ls -l
total 8
-rw-r----- 1 oracle oinstall 4341 Apr 19  2021 cwallet.sso
-rw------- 1 oracle oinstall    0 Apr 19  2022 cwallet.sso.lck

$ mv cwallet.sso cwallet.sso.expired
2. Create a new auto_login wallet with command
   orapki wallet create -wallet <wallet_location> -auto_login_only
$ echo $OMS_HOME
/u01/app/oracle/em13.5/middleware
$ pwd
/u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/default

$ $OMS_HOME/oracle_common/bin/orapki wallet create -wallet ./ -auto_login_only
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ ls -l /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/default
total 4
-rw------- 1 oracle oinstall 4085 Apr 19 11:26 cwallet.sso
-rw-r----- 1 oracle oinstall 4341 Apr 19  2021 cwallet.sso.expired
-rw------- 1 oracle oinstall    0 Apr 19  2022 cwallet.sso.lck

$ $OMS_HOME/oracle_common/bin/orapki wallet display -wallet ./
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Trusted Certificates:
3. Add the new self signed certificate to the wallet with command
   orapki wallet add -wallet <wallet_location> -dn 'CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY' -keysize 2048 -self_signed -validity 3650 -sign_alg sha256 -auto_login_only
$ $OMS_HOME/oracle_common/bin/orapki wallet add -wallet ./ -dn 'CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY' -keysize 2048 -self_signed -validity 3650 -sign_alg sha256 -auto_login_only
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ $OMS_HOME/oracle_common/bin/orapki wallet display -wallet ./
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY
Trusted Certificates:
Subject:        CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY

$ $OMS_HOME/oracle_common/bin/orapki wallet export -wallet ./ -dn 'CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY' -cert /tmp/export.cert
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
$
$ $OMS_HOME/oracle_common/bin/orapki cert display -cert /tmp/export.cert -summary
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Subject:        CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY
Issuer:         CN=localhost,OU=GCDomain ohs1,O=FOR TESTING ONLY
Valid Until:    Wed Apr 16 11:26:53 EDT 2036
New certificate valid in 10 years.

4. Copy new cwallet.sso file from stage location to instances folder location
$ cd /u01/app/oracle/em13.5/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/components
$ cp -p ./OHS/ohs1/keystores/default/cwallet.sso ./OHS/instances/ohs1/keystores/default
In case of multi OMS environment, OHS instance location is deferent on deferent OMS node, for example
On node 1,  <GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/keystores/default/
On node 2,  <GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs2/keystores/default/
On node 3,  <GC_INST>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs3/keystores/default/

5. Restart oms with commands
  <OMS_HOME>/bin/emctl stop oms -all -force 
  <OMS_HOME>/binemctl start oms

No comments: