Friday, April 29, 2022

EM 12c/13c Configure Enterprise Manager with custom or third-party SSL Certificates

When deciding to configure Oracle Enterprise Manager with custom or third-party SSL certificates,  you usually need to complete configuration tasks with following EM components

  *  EM Cloud Control Console
  *  Oracle Management Service (OMS) and EM Agent
  *  WebLogic Server (WLS)

The configuration can be done in following steps.

1. Request third-party SSL certificates

Skip this step if you already have custom or third-party certificates and have imported them into a Oracle wallet.

1.1 Create a wallet for third-party certificates using ORAPKI utility

Run following command below to set the EM environment

  <OMS_HOME>/oracle_common/bin/orapki wallet create -wallet <wallet_location> -auto_login

Here,  <OMS_HOME> is OMS Home path, <wallet_location> is directory path where the wallet will be created

For example
$ orapki wallet create -wallet /u01/app/oracle/wallet -auto_login
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Enter password:
Enter password again:
Operation is successfully completed.

$ ls -l /u01/app/oracle/wallet
total 8
-rw------- 1 oracle oinstall 194 Apr 18 20:26 cwallet.sso
-rw------- 1 oracle oinstall   0 Apr 18 20:26 cwallet.sso.lck
-rw------- 1 oracle oinstall 149 Apr 18 20:26 ewallet.p12
-rw------- 1 oracle oinstall   0 Apr 18 20:26 ewallet.p12.lck
Keep the password safely, it will be needed when importing certificates into wallet. The command should create both "cwallet.sso" and "ewallet.p12" files under specified wallet directory.

1.2 Create Certificate Signing Request (CSR) file

* Add a certificate request to the Oracle wallet created in the previous step with following command

  <OMS_HOME>/oracle_common/bin/orapki wallet add -wallet <wallet_location> -dn <certificate_dn> -keysize 512|1024|2048 -pwd <wallet_password>

Here,
  <wallet_password> is the password entered in previous step
  <certificate_dn> is Subject DN of the requestor/owner of the certificate. It consists of a number of fields which are called relative distinguished names (RDN), following are some of the most common RDNs:
    CN: Common Name
    OU: Organizational Unit
    O: Organization
    L: Locality
    S or ST: State Or Province Name
    C: Country Name

Note: Specify Common Name(CN) with the host name entered while installing OMS. It is DNS alias or host name of the machine where the OMS is installed or the Load Balancer(SLB) Name if the OMS is behind the Load Balancer. For example, I have a Linux server which host name is host01.lab.dbaplus.ca. In addition, an DNS alias oms.lab.dbaplus.ca is also resolved to same IP address and I entered oms.lab.dbaplus.ca as host name while installing OMS. Therefore, I am supposed to use oms.lab.dbaplus.ca as CN. 

For example
$ orapki wallet add -wallet /u01/app/oracle/wallet -dn "CN=oms.lab.dbaplus.ca, OU=Lab, O=DBA Plus Workshop, C=CA" -keysize 2048 -pwd oracle4U!
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.
* List certificate request in the wallet by running command

  <OMS_HOME>/oracle_common/bin/orapki wallet display -wallet <wallet_location>

It confirms if the request is created.

For example
$ orapki wallet display -wallet /u01/app/oracle/wallet
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
Subject:        CN=oms.lab.dbaplus.ca,OU=Lab,O=DBA Plus Workshop,C=CA
User Certificates:
Trusted Certificates:
* Export the CSR to a text file, run command

  <OMS_HOME>/oracle_common/bin/orapki wallet export -wallet <wallet_location> -dn <certificate_dn> -request <certificate_request_file>

Here, <certificate_request_file> is the name of the CSR file being created

For example
$ orapki wallet export -wallet /u01/app/oracle/wallet -dn "CN=oms.lab.dbaplus.ca, OU=Lab, O=DBA Plus Workshop, C=CA" -request /tmp/dbaplus_csr.txt
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ cat /tmp/dbaplus_csr.txt
-----BEGIN NEW CERTIFICATE REQUEST-----
MIICmTCCAYECAQAwVDELMAkGA1UEBhMCQ0ExGjAYBgNVBAoTEURCQSBQbHVzIFdv
cmtzaG9wMQwwCgYDVQQLEwNMYWIxGzAZBgNVBAMTEm9tcy5sYWIuZGJhcGx1cy5j
YTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAO/2LRmuH8iUgsfmgeS5
tn6sRAF5TD4Bl+Y5vP45Mlded+9cGXhDBqZlw1KI8JG8ElcPUTk+JI2mbyqjoOYx
IK9lxnOpCKLcPNyK6EzjzEEJb7H+rFCW+peOhyWKNGm7wzvNZc4iCs1DldgbZgvh
l//VJ0felnsjV5V5hWDzA3UoS94mzOA85WpiNBtOOcNH1FXHGpFs7S+IDc0WVXZv
BYaShE18YF+5hgQ6O0uAKT5ePrjRClGVtrhVr5KBzTXZvhtrM8nz5Pomlhbx3sah
rlB+xI2pxEHIqbv5Or4UWeMcArGZVYVQvEE7J5encHqzEqaof+FG3djwTzkzNUka
+MUCAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQCvFFOAeu3AooGS2HcDWg1c3P7V
oXGwER2R8j2STtjXUc1XNhbzFSF389EOJk3o9/9orlkXks/vRAZjoV47s9CeW/aq
XP5HFPhAa2uz0WVK8Pw+p/N4fcnzdxQSXrX/e59PHbs2Dmym1tLbTxeTjYx1KMuO
eahe54AZY1ZuBG5Zc+Z9f23EBtsx2WpmRYMeLZKbn06NOtCCn7Q7xEr5GaDhH9Q7
fvRpaCvYglG6+HlIFNeGdtopal6K6MghT1908V0U7JZgrdSVnwIkKWLo+EVwF4zD
jPFafgbsd/D9MIDHtZHy6v0AIb9D8On057rNXHDElDbD5gI/SaWIn+1YH4Mm
-----END NEW CERTIFICATE REQUEST-----
1.3 Submit CSR file to a third-party trusted Certificate Authority(CA) to get signed SSL certificates in Base64 format

The third-party trusted Certificate Authority (ex. Verisign, DigiCert, etc) will give you a User Certificate (a.k.a Identity or Server Certificate), one or more intermediate CA certificates (if applicable) and a Root CA certificate.

Note: 
  A. For EM 13.4 or later, EM OMS Console, OMS Upload and Agent can be secured with wildcard SSL certificates and SAN Certificates. When submitting CSR, the wildcard character (*.<DOMAIN_NAME>) must be specified instead of the host name (<HOSTNAME>.<DOMAIN_NAME>).
  B. While securing 13.4 OMS with wildcard certificates,ensure that all the Agents are at version 13.4. Pre 13.4 Agents will not be able to communicate to 13.4 OMS secured with wildcard certificates.
  C. SAN certificates can be created using the orapki command from OEM 13.4 onward
  D. For EM 12.1.0.5 to 13.3.0.0.0, a wallet with SAN certificate can be created using openssl, or another utility, that can be used to secure the OMS or Agent. The orapki cannot be used.
  E. The WebLogic instance supporting EM 13.4 can also be secured with wildcard and SAN certificate after applying RU3 or higher patches.

1.4 Import certificates given by third-party trusted CA

It usually includes three files: user certificate (server certificate for your oms) file, Intermediate CA certificates file and a Root CA certificate file.
  
* To import intermediate CA certificates & Root CA certificate, run command

  <OMS_HOME>/oracle_common/bin/orapki wallet add -wallet <wallet_location> -trusted_cert -cert <certificate_file> -pwd <wallet_password>

* To import user certificates , run command

  <OMS_HOME>/oracle_common/bin/orapki wallet add -wallet <wallet_location> -user_cert -cert <certificate_file> -pwd <wallet_password>

For example, I got following three files from CA,
  Root_CA_Certificate.txt         - Root CA certificate
  Intermediate_CA_Certificate.txt - Intermediate CA certificate
  oms_lab_dbaplus_ca.cer          - User certificate

Import these certificates as following
$ orapki wallet add -wallet /u01/app/oracle/wallet -trusted_cert -cert /tmp/Root_CA_Certificate.txt -pwd oracle4U!
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ orapki wallet add -wallet /u01/app/oracle/wallet -trusted_cert -cert /tmp/Intermediate_CA_Certificate.txt -pwd oracle4U!
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ orapki wallet display -wallet /u01/app/oracle/wallet
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
Subject:        CN=oms.lab.dbaplus.ca,OU=Lab,O=DBA Plus Workshop,C=CA
User Certificates:
Trusted Certificates:
Subject:        CN=INT-dbaplusSub-CA,DC=db,DC=dbaplus,DC=ca
Subject:        CN=INT-dbaplusROOT-CA,OU=Lab,O=DBA Plus Workshop,C=CA

$ orapki wallet add -wallet /u01/app/oracle/wallet -user_cert -cert /tmp/oms_lab_dbaplus_ca.cer -pwd oracle4U!
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

$ orapki wallet display -wallet /u01/app/oracle/wallet
Oracle PKI Tool : Version 12.2.1.4.0SECINF-BP
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=oms.lab.dbaplus.ca,OU=Lab,O=DBA Plus Workshop,C=CA
Trusted Certificates:
Subject:        CN=INT-dbaplusSub-CA,DC=db,DC=dbaplus,DC=ca
Subject:        CN=INT-dbaplusROOT-CA,OU=Lab,O=DBA Plus Workshop,C=CA
2. Secure EM Cloud Control Console with third-party certificates

In case of a multi-OMS setup, the steps below need to be performed on each OMS. The steps can be performed in a Rolling fashion.

* Secure EM console with following command

  <OMS_HOME>/bin/emctl secure console -wallet <wallet_location> [-host <SLB_host_name>]

Here, option '-host <SLB_host_name>' is only needed when the OMS is configured behind an SLB, and <SLB_host_name> is host name of SLB.

Hint: it will copy both wallet files to following directory
 <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/console
You may want manually maintain the wallet files under this directory when the OMS cannot be started/accessed because of certificate issues (e.g. certificate expired) though it is not recommended. 

* Restart OMS with following commands

  <OMS_HOME>/bin>emctl stop oms -all [-force]
  <OMS_HOME>/bin>emctl start oms

For example
$ emctl secure console -wallet /u01/app/oracle/wallet
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Securing Console... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing Console... Successful
Restart OMS

$ emctl stop oms -all -force
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down

$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
* Switch EM Cloud Control Console back to the EM self-signed certificate

In case of any issues, the EM console access can be rolled back to to self-signed certificate with following commands,

  <OMS_HOME>/bin>emctl secure console -self_signed
  <OMS_HOME>/bin>emctl stop oms -all -force
  <OMS_HOME>/bin>emctl start oms
           
Run the commands on each OMS for multiple OMS setup. The commands is applicable if the OMS is configured behind a load balancer also

3. Secure OMS and Agent with third-party certificate

* Merge all trusted certificates files into one file

Copy the trusted certificates that you obtained from the third party (intermediate certificate and the root certificate) into one file. There should be no special characters, empty lines or extra blank spaces in the file. The certificates need not be placed in any particular order.

In my example, the intermediate certificate file "Intermediate_CA_Certificate.txt"  and root certificate "Root_CA_Certificate.txt" are merged into a file "trusted_certs.txt" as following
$ cat /tmp/Intermediate_CA_Certificate.txt > /tmp/trusted_certs.txt
$ cat /tmp/Root_CA_Certificate.txt >> /tmp/trusted_certs.txt
* Import third-party trusted certificates into OMS trust store and EM Repository 

It is done by running following command, in case of multiple OMS setup, run the command on each OMS

  <OMS_HOME>/bin/emctl secure oms -trust_certs_loc <trusted_certificates_file> [-protocol TLSv1(EM 12c only)] [-protocol TLSv1.2(EM 13c only)]

If the OMS is configured behind an SLB, run following command:

  <OMS_HOME>/bin/emctl secure oms -host <SLB_host_name> -secure_port <port> -slb_port <port> -slb_console_port <port> -trust_certs_loc <trusted_certificates_file> [-protocol TLSv1(EM 12c only)] [-protocol TLSv1.2(EM 13c only)]
  
Here, <trusted_certificates_file> is the file created by merging all trusted certificates files ("/tmp/trusted_certs.txt" in my example).

For example
$ emctl secure oms -trust_certs_loc /tmp/trusted_certs.txt -protocol TLSv1.2
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Enter Agent Registration Password : 
Securing OMS... Successful
Restart OMS

* Restart OMS

Although previous command imported trusted certificates to the OMS trust store and EM repository. The OMS is still running with existing certificates. Running following commands to restart OMS to make it run with new imported certificates.

  <OMS_HOME>/bin>emctl stop oms -all -force
  <OMS_HOME>/bin>emctl start oms

In case of multiple OMS setup, OMS can be restarted in a rolling fashion.

* Secure the agent running on OMS host

  $AGENT_HOME/bin/emctl secure agent

Note: run emctl from agent home, do not run from OMS home.

For example
$ emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

$ emctl secure agent
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password : 
Securing agent...   Successful.

$ emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting agent ............ started.

* Secure agents running on all other hosts

It can be done one by one on each host in same way as on OMS hosts. Or secure multiple agents together with following emcli command

  emcli secure_agents [-agt_names="agt1;agt2;..."] [-agt_names_file="<file>"] [-group_name="group_name"] [-use_pref_creds] [-username="username"] [-credential_name="Named credentials"] [-password="password"] [-disable_ca_check]

The emcli command can be run on any hosts (OMS hosts or other target hosts) where emcli is deployed. Before executing emcli, a agent list file has to be created, each agent has one line in the file in following format

  <host_name>:<port>

Here, <host_name> is the host name of the server where agent is running on
          <port> is the port where agent listens on, default is 3872

* Secure OMS with third-party certificate

  <OMS_HOME>/bin/emctl secure oms -wallet <wallet_location> -trust_certs_loc <trusted_certificates_file> [-protocol TLSv1(EM 12c only)] [-protocol TLSv1.2(EM 13c only)]

If the OMS is configured behind an SLB, run following command

  <OMS_HOME>/bin/emctl secure oms -host <SLB_host_name> -wallet <wallet_location> -secure_port <port> -slb_port <port> -slb_console_port <port> -trust_certs_loc <trusted_certificates_file>

For example
$ emctl secure oms -wallet /u01/app/oracle/wallet -trust_certs_loc /tmp/trusted_certs.txt -protocol TLSv1.2
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Enter Agent Registration Password : 
Securing OMS... Successful
Restart OMS
Hint: it will copy both wallet files to following directory
 <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/upload
You may want manually maintain the wallet files under this directory when the OMS cannot be started/accessed because of certificate issues (e.g. certificate expired) though it is not recommended. If certificate in the wallet under this directory expired. You may get following errors when starting OMS
$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Could Not Be Started.
Error Occurred: WebTier Could Not Be Started.
Please check /u01/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details
And check OHS server log file 
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/servers/ohs1/logs/ohs1.log
following errors shows up
  Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791
  OHS:2171 NZ Library Error: Unknown error

Example OHS server log output
[2022-04-25T11:34:59.6852-04:00] [OHS] [ERROR:32] [OH99999] [ossl] [host_id: host01] [host_addr: 192.168.3.210] [pid: 3561] [tid: 1396] [user: oracle] [VirtualHost: oms.lab.dbaplus.ca:0] OHS:2057 Init: (oms.lab.dbaplus.ca:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791
[2022-04-25T11:34:59.6852-04:00] [OHS] [ERROR:32] [OH99999] [ossl] [host_id: host01] [host_addr: 192.168.3.210] [pid: 3561] [tid: 1396] [user: oracle] [VirtualHost: oms.lab.dbaplus.ca:0] OHS:2171 NZ Library Error: Unknown error
* Restart OMS
In case of multiple OMS setup, OMS can be restarted in a rolling fashion.

  <OMS_HOME>/bin/emctl stop oms -all -force
  <OMS_HOME>/bin/emctl start oms

* Rollback OMS to default EM self-signed certificates

It may be needed when OMS certificates are Expired and OMS fails to start. Run following command to rollback

  <OMS_HOME>/bin/emctl secure oms [-protocol TLSv1(EM 12c only)] [-protocol TLSv1.2(EM 13c only)]
 
In case of a multi-OMS setup configured with an SLB, secure each of the OMS with

  <OMS_HOME>/bin/emctl secure oms -host <SLB_host_nameHost name> -secure_port <HTTPS_Upload_Port> -slb_port <SLB_upload_Port> -slb_console_port <SLB_Console_port> 

4. Secure EM WLS with third-party certificates

In case of multi-OMS setup, the steps need to be first performed on all additional OMS servers and lastly on the primary OMS Server where the Admin Server is running.

4.1 Import the trusted certificates (root and intermediate certificates) into the trust store of the Agent (Chained Agent) which is monitoring the OMS target

If failed to do so, the WLS targets will show DOWN in OEM console. The certificates can be imported with following commands,

  <AGENT_HOME>/bin>emctl stop agent
  <AGENT_HOME>/bin>emctl secure add_trust_cert_to_jks -trust_certs_loc <certificate_file> -alias <certificate_alias> [-password <trust_store_password>]
  <AGENT_HOME>/bin>emctl start agent

Repeat "emctl secure add_trust_cert_to_jks" for each CA certificate involved in issuing the custom certificate. Specify a different alias each time. The command details can be found in another article OEM 13c Configure agent to monitor WebLogic Servers secured with custom certificate

List added certificate with command

  <AGENT_HOME>/jdk/bin/keytool -list -alias <certificate_alias> -keystore <AGENT_INSTANCE_HOME>/sysman/config/montrust/AgentTrust.jks -storepass welcome -v
           
Remove certificate with command

  <AGENT_HOME>/jdk/bin/keytool -delete -alias <certificate_alias> -keystore <AGENT_INSTANCE_HOME>/sysman/config/montrust/AgentTrust.jks -storepass welcome -v


4.2 Backup following directories and files on OMS host

* EM 12c
 <OMS_INSTANCE_HOME>/em/EMGC_OMS(n)/emgc.properties
  <OMS_INSTANCE_HOME>/em/EMGC_OMS(n)/embip.properties (If exist)
 <OMS_INSTANCE_HOME>/NodeManager/emnodemanager/nodemanager.properties
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/config.xml
  <OMS_INSTANCE_HOME>/WebTierIH1/config/OHS/ohs1/keystores/proxy

* EM 13c
  <OMS_INSTANCE_HOME>/em/EMGC_OMS(n)/emgc.properties
  <OMS_INSTANCE_HOME>/em/EMGC_OMS(n)/embip.properties (If exist)
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/config.xml
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/nodemanager/nodemanager.properties
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/ohs1/keystores/proxy
  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/fmwconfig/components/OHS/instances/ohs1/keystores/proxy

4.3 Stop OMS

* Run following command to stop OMS

  <OMS_HOME>/bin/emctl stop oms

Note: On the primary OMS, do not use the -all argument while stopping the OMS, the Admin server needs to be up and running.

* Stop BI Publisher if it is confiugre 

  <OMS_HOME>/bin/emctl stop oms -bip_only

* If JVMD and/or ADP is configured, stop the JVMD/ADP engines

  <OMS_HOME>/bin/emctl extended oms jvmd stop –all
  <OMS_HOME>/bin/emctl extended oms adp stop -all

4.4 Secure WLS

  <OMS_HOME>/bin/emctl secure wls -wallet <wallet_location>

For example
$ emctl secure wls -wallet /u01/app/oracle/wallet
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Operation is successfully completed.
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
Note:

 (1) If the OMS is installed with a virtual host name, then set the environment variable below, before executing the "emctl secure wls" command

 On Windows:
  set EM_COMMON_JAVA_OPTIONS="-Dweblogic.security.SSL.ignoreHostnameVerification=true"

 On Linux/Unix:
  export EM_COMMON_JAVA_OPTIONS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.log.FileName=<OMS_ INSTANCE HOME>/sysman/log/wls.log"

 (2) If using EM 12.1.0.3 and below OMS, some additional steps are required for the BI publisher if this has been integrated with the Enterprise Manager. For the details, refer to

 Oracle® Enterprise Manager Cloud Control Advanced Installation and Configuration Guide 12c Release 3 (12.1.0.3), Chapter 15 Integrating BI Publisher with Enterprise Manager, Topic - Securing BI Publisher with a Secure Socket Layer (SSL) Certificate.

 (3) In case of a multi-OMS setup, perform the above steps on each OMS and provide the wallets created with the host name of that OMS machine.

4.5 Stop OMS with -all option and re-start oms

  <OMS_HOME>/bin/emctl stop oms -all -force
  <OMS_HOME>/bin/emctl start oms -admin_only
  <OMS_HOME>/bin/emctl start oms

4.6 Rollback EM WLS to default WLS demo certificates before existing certificates are expired

* Stop OMS as described in step 4.3

Note: On the primary OMS, do not use the -all argument while stopping the OMS, the Admin server needs to be up and running.

* Rollback WLS to use demo certificates with command

  <OMS_HOME>/bin/emctl secure wls -use_demo_cert

For example
sss$ emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 13c Release 5  
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password : 
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'

* Restart OMS as described in step 4.5

4.7 Rollback EM WLS to default WLS demo certificates after existing certificates are expired

If the custom/third-party certificates used with WLS are not valid or already expired, Admin server will not start and will not be accessible from Node Manager even if it is running. Follow the steps below to revert the WLS components in the OMS installation to use the default WebLogic Server demo certificates,

4.7.1 Stop OMS

Note: In case of multi-oms setup, this step is only needed on primary OMS server,where Admin Server is installed

  <OMS_HOME>/bin/emctl stop oms -all -force

Make sure all EM processes are down and no process is running from OMS home. If the command cannot shut down all the processes, you can clean the processes with OS utility, such as "kill -9" command on Linux/Unix. 

4.7.2 Take a backup of file "<OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/config/config.xml", then change the "<key-stores>" element (line)

Note: In case of multi-oms setup, this step is only needed on primary OMS server,where Admin Server is installed

Original file (only interested lines are shown)
<server>
   <name>EMGC_ADMINSERVER</name>
   < ... ...>
	<listen-address>oms.lab.dbaplus.ca</listen-address>
   <resolve-dns-name>true</resolve-dns-name>
   <server-life-cycle-timeout-val>30</server-life-cycle-timeout-val>
   <startup-timeout>0</startup-timeout>
   <key-stores>CustomIdentityAndCustomTrust</key-stores>
   <custom-identity-key-store-file-name>/u01/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/config/keystore/jksfromwallet.jks</custom-identity-key-store-file-name>
   <custom-identity-key-store-type>jks</custom-identity-key-store-type>    <custom-identity-key-store-pass-phrase-encrypted>{AES256}Z3z8wLT54GJqu/BkM437RC5bgYNBhRYKkVMgRR1ou04GmEF8b0shpxE4/hpdeAAz</custom-identity-key-store-pass-phrase-encrypted>
   <custom-trust-key-store-file-name>/u01/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/config/keystore/WLSTrustStore.jks</custom-trust-key-store-file-name>
   <custom-trust-key-store-type>jks</custom-trust-key-store-type>
   <custom-trust-key-store-pass-phrase-encrypted>{AES256}lOGQ/SWKxRF4s3j1frQr/pQRgBXQP05yr7xKjZuPXho=</custom-trust-key-store-pass-phrase-encrypted>
   < ... ...>
</server>
New file (only interested lines are shown)
<server>
   <name>EMGC_ADMINSERVER</name>
   < ... ...>
	<listen-address>oms.lab.dbaplus.ca</listen-address>
   <resolve-dns-name>true</resolve-dns-name>
   <server-life-cycle-timeout-val>30</server-life-cycle-timeout-val>
   <startup-timeout>0</startup-timeout>
   <key-stores>DemoIdentityAndDemoTrust</key-stores>
   <custom-identity-key-store-file-name>/u01/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/config/keystore/jksfromwallet.jks</custom-identity-key-store-file-name>
   <custom-identity-key-store-type>jks</custom-identity-key-store-type>    <custom-identity-key-store-pass-phrase-encrypted>{AES256}Z3z8wLT54GJqu/BkM437RC5bgYNBhRYKkVMgRR1ou04GmEF8b0shpxE4/hpdeAAz</custom-identity-key-store-pass-phrase-encrypted>
   <custom-trust-key-store-file-name>/u01/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/config/keystore/WLSTrustStore.jks</custom-trust-key-store-file-name>
   <custom-trust-key-store-type>jks</custom-trust-key-store-type>
   <custom-trust-key-store-pass-phrase-encrypted>{AES256}lOGQ/SWKxRF4s3j1frQr/pQRgBXQP05yr7xKjZuPXho=</custom-trust-key-store-pass-phrase-encrypted>
   < ... ...>
</server>
Note: Only one line is changed, do not change any other lines, even if they contain entry related to Customkeystores. 

4.7.3 Backup file "<OMS_INSTANCE_HOME>/em/EMGC_OMS1/emgc.properties" and comment out following line by add "#" at the begining of the line

Original line
CUSTOM_TRUST_STORE=<EM_INSTANCE_HOME>/em/EMGC_OMS1/sysman/config/keystore/WLSTrustStore.jks
New line
#CUSTOM_TRUST_STORE=<EM_INSTANCE_HOME>/em/EMGC_OMS1/sysman/config/keystore/WLSTrustStore.jks
4.7.4 Open a new terminal and start Admin Server with following script

Note: In case of multi-oms setup, this step is only needed on primary OMS server,where Admin Server is installed

  <OMS_INSTANCE_HOME>/user_projects/domains/GCDomain/bin/startWebLogic.sh

Wait till the script starts up the Admin server. The script will be running until you close the terminal window or terminate the process by pressing "Ctrl + C". Do NOT stop/terminate the script until the "emctl secure wls" run successfully in next step.

4.7.5 Rollback to WLS demo certificates

Note: In case of multi-OMS setup, run this step on primary OMS server

  <OMS_HOME>/bin/emctl secure wls -use_demo_cert

4.7.6 Kill the Admin server started in step 4.7.4

Note: In case of multi-OMS setup, run this step on primary OMS server

Ensure no process is running from OMS home and kill if any process found.

4.7.5 Start OMS 

Note: In case of multi-OMS setup, run this step on primary OMS server

  <OMS_HOME>/bin/emctl stop oms -all -force
  <OMS_HOME>/bin/emctl start oms -admin_only
  <OMS_HOME>/bin/emctl start oms

4.7.6 In case of multi-OMS setup. run following steps on all other OMS servers

* As described in step 4.7.3, backup and edit file "<OMS_INSTANCE_HOME>/em/EMGC_OMS<n>/emgc.properties". Here, <n> is the sequence number of the OMS server, it is different on each OMS server.

* Secure WLS with command

  <OMS_HOME>/bin/emctl secure wls -use_demo_cert

* Restart OMS server with commands

  <OMS_HOME>/bin/emctl stop oms -all -force
  <OMS_HOME>/bin/emctl start oms

No comments: