Wednesday, August 24, 2022

Patch Oracle WebLogic Server 12c Release 2 12.2.1.x (12.2.1.3, 12.2.1.4) including WLS, OHS, Coherence, ADR and RDA

This instruction applies to Oracle WebLogic Server (WLS) 12.2.1.x standalone (Lite, Slim, Generic), Oracle Fusion Middleware (FMW) 12.2.1.x Infrastructure and WLS 12.2.1.x installed with Oracle Enterprise Manager (OEM) 13c.

The installation of these WLS releases includes following components,

  OPatch
  WebLogic Server
  Coherence Server
  Oracle HTTP Server (OHS)
  Automatic Diagnostic Repository (ADR) 

Oracle currently delivers the latest Critical Patch Updates for these components on a quarterly basis: January, April, July, and October of each year. Oracle highly recommends to apply these updates to secure your environment. 

Technically, the patches (one-off or bundle) for these components are patched independently. The patches are downloaded and installed separately for different components. It is frustrating for the administrator to figure out which component needs which patch and where to download different component patches. To simplify the download and patching process, starting with April 2022, a Stack Patch Bundle (SPB) is made available for Oracle WebLogic Server versions 12.2.1.x (12.2.1.3 and 12.2.1.4), it includes the WebLogic Server PSU and other required updates (for other components) in a single bundle.

1. Upgrade OPatch

OPatch has to be upgraded before other patches are applied. 

* Download patch 28186730, which is the latest version of OPatch for Enterprise Manager (EM) and Fusion Middle Ware (FMW) / WebLogic Server (WLS). Version 13.9.4.2.8 is current OPatch version for EM 13.5 ans WLS 12.2.1.4 and default file name is p28186730_139428_Generic.zip.

Note: WLS Stack Patch Bundle (SPB) already includes required OPatch version and It does not need to download OPatch separately if you patch WebLogic Server with SPB.

* Unzip downloaded OPatch to a temporary location outside of your Oracle home, fox example
$ unzip p28186730_139428_Generic.zip -d /stage/opatch
Archive:  p28186730_139428_Generic.zip
   creating: /stage/opatch/6880880/
  inflating: /stage/opatch/6880880/README.txt
  inflating: /stage/opatch/6880880/opatch_generic.jar
  inflating: /stage/opatch/6880880/version.txt
* If using OPatch shipped with SPB, unzip OPatch from unzipped SPB, fox example
$ unzip /stage/WLS_SPB_12.2.1.4.220418/tools/opatch/generic/p28186730_139428_Generic.zip -d /stage/opatch
Archive:  /stage/WLS_SPB_12.2.1.4.220418/tools/opatch/generic/p28186730_139428_Generic.zip
   creating: /stage/opatch/6880880/
  inflating: /stage/opatch/6880880/README.txt
  inflating: /stage/opatch/6880880/opatch_generic.jar
  inflating: /stage/opatch/6880880/version.txt

* Stop all processes which are using files from Oracle home

If Oracle Enterprise Manager OMS home, run following command to stop all processes

   $ORACLE_HOME/bin/emctl stop oms -all

* Backup Oracle home and Central Inventory, there is no mechanism to rollback to the older OPatch version without backup

* Check environment for OPatch

Log in and run commands as the OS user that installed Oracle WebLogic Server.

Unset the JAVA_HOME and ORACLE_HOME environment variables.

Verify that the JAVA_HOME setting in the <ORACLE_HOME>/oui/.globalEnv.properties is where you have installed your Java SE (JDK/JRE).

Example of OEM 13.5
$ cat /oracle/em13.5/middleware/oui/.globalEnv.properties
#This file is automatically generated
#Wed Jun 22 17:10:45 EDT 2022
COMMON_HOME=/oracle/em13.5/middleware/oracle_common
EMD_ROOT=/oracle/em13.5/agent/agent_13.5.0.0.0
JAVA_HOME=/oracle/em13.5/middleware/oracle_common/jdk
JAVA_HOME_1_8=/oracle/em13.5/middleware/oracle_common/jdk
JVM_64=
OMS_HOME=/oracle/em13.5/middleware/
ORACLE_HOME=/oracle/em13.5/agent/agent_13.5.0.0.0
PERL_HOME=/oracle/em13.5/agent/agent_13.5.0.0.0/perl
Example of standalone WebLogic Server 12.2.1.4
$ cat /oracle/Middleware/oui/.globalEnv.properties
#This file is automatically generated
#Sun Dec 12 10:59:49 EST 2021
JAVA_HOME=/oracle/jdk1.8.0_latest
JAVA_HOME_1_8=/oracle/jdk1.8.0_latest
JVM_64=

Note: To prevent OPatch from entering into self-patching mode, and to avoid inventory corruption issues, ensure that no OPatch operations are consuming a JDK/JRE located inside the ORACLE_HOME.  More details about OPatch self-patching mode, please check following document from Oracle Support

OPatch : Following Messages during Patching - "Start OOP by Prereq process. Launch OOP..." (Doc ID 2624030.1)

* Install OPatch with following command

  java [-Djava.io.tmpdir=<TEMP_DIR>] -jar <PATCH_HOME>/6880880/opatch_generic.jar -silent oracle_home=<ORACLE_HOME> [-invPtrLoc <INVENTORY_LOCATION>]

  Where 

     "-Djava.io.tmpdir=<TEMP_DIR>" is optional, it sets a custom location for the logs, and <TEMP_DIR> is an absolute path where the logs will be created. By default, /tmp is used for log location.
     "<PATCH_HOME>" is the location where OPatch file is unzipped, it is "/u01/opatch" in my example.
     "<ORACLE_HOME>" is the absolute path where you have installed FMW/WLS products. In my example, it is "/u01/app/oracle/em13.5/middleware"
     "-invPtrLoc <INVENTORY_LOCATION>" is optional and used to specify a custom Inventory location, <INVENTORY_LOCATION> is the absolute path to the oraInst.loc file

     The console output will show location of log files, it should be under "/tmp/OraInstall<TIMESTAMP>" or custom location if you have specified with "-Djava.io.tmpdir=<TEMP_DIR>". If OPatch upgrade session succeeds, logs will be copied under <CENTRAL_INVENTORY>/logs.

Example output
$ echo $ORACLE_HOME
/oracle/em13.5/middleware

$ $ORACLE_HOME/oracle_common/jdk/bin/java -jar /u01/opatch/6880880/opatch_generic.jar -silent oracle_home=$ORACLE_HOME
Launcher log file is /tmp/OraInstall2022-06-22_05-10-45PM/launcher2022-06-22_05-10-45PM.log.
Extracting the installer . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 886.505 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 20479 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (-d64 flag is not required)
Checking temp space: must be greater than 300 MB.   Actual 20201 MB    Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2022-06-22_05-10-45PM
Installation Summary


Disk Space : Required 41 MB, Available 97,832 MB
Feature Sets to Install:
        Next Generation Install Core 13.9.4.0.1
        OPatch 13.9.4.2.8
        OPatch Auto OPlan 13.9.4.2.8
Session log file is /tmp/OraInstall2022-06-22_05-10-45PM/install2022-06-22_05-10-45PM.log

Loading products list. Please wait.
  '''
The install operation completed successfully.

Logs successfully copied to /u01/app/oraInventory/logs.

2. Apply patches on WLS components

2.1 Option 1  Patch WLS with SPB, a new way starting in April 2022

If installing quarterly Patch Set Updates released in April 2022 or later, the SPB can be used to download and patch all WLS components with a single bundle.

* Downloaded SPB from Oracle Support site. For example, the April 2022 SPB for WebLogic 12.2.1.4 is Patch 34080315. The default patch file name is

 p34080315_122140_Generic.zip

* Unzip the file to stage/temporary directory

After unzipping the patch, a directory WLS_SPB_12.2.1.x.<VERSION> should be created with the following contents:

Note: Do not make any changes to this directory structure.

    WLS_SPB_12.2.1.x.<VERSION>/

        -|binary_patches (dir)
        -|README.txt (file)
        -|README.html (file)
        -|spbat-bundle.properties (file)
        -|tools (dir)

Under sub-directory tools/opatch/generic, OPatch installation file can be found, this is the version required by current SPB. If OPatch is older than required, it can upgraded with this OPatch file.

Under sub-directory binary_patches, following files (patch list file) can be found

    WLS_SPB_12.2.1.x.<VERSION>/binary_patches/
        -|aix64_patchlist.txt
        -|generic_patchlist.txt
        -|hpuxia64_patchlist.txt
        -|linux64_patchlist.txt
        -|linux_zser_patchlist.txt
        -|rollback_patchlist.txt
        -|solaris64_patchlist.txt
        -|solaris_sparc64_patchlist.txt
        -|windows64_patchlist.txt

Depending on the platform of your WebLogic is running on, the corresponding file is required by OPatch to apply the patches.

Unzipped file example,
$ unzip -q p34080315_122140_Generic.zip
$ 
$ ls
WLS_SPB_12.2.1.4.220418
$
$ ls -l WLS_SPB_12.2.1.4.220418
total 52
drwxr-xr-x 9 oracle oinstall  4096 Jun 22 17:28 binary_patches
-rw-r--r-- 1 oracle oinstall 38531 Apr 19 09:20 README.html
-rw-r--r-- 1 oracle oinstall    21 Apr 19 09:20 README.txt
-rwxr-xr-x 1 oracle oinstall   181 Apr 18 03:24 spbat-bundle.properties
drwxr-xr-x 5 oracle oinstall    44 Apr 18 03:23 tools
$
$ ls -l WLS_SPB_12.2.1.4.220418/tools/opatch/generic
total 51320
-rwxr-xr-x 1 oracle oinstall 52547596 Apr 18 03:23 p28186730_139428_Generic.zip
$
$ ls -l WLS_SPB_12.2.1.4.220418/binary_patches/*txt
-rwxr-xr-x 1 oracle oinstall 109 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/aix64_patchlist.txt
-rwxr-xr-x 1 oracle oinstall  98 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/generic_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 112 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/hpuxia64_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 111 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/linux64_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 114 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/linux_zser_patchlist.txt
-rwxr-xr-x 1 oracle oinstall  71 Apr 18 03:24 WLS_SPB_12.2.1.4.220418/binary_patches/rollback_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 113 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/solaris64_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 119 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/solaris_sparc64_patchlist.txt
-rwxr-xr-x 1 oracle oinstall 113 Apr 18 03:23 WLS_SPB_12.2.1.4.220418/binary_patches/windows64_patchlist.txt
* (Optional) Run OPatch with the -report flag to produce a log of patches that will be applied, but does not actually update anything

Navigate to the binary patches directory

  WLS_SPB_12.2.1.x.<VERSION>/binary_patches

Run OPatch with the -report flag to produce a log to review:

  <ORACLE_HOME>/OPatch/opatch napply -report -oh <ORACLE_HOME> -phBaseFile <patch_list_file>

Make sure the -phBaseFile value corresponds to your specific UNIX operating platform. For example, Oracle Linux 64 bit needs file linux64_patchlist.txt.

Note: Use generic_patchlist.txt to apply generic patches only in case their install platform specific file is not provided.

Once the opatch napply -report command is executed, review the output of the command and the logs it produces to be aware of any possible issues. No changes have been made to your Oracle home even though the command output may say that patches were successfully applied.

Example on Oracle Linux 8
$ cd WLS_SPB_12.2.1.4.220418/binary_patches/

$ /u01/app/oracle/em13.5/middleware/OPatch /opatch napply -report -oh /u01/app/oracle/em13.5/middleware -phBaseFile linux64_patchlist.txt
Oracle Interim Patch Installer version 13.9.4.2.8
Copyright (c) 2022, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/em13.5/middleware
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/em13.5/middleware/oraInst.loc
OPatch version    : 13.9.4.2.8
OUI version       : 13.9.4.0.0
Log file location : /u01/app/oracle/em13.5/middleware/cfgtoollogs/opatch/opatch2022-06-22_17-20-11PM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/em13.5/middleware"

Verifying environment and performing prerequisite checks...
Skip patch 33868012 from list of patches to apply: This patch is not needed.

  ... <<output truncated>> ...

ApplySession skipping inventory update.
Patches 1221413,32647448,32720458,33093748,34012040,34077658,34080360 successfully applied.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/em13.5/middleware/cfgtoollogs/opatch/opatch2022-06-22_17-20-11PM_1.log

OPatch completed with warnings.
* Run OPatch to install the patches included in the SPB

Navigate to the binary_patches directory

  WLS_SPB_12.2.1.x.<VERSION>/binary_patches

Run opatch napply:

  <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -phBaseFile <patch_list_file>

Make sure the -phBaseFile value corresponds to your specific UNIX operating platform.

The patchlist files are bundled into the SPB, so you do not need to provide a path to the file.
Once the opatch napply command is executed, review the output of the command and the logs it produces to be aware of any possible issues.
Changes have been made to your Oracle home if this command was successful.

Example on Oracle Linux
$ cd WLS_SPB_12.2.1.4.220418/binary_patches/

[oracle@saxon]$ /u01/app/oracle/em13.5/middleware/OPatch/opatch napply -oh /u01/app/oracle/em13.5/middleware -phBaseFile linux64_patchlist.txt
Oracle Interim Patch Installer version 13.9.4.2.8
Copyright (c) 2022, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/em13.5/middleware
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/em13.5/middleware/oraInst.loc
OPatch version    : 13.9.4.2.8
OUI version       : 13.9.4.0.0
Log file location : /u01/app/oracle/em13.5/middleware/cfgtoollogs/opatch/opatch2022-06-22_17-27-45PM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/em13.5/middleware"

Verifying environment and performing prerequisite checks...
Skip patch 33868012 from list of patches to apply: This patch is not needed.

Conflicts/Supersets for each patch are:

Patch : 32647448

Better duplicate of 31544353

Patch : 34012040

Bug Superset of 32698246
Super set bugs are:
31498405, 30510407, 32235275, 30295025, 31232471, 30362086, 32307656, 30558254, 32214441, 32016868, 30670689, 30729141, 31510290, 30771358, 30961904, 30589563, 30342923, 30469093, 31234666, 31765567, 32068710, 30468443, 29940841, 31316252, 30068341, 29660156, 32371861, 30838007, 30465861, 31567049, 30734182, 26547727, 30884852, 31142740, 31526201, 30230430, 31401659, 32108759, 30866037, 31118905, 30568713, 30155056, 30885114, 31724565, 31234573, 30885237, 30718589, 31765550, 30153412, 30633620, 30652362, 30326976, 32415913, 25219796, 30885128, 31011293, 31441174, 29247835, 31657139, 31671559, 30740009, 32248716, 32412974, 31332368, 30478451, 30067299, 30885217, 31332264, 18183882, 30563848, 30459026, 29769772, 30692988, 32244262, 30964331, 30814590, 32373306, 29878681, 32312961, 30341541, 32228228, 29971088, 31770512, 32528774, 31975423, 31113242, 25973136, 32425607, 29449188, 30284059, 32069620, 30958807, 31353368, 30362026, 32054481, 30837932, 31247235, 29630055, 30801769, 31564423, 31160218, 26444945, 31297042, 31913015, 30624882, 30285053, 30469341, 31380363, 31157988, 29671344, 31047981

Patch : 1221413

Bug Superset of 122146
Super set bugs are:
31470730, 31806259, 30689686, 30564187, 30729380, 31030896


Patches [   31544353 ] will be rolled back.

OPatch continues with these patches:   1221413  32647448  32720458  33093748  34012040  34077658  34080360  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/em13.5/middleware')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '1221413' to OH '/u01/app/oracle/em13.5/middleware'
Rolling back interim patch '31544353' from OH '/u01/app/oracle/em13.5/middleware'

...<<output truncated>>...
Patching component oracle.com.fasterxml.jackson.dataformat.jackson.dataformat.xml, 2.9.9.0.0...
Applying interim patch '34077658' to OH '/u01/app/oracle/em13.5/middleware'
ApplySession: Optional component(s) [ oracle.rda, 8.15.17.03.14 ] , [ oracle.rda, 20.1.20.1.21 ] , [ oracle.rda, 20.2.20.4.21 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rda, 19.3.19.8.2...
Applying interim patch '34080360' to OH '/u01/app/oracle/em13.5/middleware'

Patching component oracle.wls.core.app.server, 12.2.1.4.0...
Patches 1221413,32647448,32720458,33093748,34012040,34077658,34080360 successfully applied.
Sub-set patch [122146] has become inactive due to the application of a super-set patch [1221413].
Sub-set patch [32698246] has become inactive due to the application of a super-set patch [34012040].
Please refer to Doc ID 2161861.1 for any possible further required actions.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/em13.5/middleware/cfgtoollogs/opatch/opatch2022-06-22_17-27-45PM_1.log

OPatch completed with warnings.
* Start all WebLogic Servers.

2.2 Option 2  Patch one component at one time, a popular way before April 2022 and still an alternative method when SPB is not applicable

When applying patches released before April 2022 or only for single component (e.g. OHS, Coherence, etc.), the normal (old traditional) way has to be used.

* Download required patch for the specific component

  WebLogic Server Patch Set Update (PSU), such as Patch 33727616 - January 2022 Patch Set Update (PSU) for WebLogic Server 12.2.1.4  (PSU 12.2.1.4.220105)
  Oracle HTTP Server (OHS) Bundle Patch, such as Patch 32673423 - Oracle OHS (Native) Bundle Patch 12.2.1.4.210324
  Oracle Coherence cumulative patches, such as Patch 33591019 - Coherence 12.2.1.4 Cumulative Patch 12 (12.2.1.4.12)

* Unzip the downloaded file, and run "opatch apply" to install the patch

No comments: