Friday, April 24, 2020

opatchauto out-of-place patching Oracle Restart and Database home

Out of place (OOP) patching is a new feature introduced by Oracle 12.2 with the latest version of opatchauto. It is a patching method where patching is performed by following operations,

  * Creating a copy of current oracle home
  * Applying patches to the copied home
  * Switching services to the copied home

This patching method helps on moving Oracle home and/or reducing the downtime of patching Oracle home and database. OPatchauto supports OOP patching for the following configurations,

  * Grid Infrastructure (GI) and Real Application Cluster (RAC)
  * Oracle Restart (Standalone Grid Infrastructure) (SIHA) and single instance database (SIDB)
  
I am going to out-of-place patch SIHA and SIDB from 12.2.0.1.20191015 (12.2 Release Update Oct 2019) to 12.2.0.1.20200414 (12.2 Release Update April 2020).
  
1. Download following Oracle utility/patch files from My Oracle Support
  
   p6880880_122010_SOLARIS64.zip  - Latest version (12.2.0.1.19) of OPatch for Solaris on SPARC (64-bit)
   p30805580_122010_SOLARIS64.zip - Oracle JavaVM Component Release Update 12.2.0.1.200414
   p30920127_122010_SOLARIS64.zip - GI Apr 2020 Release Update 12.2.0.1.200414 
                                    Including Database Apr 2020 Release Update 12.2.0.1.200414
   p25087834_12201200414DBAPR2020RU_SOLARIS64.zip - ORACLE DATABASE Patch for Bug# 25087834
   
   All these files are created for Solaris on SPARC (64-bit) Platform.
  
2. Upgrade OPatch to latest version

It is a best practice for most Oracle database patching process to upgrade OPatch utility to latest version before patching, though latest version is found issues in very a few circumstances where old version works fine. 
  
Backup directory $ORACLE_HOME/OPatch for SIHA as user root,
[root@host01]# ls -ld $ORACLE_HOME
drwxr-x---  98 root     oinstall     115 Apr 24 15:58 /u01/app/grid/product/12.2.0/grid_1
[root@host01]# cd $ORACLE_HOME
[root@host01]# ls -ld OPatch *
drwxr-xr-x  14 grid     oinstall      28 Nov 20 14:13 OPatch
[root@host01]# mv OPatch OPatch.bak
[root@host01]# mkdir OPatch
[root@host01]# chown grid:oinstall OPatch
[root@host01]# ls -ld OPatch*
drwxr-xr-x   2 grid     oinstall       2 Apr 22 09:58 OPatch
drwxr-xr-x  14 grid     oinstall      28 Nov 20 14:13 OPatch.bak

Although grid is home owner of SIHA home, it needs root to move and create directory under the home directory because grid does not have permission to move and create directory under home directory(root has the ownership of directory /u01/app/grid/product/12.2.0/grid_1).

Install new OPatch in SIHA home as home owner grid, 
[grid@host01]$ echo $ORACLE_HOME
/u01/app/grid/product/12.2.0/grid_1
[grid@host01]$ unzip -q -d $ORACLE_HOME /u01/download/p6880880_122010_SOLARIS64.zip
[grid@host01]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.

Backup old OPatch and install new OPatch in SIDB home as home owner oracle
[oracle@host01]$ ls -ld $ORACLE_HOME/OPatch
drwxr-x---  14 oracle   oinstall      28 Apr 22 10:24 /u01/app/oracle/product/12.2.0/dbhome_1/OPatch
[oracle@host01]$ cd $ORACLE_HOME
[oracle@host01]$ ls -ld OPatch*
drwxr-xr-x  14 oracle   oinstall      28 Nov 20 15:57 OPatch
[oracle@host01]$ mv OPatch OPatch.bak
[oracle@host01]$ unzip -q /u01/download/p6880880_122010_SOLARIS64.zip
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.
  
3. Validate patch level of Oracle Home

List patches of SIHA home as user grid
[grid@host01]$ $ORACLE_HOME/OPatch/opatch lspatches
30122828;ACFS OCT 2019 RELEASE UPDATE 12.2.0.1.0 (30122828)
30122814;OCW OCT 2019 RELEASE UPDATE 12.2.0.1.191015 (30122814)
30093408;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:190208.0920) (30093408)
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)
26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)

OPatch succeeded.

Patch 30116802 was installed in Oracle Restart home. It is Grid Infrastructure Oct 2019 Release Update 12.2.0.1.191015 Bundle Patch and includes five patches 30122828, 30122814, 30093408, 30138470 and 26839277.

List patches of SIDB home as user oracle
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch lspatches
30133625;OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)
25087834;
30122814;OCW OCT 2019 RELEASE UPDATE 12.2.0.1.191015 (30122814)
30138470;Database Oct 2019 Release Update : 12.2.0.1.191015 (30138470)

OPatch succeeded.

Four patches are installed in database home. The patching process will upgrade these patches as following,

  OJVM: Patch 30133625 (RU 12.2.0.1.191015) -> Patch 30805580 (RU 12.2.0.1.200414)
  OCW:  Patch 30122814 (RU 12.2.0.1.191015) -> Patch 30882603 (RU 12.2.0.1.200414)
  DB:   Patch 30138470 (RU 12.2.0.1.191015) -> Patch 30886680 (RU 12.2.0.1.200414)
  
Note: one-off patch 25087834 is specific to database RU version, current patch version conflicts with database patch 30886680. Therefore, patch 25087834 has to be rolled back before patch 30886680 can be applied. New version of patch 25087834 will be applied after patch 30886680.

4. Unzip the patches

Make sure you are using correct version of unzip utility. Use $ORACLE_HOME/bin/unzip if you do not know which unzip should be used.

As grid
[grid@host01]$ mkdir -p /u01/stage/grid
[grid@host01]$ unzip -q -d /u01/stage/grid /u01/download/p30920127_122010_SOLARIS64.zip

As oracle
[oracle@host01]$ mkdir -p /u01/stage/database
[oracle@host01]$ unzip -q -d /u01/stage/database /u01/download/p30920127_122010_SOLARIS64.zip
[oracle@host01]$ unzip -q -d /u01/stage/database /u01/download/p30805580_122010_SOLARIS64.zip
[oracle@host01]$ mkdir -p /u01/stage/one-off
[oracle@host01]$ unzip -q -d /u01/stage/one-off  /u01/download/p25087834_12201200414DBAPR2020RU_SOLARIS64.zip

5. Run OPatch Conflict Check

For SIHA home, run following commands as home owner grid

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/grid/30920127/30886680
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/grid/30920127/30882603
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/grid/30920127/30869447
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/grid/30920127/26839277
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/grid/30920127/30888810

Following is sample output of successful OPatch Conflict check, it means the patch can be applied on current home without conflict
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/grid/product/12.2.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/grid/product/12.2.0/grid_1/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/grid/product/12.2.0/grid_1/cfgtoollogs/opatch/opatch2020-04-22_11-12-14AM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

For database home, run following commands as home owner oracle

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/database/30920127/30886680
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/database/30920127/30882603
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/database/30805580

The check succeeds with patch 30882603 and 30805580, but fails with 30886680 (Database Apr 2020 Release Update) because of conflict with patch 25087834. Following is sample output of failed OPatch Conflict check
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/database/30920127/30886680
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/oracle/product/12.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.2.0/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatch/opatch2020-04-22_11-47-20AM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-47: The patch(es) has supersets with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

The details are:
Please rebuild the superset patch [30886680] to make sure it supersedes all the relevant patch(es) [30138470,25087834].
The rebuild patch should contain bug fix [23738304, 27203055, 25223839, 25963024, 28388910, 25437699, 26588069, 25435038, 27607563, 29376346, 26546664,
   <<message truncated>> 
25335249, 28537715, 28856060, 26091786, 24573817, 28951382, 26327624, 20003668, 30244787, 30218044, 27242226, 25489607, 29013832, 26373967, 25442559].

Summary of Conflict Analysis:

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches : 
25087834, 30886680

Conflicts/Supersets for each patch are:

Patch : 30886680

        Conflict with 25087834
        Conflict details:
        /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/kspasv5.var
        /u01/app/oracle/product/12.2.0/dbhome_1/lib/libserver12.a:qesmm.o

        Bug Superset of 30138470
        Super set bugs are:
        25223839, 25963024, 25437699, 25435038, 27607563, 29376346, 26546664, 25872127, 29483723, 28774416, 24907917, 23741944, 24796092, 22310426, 27369515, 
             <<message truncated>> 
        27242226, 25489607, 29013832, 25442559

OPatch succeeded.

Therefore, one-off patch 25087834 has to be rolled back in new Oracle Database home before patch 30886680 is applied.

6. Prepare a clone home by running the following command as user root against Oracle homes (both SIHA and SIDB) 

   $ORACLE_HOME/OPatch/opatchauto apply -phBaseDir <patch-base-directory> -prepare-clone -oh $ORACLE_HOME -logLevel FINEST

Here, option -logLevel FINEST is optional. But detailed log information could be more helpful when diagnosing opatchauto errors.
   
For SIHA home
[root@host01]# echo $ORACLE_HOME
/u01/app/grid/product/12.2.0/grid_1
[root@host01]# $ORACLE_HOME/OPatch/opatchauto apply -phBaseDir /u01/stage/grid -prepare-clone -oh $ORACLE_HOME -logLevel FINEST

OPatchauto session is initiated at Wed Apr 22 18:04:16 2020

System initialization log file is /u01/app/grid/product/12.2.0/grid_1/cfgtoollogs/opatchautodb/systemconfig2020-04-22_06-04-24PM.log.

Session log file is /u01/app/grid/product/12.2.0/grid_1/cfgtoollogs/opatchauto/opatchauto2020-04-22_06-04-35PM.log
The id for this session is DP1E
Please press ENTER button to accept system generated default clone path. 
Please enter clone path [/u01/app/grid/product/12.2.0/grid_2 ] :


Executing OPatch prereq operations to verify patch applicability on home /u01/app/grid/product/12.2.0/grid_1
Patch applicability verified successfully on home /u01/app/grid/product/12.2.0/grid_1


Copying the files from the existing oracle home /u01/app/grid/product/12.2.0/grid_1 to a new location. Please wait...
Clone of oracle home /u01/app/grid/product/12.2.0/grid_1 is /u01/app/grid/product/12.2.0/grid_2 on host host01
Copying the files from the existing oracle home /u01/app/grid/product/12.2.0/grid_1 to a new location is successful.


Unlocking CRS clone home for home /u01/app/grid/product/12.2.0/grid_1.
Prepatch operation log file location: /u01/app/grid/crsdata/host01/crsconfig/hapatch_2020-04-22_06-11-34PM.log
Unlocked CRS clone home successfully for home /u01/app/grid/product/12.2.0/grid_1.


Creating clone for oracle home /u01/app/grid/product/12.2.0/grid_1.
Clone operation successful for oracle home /u01/app/grid/product/12.2.0/grid_1.


Performing post clone operation for oracle home /u01/app/grid/product/12.2.0/grid_1.
Performing post clone operation was successful for oracle home /u01/app/grid/product/12.2.0/grid_2.


Start applying binary patch on home /u01/app/grid/product/12.2.0/grid_2
Binary patch applied successfully on home /u01/app/grid/product/12.2.0/grid_2



OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:host01
SIHA Home:/u01/app/grid/product/12.2.0/grid_1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/grid/30920127/26839277
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/grid/30920127/30869447
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_18-13-10PM_1.log

Patch: /u01/stage/grid/30920127/30882603
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_18-13-10PM_1.log

Patch: /u01/stage/grid/30920127/30886680
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_18-13-10PM_1.log

Patch: /u01/stage/grid/30920127/30888810
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_18-13-10PM_1.log


Out of place patching clone home(s) summary
____________________________________________
Host : host01
Actual Home : /u01/app/grid/product/12.2.0/grid_1
Version:12.2.0.1.0
Clone Home Path : /u01/app/grid/product/12.2.0/grid_2


OPatchauto session completed at Wed Apr 22 18:30:44 2020
Time taken to complete the session 26 minutes, 29 seconds

For SIDB home, command option -force_conflict will be used to roll back conflict patch (Patch 25087834)
[root@host01]# echo $ORACLE_HOME
/u01/app/oracle/product/12.2.0/dbhome_1
[root@host01]# $ORACLE_HOME/OPatch/opatchauto apply -phBaseDir /u01/stage/database -prepare-clone -oh $ORACLE_HOME -force_conflict -logLevel FINEST

OPatchauto session is initiated at Wed Apr 22 19:47:19 2020

System initialization log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-04-22_07-47-28PM.log.

Session log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-04-22_07-48-05PM.log
The id for this session is PVMP
Please press ENTER button to accept system generated default clone path. 
Please enter clone path [/u01/app/oracle/product/12.2.0/dbhome_2 ] :


Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0/dbhome_1


Copying the files from the existing oracle home /u01/app/oracle/product/12.2.0/dbhome_1 to a new location. Please wait...
Clone of oracle home /u01/app/oracle/product/12.2.0/dbhome_1 is /u01/app/oracle/product/12.2.0/dbhome_2 on host host01
Copying the files from the existing oracle home /u01/app/oracle/product/12.2.0/dbhome_1 to a new location is successful.


Creating clone for oracle home /u01/app/oracle/product/12.2.0/dbhome_1.
Clone operation successful for oracle home /u01/app/oracle/product/12.2.0/dbhome_1.


Performing post clone operation for oracle home /u01/app/oracle/product/12.2.0/dbhome_1.
Performing post clone operation was successful for oracle home /u01/app/oracle/product/12.2.0/dbhome_2.


Start applying binary patch on home /u01/app/oracle/product/12.2.0/dbhome_2
Binary patch applied successfully on home /u01/app/oracle/product/12.2.0/dbhome_2



OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:host01
SIDB Home:/u01/app/oracle/product/12.2.0/dbhome_1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/database/30920127/30869447
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /u01/stage/database/30920127/26839277
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /u01/stage/database/30920127/30888810
Reason: This patch is not applicable to this specified target type - "oracle_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/database/30805580
Log: /u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_20-55-43PM_1.log

Patch: /u01/stage/database/30920127/30882603
Log: /u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_20-55-43PM_1.log

Patch: /u01/stage/database/30920127/30886680
Log: /u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-22_20-55-43PM_1.log


Out of place patching clone home(s) summary
____________________________________________
Host : host01
Actual Home : /u01/app/oracle/product/12.2.0/dbhome_1
Version:12.2.0.1.0
Clone Home Path : /u01/app/oracle/product/12.2.0/dbhome_2


OPatchauto session completed at Wed Apr 22 21:08:04 2020
Time taken to complete the session 80 minutes, 45 seconds

It can be seen that new home (clone home) was created with new patches Database Release Update 12.2.0.1.200414 (Patch 30886680) and OJVM Release Update 12.2.0.1.200414 (Patch 30805580). However, the output does not show any information about patch 25087834 which conflicts with patch 30886680.

Check out Patch 25087834 status from log file opatch2020-04-22_20-55-43PM_1.log
[Apr 22, 2020 8:56:40 PM] [INFO]    Checking conflicts against Oracle Home...
[Apr 22, 2020 8:56:40 PM] [INFO]    Running prereq checkConflictAgainstOHWithDetail
[Apr 22, 2020 8:56:42 PM] [INFO]    Following patches can be applied:  30886680
[Apr 22, 2020 8:56:42 PM] [INFO]    Following patches are not required:
[Apr 22, 2020 8:56:42 PM] [INFO]    Following patches are auto rollbackable:  25087834
[Apr 22, 2020 8:56:42 PM] [INFO]    Finished checking prereq checkConflictAgainstOHWithDetail
[Apr 22, 2020 8:56:42 PM] [INFO]    Inter-Conflict check PASSED
    <<message truncated>> 
[Apr 22, 2020 9:09:21 PM] [INFO]     OUI-67087:ApplySession rolling back interim patch '25087834' from OH '/u01/app/oracle/product/12.2.0/dbhome_2'
[Apr 22, 2020 9:09:21 PM] [INFO]     Rolling back interim patch '25087834' from OH '/u01/app/oracle/product/12.2.0/dbhome_2'
[Apr 22, 2020 9:09:21 PM] [INFO]     Starting auto rollback of patch 25087834
[Apr 22, 2020 9:09:21 PM] [INFO]     Rollback preScript for patch 25087834 processed.

The log shows that Patch 25087834 was rolled back during prepare-clone.

Check the patch level of clone home,
[oracle@host01]$ export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_2
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch lspatches
30886680;Database Apr 2020 Release Update : 12.2.0.1.200414 (30886680)
30882603;OCW APR 2020 RELEASE UPDATE 12.2.0.1.200414 (30882603)
30805580;OJVM RELEASE UPDATE: 12.2.0.1.200414 (30805580)

OPatch succeeded.

Note: Do not change configurations in both original home and clone home between prepare-clone and switch-clone (next step). Otherwise, switch-clone will fail because of change of configurations. 

If you dropped a database from original home (e.g. DB03 is dropped) after prepare-clone, switch-clone will fail with errors like following,
[root@magnum]# $ORACLE_HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE_HOME -logLevel FINEST

OPatchauto session is initiated at Fri Apr 24 16:29:15 2020
Session log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-04-24_04-29-24PM.log
The id for this session is Q48R

  <<message truncated>>

Modifying  the database(s) to use new location home /u01/app/oracle/product/12.2.0/dbhome_2.
Failed to configure database to run from   Oracle home home /u01/app/oracle/product/12.2.0/dbhome_2. 

Execution of [DatabaseStartupFinalizeAction] patch action failed, check log for more details. Failures:
Patch Target : host01->/u01/app/oracle/product/12.2.0/dbhome_2 Type[sidb]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/12.2.0/dbhome_2, host: host01.
Command failed:  /u01/app/oracle/product/12.2.0/dbhome_1/bin/srvctl config database -d DB03 -a
Command failure output: 
PRCD-1120 : The resource for database DB03 could not be found.
PRCR-1001 : Resource ora.db03.db does not exist 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Fri Apr 24 16:29:59 2020
Time taken to complete the session 0 minute, 44 seconds

 opatchauto failed with error code 42

If you applied new patch (e.g. new version of patch 25087834 in clone home), switch-clone will fail with errors like that,
[root@host01]# $ORACLE_HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE_HOME -logLevel FINEST

OPatchauto session is initiated at Fri Apr 24 16:15:32 2020
Session log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-04-24_04-15-41PM.log
The id for this session is I34J
OPATCHAUTO-72137: Cannot execute out of place patching switch session.
OPATCHAUTO-72137: Oracle Home patch level is changed.
OPATCHAUTO-72137: Please bring  all oracle home to original patch level and re-run out of place patching switch operation.

Host :: host01

==============================================

Original Patch Level ::
 Oracle Home /u01/app/oracle/product/12.2.0/dbhome_1:  Patch ID: 30133625 Patch ID: 25087834 Patch ID: 30122814 Patch ID: 30138470
 Oracle Home /u01/app/oracle/product/12.2.0/dbhome_2:  Patch ID: 30886680 Patch ID: 30882603 Patch ID: 30805580


==============================================

Current Patch Level ::
 Oracle Home /u01/app/oracle/product/12.2.0/dbhome_1:  Patch ID: 30133625 Patch ID: 25087834 Patch ID: 30122814 Patch ID: 30138470
 Oracle Home /u01/app/oracle/product/12.2.0/dbhome_2:  Patch ID: 25087834 Patch ID: 30886680 Patch ID: 30882603 Patch ID: 30805580


OPatchauto session completed at Fri Apr 24 16:15:57 2020
Time taken to complete the session 0 minute, 26 seconds

 opatchauto failed with error code 106

7. Switch to clone home by running the following command as root

   $ORACLE_HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE_HOME -logLevel FINEST

Switch SIHA home,
[root@host01]# echo $ORACLE_HOME
/u01/app/grid/product/12.2.0/grid_1
[root@dr-jurassic]# $ORACLE _HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE _HOME -logLevel FINEST

OPatchauto session is initiated at Fri Apr 24 15:27:48 2020

Session log file is /u01/app/grid/12.2.0/grid_1/cfgtoollogs/opatchauto/opatchauto2020-04-24_03-28-07PM.log
The id for this session is DSN1


Verifying clone home details before switch for oracle home /u01/app/grid/product/12.2.0/grid_2.
Verified clone home details successfully for oracle home /u01/app/grid/product/12.2.0/grid_2 before switch. 


Update nodelist in the inventory for oracle home /u01/app/grid/product/12.2.0/grid_2.
Update nodelist in the inventory is completed for oracle home /u01/app/grid/product/12.2.0/grid_2. 


Starting CRS service on home /u01/app/grid/product/12.2.0/grid_2
Postpatch operation log file location: /u01/app/grid/crsdata/host01/crsconfig/hapatch_2020-04-24_04-01-41PM.log
CRS service started successfully on home /u01/app/grid/product/12.2.0/grid_2


Confirm that all resources have been started from home /u01/app/grid/product/12.2.0/grid_2.
All resources have been started successfully from home /u01/app/grid/product/12.2.0/grid_2.



OPatchAuto successful.

--------------------------------Summary--------------------------------
Patching session reported following warning(s): 
_________________________________________________

[Note]: Please verify the database is running from the desired Oracle home, if not then manually execute 
 $ORACLE_HOME/bin/srvctl modify database command to fix the problem

 
Out of place patching clone home(s) summary
____________________________________________
Host : host01
Actual Home : /u01/app/grid/12.2.0/grid_1
Version:12.2.0.1.0
Clone Home Path : /u01/app/grid/product/12.2.0/grid_2


OPatchauto session completed at Fri Apr 24 15:31:20 2020
Time taken to complete the session 3 minutes, 32 seconds 
 
Verify if HAS stack (Oracle Restart) is running from new home
[grid@host01]$ ps -ef | grep has
    root 46201     1   0   Apr 24 ?           0:00 /bin/sh /etc/init.d/init.ohasd run
    root  1961     1   0   Feb 27 ?           0:00 /usr/lib/devchassis/devchassisd
    grid 51445     1   0   Apr 24 ?          48:36 /u01/app/grid/product/12.2.0/grid_2/bin/ohasd.bin reboot
    grid  4926  4728   0 17:36:40 pts/6       0:00 grep has
[grid@host01]$ ps -ef | grep asm_pmon
    grid  8210  4728   0 17:42:19 pts/6       0:00 grep asm_pmon
    grid  2204     1   0   Apr 24 ?           2:02 asm_pmon_+ASM
[grid@host01]$ pwdx 2204
2204:   /u01/app/grid/product/12.2.0/grid_2/dbs
[grid@host01]$
[grid@host01]$ ps -ef | grep tns
    grid 10758  4728   0 17:44:56 pts/6       0:00 grep tns
    grid 52721     1   0   Apr 24 ?          68:59 /u01/app/grid/product/12.2.0/grid_2/bin/tnslsnr LISTENER -no_crs_notify -inherit
[grid@host01]$
 
HAS stack (ohasd.bin), ASM instance (asm_pmon_+ASM) and listener (tnslsnr) are all running from new home (/u01/app/grid/product/12.2.0/grid_2).
 
Switch SIDB home,
[root@host01]# echo $ORACLE_HOME
/u01/app/oracle/product/12.2.0/dbhome_1
[root@host01]# $ORACLE_HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE_HOME -logLevel FINEST

OPatchauto session is initiated at Fri Apr 24 16:29:15 2020
Session log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-04-24_04-29-24PM.log
The id for this session is Q48R

Verifying clone home details before switch for oracle home /u01/app/oracle/product/12.2.0/dbhome_2.
Verified clone home details successfully for oracle home /u01/app/oracle/product/12.2.0/dbhome_2 before switch. 


Bringing down database service on home /u01/app/oracle/product/12.2.0/dbhome_2
Database service successfully brought down on home /u01/app/oracle/product/12.2.0/dbhome_2


Bringing down database service on home /u01/app/oracle/product/12.2.0/dbhome_2
Following database has been stopped and will be restarted later during the session: db01,db02,db03
Database service successfully brought down on home /u01/app/oracle/product/12.2.0/dbhome_2


Starting database service on home /u01/app/oracle/product/12.2.0/dbhome_2
Database service successfully started on home /u01/app/oracle/product/12.2.0/dbhome_2


Preparing home /u01/app/oracle/product/12.2.0/dbhome_2 after database service restarted
No step execution required.........
 

Modifying  the database(s) to use new location home /u01/app/oracle/product/12.2.0/dbhome_2.
All database(s) modified successfully to run from new location home /u01/app/oracle/product/12.2.0/dbhome_2.


Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0/dbhome_2
SQL patch applied successfully on home /u01/app/oracle/product/12.2.0/dbhome_2



OPatchAuto successful.

--------------------------------Summary--------------------------------
Out of place patching clone home(s) summary
____________________________________________
Host : host01
Actual Home : /u01/app/oracle/product/12.2.0/dbhome_1
Version:12.2.0.1.0
Clone Home Path : /u01/app/oracle/product/12.2.0/dbhome_2


OPatchauto session completed at Fri Apr 24 16:42:59 2020
Time taken to complete the session 7 minutes, 14 seconds

If Opatch 12.2.0.1.21 is used instead of 12.2.0.1.19, switch-clone could fail with error PRCD-1163 PRCR-1071 CRS-0245 and rollback also fail with error OPATCHAUTO-72114 as following,
[root@host01]# $ORACLE_HOME/OPatch/opatchauto apply -switch-clone -oh $ORACLE_HOME -logLevel FINEST

       <<Content truncated>>
   
Modifying  the database(s) to use new location home /u01/app/oracle/product/12.2.0/dbhome_2.
Failed to configure database to run from   Oracle home home /u01/app/oracle/product/12.2.0/dbhome_2.

Execution of [DatabaseStartupFinalizeAction] patch action failed, check log for more details. Failures:
Patch Target : host01->/u01/app/oracle/product/12.2.0/dbhome_2 Type[sidb]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/12.2.0/dbhome_2, host: host01.
Command failed:  /u01/app/oracle/product/12.2.0/dbhome_2/bin/srvctl modify database -d DB03 -o /u01/app/oracle/product/12.2.0/dbhome_2
Command failure output:
PRCD-1163 : Failed to modify database DB03
PRCR-1071 : Failed to register or update resource ora.db03.db
CRS-0245:  User doesn't have enough privilege to perform the operation

After fixing the cause of failure Run opatchauto resume

]

       <<Content truncated>>
   
[root@host01]#
[root@host01]# $ORACLE_HOME/OPatch/opatchauto rollback -switch-clone -oh $ORACLE_HOME

OPatchauto session is initiated at Thu May 28 21:15:59 2020

System initialization log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-05-28_09-16-22PM.log.

Session log file is /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-05-28_09-16-51PM.log
OPATCHAUTO-72114: Out of place patching rollback session cannot be performed.
OPATCHAUTO-72114: Previous apply/switch session  is not completed on current node.
OPATCHAUTO-72114: Please complete the previous apply/switch  session across all nodes to perform rollback session.
OPatchAuto failed.

OPatchauto session completed at Thu May 28 21:16:53 2020
Time taken to complete the session 0 minute, 55 seconds

 opatchauto failed with error code 42

The error happened because opatchauto shipped with OPatch 12.2.0.1.21 is trying to run srvctl modify database as SIDB home owner oracle to update Oracle home to new home. However, opatchauto shipped with OPatch 12.2.0.1.19 runs the command as SIHA home owner grid. It sounds weird, database (home) owner oracle cannot update information on his own database, but SIHA owner grid can. It does not make sense, but is truth. Of course, both versions of opatchauto will work if SIHA and SIDB are owned by same os user.

As workaround, one of following can be used

   * Update databases' home information by running following command as SIHA user grid,

     <ORIGINAL_DB_HOME>/bin/srvctl modify database -db <database unique name>  -oraclehome <NEW_DB_HOME>  
 
   * Use Opatch 12.2.0.1.19 instead of 12.2.0.1.21
   
Note: Since the failed switch-clone cannot be rolled back (I believe it is Oracle's bug), opatchauto (no matter which version) will not be able to be used for OOP. Not sure how Oracle Support will deal with this issue. I just clean the OOP session information by deleting following file,

   $ORACLE_HOME/.opatchauto_storage/<hostname>/oopsessioninfo.ser

8. Post switch-clone SIDB home tasks as database home owner oracle

After switch database home to new home, perform following actions:

  * Verify all databases are switched to new home
  * Apply Conflict Resolution Patches 
  * Run datapatch to load modified SQL files into the database

8.1 Verify if all databases are switched to new home,
[oracle@qadesh]$ srvctl config database -verbose
DB01  /u01/app/oracle/product/12.2.0/dbhome_2 12.2.0.1.0
DB02  /u01/app/oracle/product/12.2.0/dbhome_2 12.2.0.1.0
DB03  /u01/app/oracle/product/12.2.0/dbhome_2 12.2.0.1.0

The configuration of all three database are changed to new home.

Check the status of databases' instances,
[oracle@host01]$ ps -ef | grep ora_pmon | grep -v grep
  oracle 26280     1   0 16:33:42 ?           0:00 ora_pmon_DB02
  oracle 32375     1   0 16:33:42 ?           0:00 ora_pmon_DB03
  oracle 60303     1   0 16:36:23 ?           0:00 ora_pmon_DB01
[oracle@host01]
[oracle@host01]$ pwdx 26280
26280:   /u01/app/oracle/product/12.2.0/dbhome_1/dbs
[oracle@host01]$ pwdx 32375
32375:   /u01/app/oracle/product/12.2.0/dbhome_1/dbs
[oracle@host01]$ pwdx 60303
60303:   /u01/app/oracle/product/12.2.0/dbhome_2/dbs

Note: If you get 'Permission denied' error while running pwdx as user oracle, try command 'sudo pwdx' if sudo is configured with pwdx, or run pwdx as user root. Command pwdx is only for Solaris, equivalent command on AIX & Linux is ls. If DB01 is running on AIX/Linux, find its home as following,

  ls -l /proc/60303/cwd

Output of pwdx shows that DB01 is running from new home and other databases (DB02 & DB03) are still running from old home. Anyway, it can be fixed by restarting the databases because the configuration of these databases have been changed to new home.

Restart databases (DB02 & DB03) running from old home in new home,
[oracle@host01]$ ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1
[oracle@host01]$ $ORACLE_HOME/bin/srvctl stop database -db DB02
[oracle@host01]$ $ORACLE_HOME/bin/srvctl stop database -db DB03
[oracle@host01]$ ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_2
[oracle@host01]$ $ORACLE_HOME/bin/srvctl start database -db DB02
[oracle@host01]$ $ORACLE_HOME/bin/srvctl start database -db DB03
[oracle@host01]$ ps -ef | grep ora_pmon | grep -v grep
  oracle 60303     1   0 16:36:23 ?           0:00 ora_pmon_DB01
  oracle 65374     1   0 16:40:23 ?           0:00 ora_pmon_DB02
  oracle 65363     1   0 16:46:53 ?           0:00 ora_pmon_DB03
[oracle@host01]
[oracle@host01]$ pwdx 65374
65374:   /u01/app/oracle/product/12.2.0/dbhome_2/dbs
[oracle@host01]$ pwdx 65363
65363:   /u01/app/oracle/product/12.2.0/dbhome_2/dbs

Now, all databases are running from new home.

8.2 Apply Conflict Resolution Patches

One-off patch 25087834 was rolled back by prepare-clone from new home because conflicting with patch 30886680. As the conflict resolution patch, a new version of patch 25087834 has been downloaded and extracted (unzip) under directory /u01/stage/one-off before. Now, it is time to apply this conflict resolution patch.

Apply new version of patch 25087834
[oracle@host01]$ ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_2
[oracle@host01]$ $ORACLE_HOME/bin/srvctl stop home -oraclehome $ORACLE_HOME -statefile /u01/stage/stophome12.2.txt
[oracle@host01]$ cd /u01/stage/one-off/25087834
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/12.2.0/dbhome_2
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.2.0/dbhome_2/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/opatch/opatch2020-04-24_17-11-16PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   25087834  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.2.0/dbhome_2')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '25087834' to OH '/u01/app/oracle/product/12.2.0/dbhome_2'

Patching component oracle.rdbms, 12.2.0.1.0...
Patch 25087834 successfully applied.
Log file location: /u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/opatch/opatch2020-04-24_17-11-16PM_1.log

OPatch succeeded.
[oracle@host01]$ 
[oracle@host01]$ $ORACLE_HOME/OPatch/opatch lspatches
25087834;
30886680;Database Apr 2020 Release Update : 12.2.0.1.200414 (30886680)
30882603;OCW APR 2020 RELEASE UPDATE 12.2.0.1.200414 (30882603)
30805580;OJVM RELEASE UPDATE: 12.2.0.1.200414 (30805580)

OPatch succeeded.
[oracle@host01]$ 
[oracle@host01]$ $ORACLE_HOME/bin/srvctl start home -oraclehome $ORACLE_HOME -statefile /u01/stage/stophome12.2.txt

8.3 Load modified SQL files into the batabase

The output of switch-clone shows that it was trying to apply SQL patch on new home, which probably means all databases running from new home should be applied sql patches (by datapatch). Is it true? 

Check the sql patch status of databases,
SYS@DB01> select c.name,patch_id,version,action,status,description,action_time
  2  from cdb_registry_sqlpatch r, v$containers c
  3  where r.con_id=c.con_id
  4  order by r.con_id,action_time;

DB/PDB
NAME         PATCH_ID VERSION    ACTION     STATUS     DESCRIPTION                                        ACTION_TIME
---------- ---------- ---------- ---------- ---------- -------------------------------------------------- ------------------------------
CDB$ROOT     30133625 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    22-NOV-19 01.47.20.549204 PM
CDB$ROOT     30138470 12.2.0.1   APPLY      SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015   22-NOV-19 01.47.21.656571 PM
CDB$ROOT     30133625 12.2.0.1   ROLLBACK   SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    24-APR-20 06.06.29.060129 PM
CDB$ROOT     30805580 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE 12.2.0.1.200414                24-APR-20 06.06.29.067634 PM
CDB$ROOT     30886680 12.2.0.1   APPLY      SUCCESS    DATABASE APR 2020 RELEASE UPDATE 12.2.0.1.200414   24-APR-20 06.06.31.006964 PM
PDB01        30133625 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    22-NOV-19 01.47.22.906242 PM
PDB01        30138470 12.2.0.1   APPLY      SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015   22-NOV-19 01.47.24.042855 PM
PDB01        30133625 12.2.0.1   ROLLBACK   SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    24-APR-20 06.06.33.141294 PM
PDB01        30805580 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE 12.2.0.1.200414                24-APR-20 06.06.33.147667 PM
PDB01        30886680 12.2.0.1   APPLY      SUCCESS    DATABASE APR 2020 RELEASE UPDATE 12.2.0.1.200414   24-APR-20 06.06.35.161144 PM

SYS@DB02> select c.name,patch_id,version,action,status,description,action_time
  2  from cdb_registry_sqlpatch r, v$containers c
  3  where r.con_id=c.con_id
  4  order by r.con_id,action_time;

DB/PDB
NAME     PATCH_ID VERSION    ACTION     STATUS     DESCRIPTION                                        ACTION_TIME
------ ---------- ---------- ---------- ---------- -------------------------------------------------- ------------------------------
DB02     30133625 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    22-NOV-19 03.46.08.441815 PM
DB02     30138470 12.2.0.1   APPLY      SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015   22-NOV-19 03.46.09.745137 PM

SYS@DB03> select c.name,patch_id,version,action,status,description,action_time
  2  from cdb_registry_sqlpatch r, v$containers c
  3  where r.con_id=c.con_id
  4  order by r.con_id,action_time;

DB/PDB
NAME         PATCH_ID VERSION    ACTION     STATUS     DESCRIPTION                                        ACTION_TIME
---------- ---------- ---------- ---------- ---------- -------------------------------------------------- ------------------------------
CDB$ROOT     30133625 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    22-NOV-19 03.11.21.406332 PM
CDB$ROOT     30138470 12.2.0.1   APPLY      SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015   22-NOV-19 03.11.22.706324 PM
PDB03        30133625 12.2.0.1   APPLY      SUCCESS    OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)    22-NOV-19 03.11.24.022250 PM
PDB03        30138470 12.2.0.1   APPLY      SUCCESS    DATABASE OCT 2019 RELEASE UPDATE 12.2.0.1.191015   22-NOV-19 03.11.25.221969 PM

SQL files of patches 30886680 and 30805580 have been loaded into DB01 (both root container and pdb), but not into DB02 & DB03.

Run datapatch to load sql files,
[oracle@host01]$ export ORACLE_SID=DB02
[oracle@host01]$ export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_2
[oracle@host01]$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Fri Apr 24 17:36:42 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_44308_2020_04_24_17_36_42/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Patch 30133625 (OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)):
  Installed in the SQL registry only
Patch 30805580 (OJVM RELEASE UPDATE 12.2.0.1.200414):
  Installed in the binary registry only
Bundle series DBRU:
  ID 200414 in the binary registry and ID 191015 in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  The following patches will be rolled back:
    30133625 (OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625))
  The following patches will be applied:
    30805580 (OJVM RELEASE UPDATE 12.2.0.1.200414)
    30886680 (DATABASE APR 2020 RELEASE UPDATE 12.2.0.1.200414)

Installing patches...
Patch installation complete.  Total patches installed: 3

Validating logfiles...
Patch 30133625 rollback: SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30133625/23095898/30133625_apply_DB02_2020Apr24_17_37_31.log (no errors)
Patch 30805580 apply: SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30805580/23439360/30805580_apply_DB02_2020Apr24_17_37_31.log (no errors)
Patch 30886680 apply: SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30886680/23483391/30886680_apply_DB02_2020Apr24_17_37_31.log (no errors)
SQL Patching tool complete on Fri Apr 24 17:38:51 2020

[oracle@host01]$ export ORACLE_SID=DB03
[oracle@host01]$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Fri Apr 24 17:38:57 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_46606_2020_04_24_17_38_57/sqlpatch_invocation.log

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Patch 30133625 (OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625)):
  Installed in CDB$ROOT PDB03 PDB$SEED only
Patch 30805580 (OJVM RELEASE UPDATE 12.2.0.1.200414):
  Installed in the binary registry only
Bundle series DBRU:
  ID 200414 in the binary registry and ID 191015 in PDB CDB$ROOT, ID 191015 in PDB PDB$SEED, ID 191015 in PDB PDB03

Adding patches to installation queue and performing prereq checks...
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED PDB03
    The following patches will be rolled back:
      30133625 (OJVM RELEASE UPDATE: 12.2.0.1.191015 (30133625))
    The following patches will be applied:
      30805580 (OJVM RELEASE UPDATE 12.2.0.1.200414)
      30886680 (DATABASE APR 2020 RELEASE UPDATE 12.2.0.1.200414)

Installing patches...
Patch installation complete.  Total patches installed: 9

Validating logfiles...
Patch 30133625 rollback (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30133625/23095898/30133625_apply_DB03_CDBROOT_2020Apr24_17_40_03.log (no errors)
Patch 30805580 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30805580/23439360/30805580_apply_DB03_CDBROOT_2020Apr24_17_41_07.log (no errors)
Patch 30886680 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30886680/23483391/30886680_apply_DB03_CDBROOT_2020Apr24_17_41_07.log (no errors)
Patch 30133625 rollback (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30133625/23095898/30133625_apply_DB03_PDBSEED_2020Apr24_17_42_09.log (no errors)
Patch 30805580 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30805580/23439360/30805580_apply_DB03_PDBSEED_2020Apr24_17_42_25.log (no errors)
Patch 30886680 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30886680/23483391/30886680_apply_DB03_PDBSEED_2020Apr24_17_42_25.log (no errors)
Patch 30133625 rollback (pdb PDB03): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30133625/23095898/30133625_apply_DB03_PDB03_2020Apr24_17_42_09.log (no errors)
Patch 30805580 apply (pdb PDB03): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30805580/23439360/30805580_apply_DB03_PDB03_2020Apr24_17_42_25.log (no errors)
Patch 30886680 apply (pdb PDB03): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30886680/23483391/30886680_apply_DB03_PDB03_2020Apr24_17_42_25.log (no errors)
SQL Patching tool complete on Fri Apr 24 17:43:20 2020

Patch 25087834 is binary-only patch, it does not include SQL files. Therefore, no patch information is shown in the datapatch log.

So far, SIHA & SIDB have been successfully patched by opatchauto OOP. As shown before, opatchauto completed the OOP patching in two steps (prepare-clone and switch-clone), it is called two-step method. Actually, opatchauto can do OOP in one step with option outofplace, this is called single-session method.

9. OOP patching with single-session method

Single-session method create clone home and switch the original home to the clone home by one opatchauto command,

  opatchauto apply -phBaseDir <patch-base-directory> -outofplace -oh <ORACLE_HOME>

I prefer twp-step method, because the downtime is only needed at second step (swith-clone) which is usually done in minutes and minimize downtime. However, single-session method finishes everything in one step which takes much longer time and the downtime window is hardly determined and scheduled.
  
Following is an example of single-session method,
[root@host01]# ${ORACLE_HOME}/OPatch/opatchauto apply -phBaseDir /u01/stage/grid -outofplace -oh ${ORACLE_HOME} -logLevel FINEST

OPatchauto session is initiated at Fri Apr 24 15:27:48 2020

System initialization log file is /u01/app/grid/12.2.0/grid_1/cfgtoollogs/opatchautodb/systemconfig2020-04-24_03-27-56PM.log.

Session log file is /u01/app/grid/12.2.0/grid_1/cfgtoollogs/opatchauto/opatchauto2020-04-24_03-28-07PM.log
The id for this session is DSN1
Please press ENTER button to accept system generated default clone path. 
Please enter clone path [/u01/app/grid/12.2.0/grid_2 ] :


Executing OPatch prereq operations to verify patch applicability on home /u01/app/grid/12.2.0/grid_1
Patch applicability verified successfully on home /u01/app/grid/12.2.0/grid_1


Copying the files from the existing oracle home /u01/app/grid/12.2.0/grid_1 to a new location. Please wait...
Clone of oracle home /u01/app/grid/12.2.0/grid is /u01/app/grid/product/12.2.0/grid_2 on host host01
Copying the files from the existing oracle home /u01/app/grid/12.2.0/grid_1 to a new location is successful.


Unlocking CRS clone home for home /u01/app/grid/12.2.0/grid_1.
Prepatch operation log file location: /u01/app/grid/crsdata/host01/crsconfig/hapatch_2020-04-24_03-38-00PM.log
Unlocked CRS clone home successfully for home /u01/app/grid/12.2.0/grid_1.


Creating clone for oracle home /u01/app/grid/12.2.0/grid_1.
Clone operation successful for oracle home /u01/app/grid/12.2.0/grid_1.


Performing post clone operation for oracle home /u01/app/grid/12.2.0/grid_1.
Performing post clone operation was successful for oracle home /u01/app/grid/product/12.2.0/grid_2.


Start applying binary patch on home /u01/app/grid/product/12.2.0/grid_2
Binary patch applied successfully on home /u01/app/grid/product/12.2.0/grid_2


Update nodelist in the inventory for oracle home /u01/app/grid/product/12.2.0/grid_2.
Update nodelist in the inventory is completed for oracle home /u01/app/grid/product/12.2.0/grid_2. 


Starting CRS service on home /u01/app/grid/product/12.2.0/grid_2
Postpatch operation log file location: /u01/app/grid/crsdata/host01/crsconfig/hapatch_2020-04-24_04-01-41PM.log
CRS service started successfully on home /u01/app/grid/product/12.2.0/grid_2


Confirm that all resources have been started from home /u01/app/grid/product/12.2.0/grid_2.
All resources have been started successfully from home /u01/app/grid/product/12.2.0/grid_2.



OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:host01
SIHA Home:/u01/app/grid/12.2.0/grid_1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/grid/30920127/26839277
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/grid/30920127/30869447
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-24_15-39-45PM_1.log

Patch: /u01/stage/grid/30920127/30882603
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-24_15-39-45PM_1.log

Patch: /u01/stage/grid/30920127/30886680
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-24_15-39-45PM_1.log

Patch: /u01/stage/grid/30920127/30888810
Log: /u01/app/grid/product/12.2.0/grid_2/cfgtoollogs/opatchauto/core/opatch/opatch2020-04-24_15-39-45PM_1.log


Patching session reported following warning(s): 
_________________________________________________

[Note]: Please verify the database is running from the desired Oracle home, if not then manually execute 
 $ORACLE_HOME/bin/srvctl modify database command to fix the problem


Out of place patching clone home(s) summary
____________________________________________
Host : host01
Actual Home : /u01/app/grid/12.2.0/grid_1
Version:12.2.0.1.0
Clone Home Path : /u01/app/grid/product/12.2.0/grid_2


OPatchauto session completed at Fri Apr 24 16:03:20 2020
Time taken to complete the session 35 minutes, 32 seconds
[root@host01]# 
  

No comments: