Saturday, November 14, 2020

Zero-Downtime patching Oracle Grid Infrastructure 19c on Linux

 In the past, when we were patching Grid Infrastructure (GI) in rolling mode, the database instances had to be shut down on the node which we were patching on. 

Starting with Oracle Database 19c Release Update (RU) 19.8, Oracle announced that Oracle RAC database instances can keep running and accessible to the database users during Oracle Grid Infrastructure patching. My patching practice is done in following environment

   * Two nodes Grid Infrastructure 19.8 running on Oracle Linux 7 update 8
   * Host name of nodes are rac01.lab.dbaplus.ca and rac02.lab.dbaplus.ca
   * GI RU 19.9.0.0.201020 and OJVM RU 19.9.0.0.201020 are applied in out-of-place(OOP) mode
   * ACFS and AFD are being used
   
1. Create new Oracle Grid Infrastructure (GI) 19c home and prepare Oracle GI Release Update (RU) 19.9 for patching

As the root user on all nodes, create directories for new GI home
[root@rac01]# mkdir -p /u01/app/19.9.0/grid
[root@rac01]# chown -R grid:oinstall /u01/app/19.9.0

As the grid user on first node, download the Oracle Grid Infrastructure 19c image file and extract the files into new created GI home
[grid@rac01]$ cd /u01/app/19.9.0/grid
[grid@rac01]$ unzip -q /u01/media/LINUX.X64_193000_grid_home.zip

As grid user on first node, download and install latest version of OPatch (12.2.0.1.21) into new GI home
[grid@rac01]$ cd /u01/app/19.9.0/grid
[grid@rac01]$ mv OPatch OPatch.old
[grid@rac01]$ unzip -q /u01/media/p6880880_122010_Linux-x86-64_12.2.0.1.21.zip

As grid user on first node, download Oracle GI RU 19.9.0.0.201020 and extract the files into stage/temporary directory
[grid@rac01]$ mkdir /u01/stage/RU
[grid@rac01]$ cd /u01/stage/RU
[grid@rac01]$ unzip -q /u01/media/p31750108_190000_Linux-x86-64.zip

As grid user on first node, download Oracle JavaVM Component Release Update (OJVM RU) 19.9.0.0.201020 and extract the files into stage/temporary directory
[grid@rac01]$ mkdir /u01/stage/OJVM
[grid@rac01]$ cd /u01/stage/OJVM
[grid@rac01]$ unzip -q /u01/media/p31668882_190000_Linux-x86-64.zip

2. Software-Only installing GI with RU applying

GI Software-Only installation is optional, it minimizes my waiting time while working on the patch during the maintenance window.

Starting with Oracle Grid Infrastructure 18c, you can download and apply Release Updates (RUs) and one-off patches during an Oracle GI installation or upgrade with following gridSetup.sh command options

   gridSetup.sh -applyRU <patch_directory_location> -applyOneOffs <comma_seperated_list_of_patch_directory_locations>

As grid user on first node, start GI installation and apply GI RU and OJVM RU (as one-off patch)
[grid@rac01]$ cd /u01/app/19.9.0/grid
[grid@rac01]$ ./gridSetup.sh -applyRU /u01/stage/RU/31750108 -applyOneOffs /u01/stage/OJVM/31668882
Preparing the home to patch...
Applying the patch /u01/stage/RU/31750108...
Successfully applied the patch.
Applying the patch /u01/stage/OJVM/31668882...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2020-11-04_11-33-11AM/installerPatchActions_2020-11-04_11-33-11AM.log
Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at:
 /u01/app/19.9.0/grid/install/response/grid_2020-11-04_11-33-11AM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2020-11-04_11-33-11AM/gridSetupActions2020-11-04_11-33-11AM.log

After successfully applied GI RU 19.9.0.0.201020 and OJVM RU 19.9.0.0.201020, gridSetup.sh will start installation graphic interface, respond to the prompts as following,

  * In the 'Select Configuration Option' screen, select the 'Set Up Software Only' option to perform a software-only installation of Oracle Grid Infrastructure for a standalone server. Click Next.
  * In the 'Cluster Node Information' screen, click Add button to add public host names of all cluster nodes(rac01.lab.dbaplus.ca & rac02.lab.dbaplus.ca). Click Next.
  * Respond to the prompts as needed to set up Oracle Grid Infrastructure
  * The Oracle Grid Infrastructure setup wizard prompts you to run the root.sh script [on each node].

Example of root.sh execution
[root@rac01]# /u01/app/19.9.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.9.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster execute the following command as grid user:
/u01/app/19.9.0/grid/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

The installer installed new GI on all nodes, and the new GI home has GI RU 19.9.0.201020 and OJVM RU 19.9.0.201020 installed.

3. Switching the GI Home without shutting down database instances

As grid user on first node, execute gridSetup.sh with following option to switch GI home

   gridSetup.sh -switchGridHome

If you did not perform Software-Only installation (step 2), run gridSetup.sh with following options to switch GI home

   gridSetup.sh -switchGridHome -applyRU <patch_directory_location> -applyOneOffs <comma_seperated_list_of_patch_directory_locations>

Switch GI home to new patched home
[grid@rac01]$ /u01/app/19.9.0/grid/gridSetup.sh -switchGridHome
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2020-11-05_11-05-00AM.log

Follow the steps in the configuration wizard to complete the Oracle Grid Infrastructure installation.

Note: During configuration, do NOT select the option to Automatically run configuration scripts.
   
When asked to run root.sh, running it as following with the -transparent and -nodriverupdate flags on the first node.

In order to watch process of Zero-Downtime patch, open two new terminals before execute root.sh as following

  * Terminal 1 (T1) Connect to first node rac01 as grid user, while root.sh is running on first node, keep running following commands to monitor Oracle running processes and database instances

      ps -ef | grep 'd.bin' | grep -v grep
      ps -ef | grep pmon | grep -v grep

    The script root.sh takes a while to complete, in order to keep running above commands, I created a script process.sh to repeat them every 2 seconds, the script can be terminated by press Ctrl + C, or wait for about one hour (should not be that long, it took me about 20 minutes). The source code of the script can be found at the bottom of this post.

  * Terminal 2 (T2) Connect to second node rac02 as grid user and log into ASM instance using sqlplus, while root.sh is running on first node, keep running following sql statements to monitor ASM clients connecting to ASM instance (+ASM2) running on second node rac02,

      select instance_name,count(*) from v$asm_client group by instance_name order by 1;

    I created another script asmclient.sh to run the sql every second.

3.1 Run root.sh on first node

On first node, as root user, 'run root.sh -transparent -nodriverupdate'
[root@rac01]# /u01/app/19.9.0/grid/root.sh -transparent -nodriverupdate
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.9.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19.8.0/grid/lib:/u01/app/19.9.0/grid/lib:'
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac01/crsconfig/rootcrs_rac01_2020-11-05_11-14-18AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac01/crsconfig/rootcrs_rac01_2020-11-05_11-14-18AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac01/crsconfig/crs_prepatch_apply_oop_rac01_2020-11-05_11-14-18AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac01/crsconfig/crs_prepatch_apply_oop_rac01_2020-11-05_11-14-18AM.log
2020/11/05 11:14:26 CLSRSC-347: Successfully unlock /u01/app/19.9.0/grid
2020/11/05 11:14:27 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac01/crsconfig/crs_postpatch_apply_oop_rac01_2020-11-05_11-14-27AM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [481622232].
2020/11/05 11:14:44 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2020/11/05 11:27:19 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [481622232].
2020/11/05 11:28:46 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/11/05 11:28:47 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

On T1, start process.sh while/before starting root.sh on first node, the script will keep running for about one hour unless you press ctrl+c. The output as following
================== Before root.sh started =====================
root     27131     1  0 Sep10 ?        05:56:43 /u01/app/19.8.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
grid     27478     1  0 Sep10 ?        01:53:53 /u01/app/19.8.0/grid/bin/mdnsd.bin
grid     27480     1  0 Sep10 ?        04:55:13 /u01/app/19.8.0/grid/bin/evmd.bin
grid     27520     1  0 Sep10 ?        02:09:43 /u01/app/19.8.0/grid/bin/gpnpd.bin
grid     27584     1  0 Sep10 ?        05:54:32 /u01/app/19.8.0/grid/bin/gipcd.bin
root     27640     1  4 Sep10 ?        2-16:05:32 /u01/app/19.8.0/grid/bin/osysmond.bin
grid     27671     1  0 Sep10 ?        06:40:33 /u01/app/19.8.0/grid/bin/ocssd.bin
root     28227     1  0 Sep10 ?        04:58:39 /u01/app/19.8.0/grid/bin/octssd.bin reboot
root     28296     1  0 Sep10 ?        05:15:10 /u01/app/19.8.0/grid/bin/crsd.bin reboot

oracle     902     1  0 Sep25 ?        00:06:07 ora_pmon_DB19C01_2
oracle   19865     1  0 Sep17 ?        00:07:23 ora_pmon_DB19C02_2
oracle   22872     1  0 Sep10 ?        00:07:45 ora_pmon_DB12C01_2
grid     28754     1  0 Sep10 ?        00:04:27 asm_pmon_+ASM1
grid     28997     1  0 Sep10 ?        00:03:55 apx_pmon_+APX1
grid     30193     1  0 Sep10 ?        00:04:03 mdb_pmon_-MGMTDB

================== After root.sh started and ran a while =====================
root     21092     1  0 11:27 ?        00:00:00 /u01/app/grid/crsdata/rac01/csswd/oracsswd.bin
-----------------------------------------------
oracle     902     1  0 Sep25 ?        00:06:07 ora_pmon_DB19C01_2
oracle   19865     1  0 Sep17 ?        00:07:23 ora_pmon_DB19C02_2

================== After root.sh completed running =====================
root     21586     1  5 11:27 ?        00:00:03 /u01/app/19.9.0/grid/bin/ohasd.bin reboot CRS_AUX_DATA=CRS_AUXD_TGIP=yes;_ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
grid     21764     1  0 11:27 ?        00:00:00 /u01/app/19.9.0/grid/bin/gpnpd.bin
grid     21766     1  0 11:27 ?        00:00:00 /u01/app/19.9.0/grid/bin/mdnsd.bin
grid     21768     1  2 11:27 ?        00:00:01 /u01/app/19.9.0/grid/bin/evmd.bin
grid     21945     1  1 11:27 ?        00:00:00 /u01/app/19.9.0/grid/bin/gipcd.bin
grid     22064     1  4 11:27 ?        00:00:02 /u01/app/19.9.0/grid/bin/ocssd.bin -P
root     22227     1  0 11:27 ?        00:00:00 /u01/app/19.9.0/grid/bin/octssd.bin reboot
root     22300     1  8 11:27 ?        00:00:03 /u01/app/19.9.0/grid/bin/crsd.bin reboot
root     22319     1  4 11:27 ?        00:00:01 /u01/app/19.9.0/grid/bin/osysmond.bin
-----------------------------------------------
oracle     902     1  0 Sep25 ?        00:06:07 ora_pmon_DB19C01_2
oracle   19865     1  0 Sep17 ?        00:07:23 ora_pmon_DB19C02_2
grid     22748     1  0 11:27 ?        00:00:00 apx_pmon_+APX1
grid     23220     1  0 11:28 ?        00:00:00 asm_pmon_+ASM1
oracle   23546     1  0 11:28 ?        00:00:00 ora_pmon_DB12C01_2

The root.sh shuts down old version database instance (DB12C01 is version 12.2.0.1), ASM instance and crs stack running from old GI home, and starts a watchdog process (oracsswd.bin) as dummy crs service from crsdata directory (neither old GI home nor new GI home).  Then, brings up crs stack from new GI home and shuts down watchdog process. At last, ASM instance and all databases instances are brought up.

How the running 19c database instances access ASM storage while ASM instance is down on local node? The answer is that ASM instance on remote node will help.

On T2, start asmclient.sh while/before starting root.sh on first node, the script will keep running for about one hour unless you press ctrl+c. The output as following
================== Before root.sh started on first node =====================
  +APX2         1
  +ASM2         1
  DB19C01_1     4
  DB12C01_1     4
  DB19C02_1     4
  rac02.lab.dbaplus.ca      1

================== After root.sh started and ASM instance is being shut down on first node =====================
  +APX2         1
  +ASM2         1
  DB19C01_1     4
  DB19C01_2     4
  DB12C01_1     4
  DB19C02_1     4
  DB19C02_2     4
  rac02.lab.dbaplus.ca      1

When ASM instance (+ASM1) is being shut down on first node (rac01), the 19c database instances running on this node (DB19C01_2, DB19C02_2) will be redirected to ASM instance (+ASM2) running on remote node (rac02). That's why database instances can still access ASM storage while the local ASM instance is down.

3.2 Run root.sh on second node (other nodes for more than two nodes cluster)

On second node, as root user, run 'root.sh -transparent -nodriverupdate'
[root@rac02]# /u01/app/19.9.0/grid/root.sh -transparent -nodriverupdate
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19.9.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19.0.0/grid/lib:/u01/app/19.9.0/grid/lib:'
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/rootcrs_rac02_2020-11-05_12-33-56AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/rootcrs_rac02_2020-11-05_12-33-56AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_12-33-57AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_12-33-57AM.log
2020/11/05 12:34:03 CLSRSC-347: Successfully unlock /u01/app/19.9.0/grid
2020/11/05 12:34:04 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/crs_postpatch_apply_oop_rac02_2020-11-05_12-34-04AM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [481622232].
2020/11/05 12:34:17 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2020/11/05 12:47:01 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2862754030].
SQL Patching tool version 19.9.0.0.0 Production on Thu Nov  5 12:49:37 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/grid/cfgtoollogs/sqlpatch/sqlpatch_21650_2020_11_05_12_49_37/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of interim SQL patches:
Interim patch 31219897 (OJVM RELEASE UPDATE: 19.8.0.0.200714 (31219897)):
  Binary registry: Not installed
  PDB CDB$ROOT: Applied successfully on 10-SEP-20 07.05.22.344726 PM
  PDB GIMR_DSCREP_10: Applied successfully on 10-SEP-20 07.05.26.197751 PM
  PDB PDB$SEED: Applied successfully on 10-SEP-20 07.05.24.262950 PM
Interim patch 31668882 (OJVM RELEASE UPDATE: 19.9.0.0.201020 (31668882)):
  Binary registry: Installed
  PDB CDB$ROOT: Not installed
  PDB GIMR_DSCREP_10: Not installed
  PDB PDB$SEED: Not installed

Current state of release update SQL patches:
  Binary registry:
    19.9.0.0.0 Release_Update 200930183249: Installed
  PDB CDB$ROOT:
    Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 10-SEP-20 07.05.22.342186 PM
  PDB GIMR_DSCREP_10:
    Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 10-SEP-20 07.05.26.195906 PM
  PDB PDB$SEED:
    Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 10-SEP-20 07.05.24.260687 PM

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED GIMR_DSCREP_10
    The following interim patches will be rolled back:
      31219897 (OJVM RELEASE UPDATE: 19.8.0.0.200714 (31219897))
    Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
      Apply from 19.8.0.0.0 Release_Update 200703031501 to 19.9.0.0.0 Release_Update 200930183249
    The following interim patches will be applied:
      31668882 (OJVM RELEASE UPDATE: 19.9.0.0.201020 (31668882))

Installing patches...
Patch installation complete.  Total patches installed: 9

Validating logfiles...done
Patch 31219897 rollback (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31219897/23619699/31219897_rollback__MGMTDB_CDBROOT_2020Nov05_12_50_01.log (no errors)
Patch 31771877 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply__MGMTDB_CDBROOT_2020Nov05_12_50_01.log (no errors)
Patch 31668882 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31668882/23790068/31668882_apply__MGMTDB_CDBROOT_2020Nov05_12_50_01.log (no errors)
Patch 31219897 rollback (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31219897/23619699/31219897_rollback__MGMTDB_PDBSEED_2020Nov05_12_50_40.log (no errors)
Patch 31771877 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply__MGMTDB_PDBSEED_2020Nov05_12_50_40.log (no errors)
Patch 31668882 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31668882/23790068/31668882_apply__MGMTDB_PDBSEED_2020Nov05_12_50_40.log (no errors)
Patch 31219897 rollback (pdb GIMR_DSCREP_10): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31219897/23619699/31219897_rollback__MGMTDB_GIMR_DSCREP_10_2020Nov05_12_50_39.log (no errors)
Patch 31771877 apply (pdb GIMR_DSCREP_10): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply__MGMTDB_GIMR_DSCREP_10_2020Nov05_12_50_40.log (no errors)
Patch 31668882 apply (pdb GIMR_DSCREP_10): SUCCESS
  logfile: /u01/app/grid/cfgtoollogs/sqlpatch/31668882/23790068/31668882_apply__MGMTDB_GIMR_DSCREP_10_2020Nov05_12_50_40.log (no errors)
SQL Patching tool complete on Thu Nov  5 12:51:05 2020
2020/11/05 12:51:31 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/11/05 12:51:33 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

After root.sh ran successfully on all nodes, all Oracle Grid Infrastructure services start running from the new Grid home. 

Zero-downtime GI patching is completed successfully if no ACFS file system is created and ASM storage is not using AFD. Unfortunately, they both are used and next step is required.

4. Update operating system drivers for ACFS and/or AFD

As grid user, validate the status of Oracle drivers (ACFS & AFD) by 'crsctl query driver'
[grid@rac02]$ /u01/app/19.9.0/grid/bin/crsctl query driver activeversion -all
Node Name : rac01
Driver Name : ACFS
BuildNumber : 200626
BuildVersion : 19.0.0.0.0 (19.8.0.0.0)

Node Name : rac01
Driver Name : AFD
BuildNumber : 200626
BuildVersion : 19.0.0.0.0 (19.8.0.0.0)

Node Name : rac02
Driver Name : ACFS
BuildNumber : 200626
BuildVersion : 19.0.0.0.0 (19.8.0.0.0)

Node Name : rac02
Driver Name : AFD
BuildNumber : 200626
BuildVersion : 19.0.0.0.0 (19.8.0.0.0)

[grid@rac02]$ /u01/app/19.9.0/grid/bin/crsctl query driver softwareversion -all
Node Name : rac01
Driver Name : ACFS
BuildNumber : 200813.1
BuildVersion : 19.0.0.0.0 (19.9.0.0.0)

Node Name : rac01
Driver Name : AFD
BuildNumber : 200813.1
BuildVersion : 19.0.0.0.0 (19.9.0.0.0)

Node Name : rac02
Driver Name : ACFS
BuildNumber : 200813.1
BuildVersion : 19.0.0.0.0 (19.9.0.0.0)

Node Name : rac02
Driver Name : AFD
BuildNumber : 200813.1
BuildVersion : 19.0.0.0.0 (19.9.0.0.0)

ACFS & AFD drivers are installed and used, the driver are updated to new version on both nodes, but active driver is still old version. At this time, new version driver cannot be activated because '-nodriverupdate' is used with root.sh which made root.sh skip updating operating system drivers for ACFS & AFD.

In order to activate new version of driver, run 'rootcrs.sh -updateosfiles' on each cluster node and restart the cluster nodes. Oops, where is Zero-downtime? Anyway, at least, we can postpone this downtime to when OS maintenance is planned. Or it may be better not to user ACFS & AFD.

Source code of process.sh
#!/bin/bash
for i in {1..1800}
do
  echo 
  echo "================== `date` ====================="
  ps -ef | grep 'd\.bin' | grep -v grep
  echo "-----------------------------------------------"
  ps -ef | grep pmon | grep -v grep
  sleep 2
done

Source code of asmclient.sh
#!/bin/bash
F_TMP=`ps -ef | grep asm_pmon_+ASM | grep -v grep`
F_PID=`echo $F_TMP | awk -F' ' '{print $2}'`
export ORACLE_SID=`echo $F_TMP | awk -F'_' '{ print $NF }'`
export ORACLE_HOME=`pwdx $F_PID | awk -F' ' '{print $NF}' | sed 's/\/dbs\/*$//'`
for i in {1..1800}
do
echo "
  ================== `date` ====================="
  $ORACLE_HOME/bin/sqlplus -S / as sysdba <<EOF
     set head off
     set feed off
     set pagesize 999
     col instance_name for a25
     select instance_name,count(*) from v\$asm_client group by instance_name order by 1;
EOF
  sleep 1
done

No comments: