Upgrading Grid Infrastructure from 12.1 to 12.2 failed with [INS-20802]

Customer is upgrading two-node Oracle Grid Infrastructure from version 12.1.0.2 to version 12.2.0.1, and gridSetup.sh returned error [INS-20802] during Preparing for deploying Oracle Grid Infrastructure Management Repository.

It seems that Installer (gridSetup.sh) got error while dropping Grid Infrastructure Management Repository database (GIMR DB). Check the status of GIMR DB instance (-MGMTDB),

[grid@host01]$ . oraenv
ORACLE_SID = [grid] ? -MGMTDB
The Oracle base remains unchanged with value /u01/app/grid

[grid@host01]$ srvctl status mgmtdb
Database is disabled
Database is not running.

[grid@host01]$ srvctl config mgmtdb
Database unique name: _mgmtdb
Database name:
Oracle home: /u01/app/12.1.0/grid
Oracle user: grid
Spfile: +DATA/_MGMTDB/PARAMETERFILE/spfile.268.918488265
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Type: Management
PDB name: cluster01
PDB service: cluster01
Cluster name: cluster01
Database instance: -MGMTDB

Database is configured in 12.1 home (old version) and not running. As we know, the database is usually expected to be open when dropped by Oracle Utility (e.g. dbca). As common sense/best guess, -MGMTDB should also be open when dropped by Oracle Installer. However, The alert log of -MGMTDB does not have any messages created in last 7 days. The database has not been started since 7 days before. Confirmed with customer and got that a failed same GI upgrade (12.1.0.2 to 12.2.0.1) was rolled back to 12.1.0.2 7 days ago, and DBA was not aware of stop of -MGMTDB database. Try to manually start the database,

[grid@host01]$ srvctl start mgmtdb
PRCR-1079 : Failed to start resource ora.mgmtdb
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00101: unknown parameter name '__inmemory_ext_roarea'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/host02/crs/trace/crsd_oraagent_grid.trc".

CRS-2674: Start of 'ora.mgmtdb' on 'host02' failed
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00101: unknown parameter name '__inmemory_ext_roarea'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/host01/crs/trace/crsd_oraagent_grid.trc".

CRS-2674: Start of 'ora.mgmtdb' on 'host01' failed
CRS-2632: There are no more servers to try to place resource 'ora.mgmtdb' on that would satisfy its placement policy

The database can be started on neither node of the cluster because of unknown parameter '__inmemory_ext_roarea'. Actually, the parameter is introduced by Oracle database 12.2.0.1, and should not exist in 12.1.0.2 database. Most likely, the parameter was left there by last rollback of failed GI upgrade because of bug 26695380. Therefore, we are going to remove the 12.2.0.1 parameters from spfile.

Find out current spfile of -MGMTDB,

[grid@host01]$ srvctl config mgmtdb | grep Spfile
Spfile: +DATA/_MGMTDB/PARAMETERFILE/spfile.268.918488265

Create pfile from spfile,

SQL> create pfile='/tmp/mgmt.ora' from spfile='+DATA/_MGMTDB/PARAMETERFILE/spfile.268.918488265';

Confirm that the parameters do exist in pfile,

[grid@host01]$ grep '__inmemory_ext' /tmp/mgmt.ora
-MGMTDB.__inmemory_ext_roarea=0
-MGMTDB.__inmemory_ext_rwarea=0

Both __inmemory_ext* parameters are involved in bug 26695380. Therefore, remove all of them from pfile with text editor(e.g. vi) and create new spfile from pfile,

SQL> create spfile from pfile='/tmp/mgmt.ora';

Get new spfile name and start the database,

[grid@host01]$ srvctl config mgmtdb | grep Spfile
Spfile: +DATA/_MGMTDB/PARAMETERFILE/spfile.464.965927697

[grid@host01]$ srvctl start mgmtdb
[grid@host01]$ srvctl status mgmtdb
Database is disabled
Instance -MGMTDB is running on node host02

Click 'Retry' on Oracle Installer, it failed with same message. Then 'Retry' after shutting down the database and same issue. Looks like Oracle Installer does not take the new spfile. Anyway, __inmemory_ext* parameters are still the suspects. Find out the log files including the parameters,

[grid@host01]$ cd $ORACLE_HOME/cfgtoollogs
[grid@host01]$ pwd
/u01/app/12.2.0/grid/cfgtoollogs

[grid@host01]$ grep -l '__inmemory_ext' *.log
configTools_2018-01-20_01-19-13PM.log

[grid@host01] cat configTools_2018-01-20_01-19-13PM.log
 <<Message truncated>>
LD_LIBRARY_PATH=/u01/app/12.2.0/grid/lib:/u01/app/12.2.0/grid/oui/lib/aix:/u01/app/12.2.0/grid/bin
Starting the Grid Infrastructure Management Repository database
Using PFILE='/u01/app/12.1.0/grid/dbs/init-dropmgmtdb.ora'
Command output:
SQL*Plus: Release 12.2.0.1.0 Production on Sat Jan 20 16:41:16 2018 Copyright (c) 1982, 2017, Oracle. All rights reserved.
Connected to an idle instance.
SQL> ORA-01078: failure in processing system parameters
LRM-00101: unknown parameter name '__inmemory_ext_roarea' ORA-01078: failure in processing system parameters
SQL> Disconnected Starting the Grid Infrastructure Management Repository database succeeded
 <<Message truncated>>

Oracle Installer starts -MGMTDB using PFILE '/u01/app/12.1.0/grid/dbs/init-dropmgmtdb.ora', which is still pointing to old spfile,

[grid@host01] cat /u01/app/12.1.0/grid/dbs/init-dropmgmtdb.ora
SPFILE='+DATA/_MGMTDB/PARAMETERFILE/spfile.268.918488265'

Re-point to new spfile,

[grid@host01] echo "SPFILE='+DATA/_MGMTDB/PARAMETERFILE/spfile.464.965927697'" > /u01/app/12.1.0/grid/dbs/init-dropmgmtdb.ora
[grid@host01] cat /u01/app/12.1.0/grid/dbs/init-dropmgmtdb.ora
SPFILE='+DATA/_MGMTDB/PARAMETERFILE/spfile.464.965927697'

Click 'Retry' on Oracle Installer, the issue is gone!