Saturday, February 27, 2021

19c Upgrade - Fail with ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1260]

When 12.2 pluggable database is upgraded to 19c, dbupgrade fails with message
**********   Upgrading Java and non-Java   *********
Serial   Phase #:53   [PDB1] Files:2 
Unexpected error encountered in catctlMain; Error Stack Below; exiting
Died at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 9188.
 at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 9188.
main::catctlDie("\x{a}Unexpected error encountered in catconExec; exiting\x{a} No such"...) called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 5650
main::catctlExecutePhaseFiles(53, 2, "PDB1", "", undef) called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 2035
main::catctlRunPhase(53, 2, "PDB1", "", undef) called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 2197
main::catctlRunPhases(0, 108, 108, "PDB1", "", undef) called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 2816
main::catctlRunMainPhases() called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 1463
main::catctlMain() called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 1370
eval {...} called at /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catctl.pl line 1368

------------------------------------------------------
CATCTL FATAL ERROR
------------------------------------------------------

 LOG FILES: (/u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/DB01/upgrade20210222125535/catupgrdPDB1*.log)
TRACE FILE: (/u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/DB01/upgrade20210222125535/catupgrdPDB1_trace.log)

In log file catupgrdPDB1*.log
13:09:24 SQL> -- Load all the Java classes
13:09:24 SQL> begin if initjvmaux.startstep('CREATE_JAVA_SYSTEM') then
13:09:24   2  initjvmaux.rollbacksetup;
13:09:24   3  commit;
13:09:24   4  initjvmaux.rollbackset;
13:09:24   5  initjvmaux.exec('create or replace java system');
13:09:24   6  commit;
13:09:24   7  initjvmaux.rollbackcleanup;
13:09:24   8  initjvmaux.endstep;
13:09:24   9  dbms_registry.update_schema_list('JAVAVM',
13:09:24  10        dbms_registry.schema_list_t('OJVMSYS'));
13:09:24  11  end if; end;
13:09:24  12  /
begin if initjvmaux.startstep('CREATE_JAVA_SYSTEM') then
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
ERROR:
ORA-03114: not connected to ORACLE

In database alert log file
2021-02-22T13:09:41.671656-05:00
PDB1(4):Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4] [PC:0x118CE298C, joevm_invokevirtual()+1260] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/db01/DB01/trace/DB01_2_ora_4739.trc  (incident=28841) (PDBNAME=PDB1):
ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1260] [SIGSEGV] [ADDR:0x4] [PC:0x118CE298C] [Address not mapped to object] []
PDB1(4):Incident details in: /u01/app/oracle/diag/rdbms/db01/DB01/incident/incdir_28841/DB01_ora_4739_i28841.trc
PDB1(4):Use ADRCI or Support Workbench to package the incident.

The culprit is that the PDB (PDB1) was plugged from NONCDB which was followed by running noncdb_to_pdb.sql.  It can be fixed as following,

1. Connect to the database as SYS and run following sql commands in root container CDB$ROOT
alter session set container=CDB$ROOT;

alter session set "_ORACLE_SCRIPT"=true;

create or replace view rootobj sharing=object as
   select obj#,o.name,u.name uname,o.type#,o.flags
     from obj$ o,user$ u where owner#=user#;

2. Connect to the database as SYS and run following sql commands in PDB container PDB1 (your PDB name may be different)
alter session set container=PDB1;

create or replace view rootobj sharing=object as
   select obj#,o.name,u.name uname,o.type#,o.flags
     from obj$ o,user$ u where owner#=user#;

update obj$ set flags=flags+65536
 where type# in (28,29,30,56)
       and bitand(flags,65536)=0
       and obj# in (select o.obj# from obj$ o,user$ u,rootobj r
                     where o.name=r.name and o.type#=r.type# and o.owner#=u.user# and u.name=r.uname and bitand(r.flags,65536)!=0
                    union
                    select obj# from obj$ where bitand(flags,4259840)=4194304);

delete from sys.idl_ub1$
 where obj# in (select obj# from sys.obj$ where bitand(flags, 65536)=65536 and type# in (28,29,30,56));

commit;

3. Re-start upgrade from failed step (Phase #:53 in my case)
$ORACLE_HOME/bin/dbupgrade -p 53 -c 'PDB1'

2 comments:

Syed said...

Hi,
Thanks for posting where did you find the Fix, is there a known bug/Oracle article you can share if you have please share. Thanks

Syed said...

Never mind, I found it.

12.2 or higher Database Upgrade may fail with ORA-07445: exception encountered: core dump [joevm_invokevirtual()+1414] (Doc ID 2691097.1)