Showing posts with label Grid Infrastructure & ASM. Show all posts
Showing posts with label Grid Infrastructure & ASM. Show all posts

Sunday, May 16, 2021

Grid Infrastructure 12.2 restore OCR, Voting File and mgmtdb when disk group corrupted

For some reason, Oracle desupported the placement of OCR and voting files directly on a shared file system from Grid Infrastructure 12.2 until rescinding the desupport for Standalone Clusters from 19.3. Therefore, when GI 12.2 is installed, OCR, Voting file, and OCR backup location are, by default, configured in the ASM disk group, and mostly likely in the same disk group. When the disk group cannot be mounted because of any issues, the cluster will not be brought up anymore.

Technically, GI has to be re-configured as like a new installation, that could be a big job. However, it may not be that bad. Although 12.2 does not allow you have OCR backup location out of ASM disk group, DBA still can copy the OCR backup file from ASM disk group to file system with command 'asmcmd cp'. I am going to demonstrate how we can maximize the opportunity of bringing the cluster backup without rebuild/reinstall it.

Monday, March 1, 2021

19c runcluvfy.sh faile with PRVF-7596 PRVG-2002

When upgrading Oracle Infrastructure 12.2 to 19c, runcluvfy.sh failed with following messages
Verifying OCR Integrity ...FAILED
host02: PRVF-7596 : CSS is probably working with a non-clustered, local-only
          configuration on node "host02"
Verifying resolv.conf Integrity ...FAILED
host02: PRVG-2002 : Encountered error in copying file "/etc/resolv.conf" from
          node "host02" to node "host01"
          protocol error: filename does not match request
Verifying DNS/NIS name service ...FAILED
host02: PRVG-2002 : Encountered error in copying file "/etc/netsvc.conf" from
          node "host02" to node "host01"
          protocol error: filename does not match request

Set log location and re-run runcluvfy.sh
# runcluvfy.sh will write log files to directory /u01/tmp
export CV_TRACELOC=/u01/tmp
# re-run runcluvfy.sh
./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.2.0/grid_1 -dest_crshome /u01/app/19.0.0/grid_1 -dest_version 19.0.0.0.0 -fixup -verbose
# Log files created
cd /u01/tmp
ls -l
total 15200
-rw-r--r--    1 grid     oinstall     952467 Feb 24 16:52 cvuhelper.log.0
-rw-r--r--    1 grid     oinstall          0 Feb 24 16:48 cvuhelper.log.0.lck
-rw-r--r--    1 grid     oinstall    6824223 Feb 24 16:52 cvutrace.log.0

The log file cvutrace.log.0 shows messsage
[Worker 0] [ 2021-02-24 16:47:49.784 EST ] [UnixSystem.remoteCopyFile:848]  UnixSystem: /usr/bin/scp -p host02:'/tmp/CVU_19.0.0.0.0_grid/scratch/getFileInfo3605304.out' /tmp/host02.getFileInfo3605304.out
[Worker 0] [ 2021-02-24 16:47:49.798 EST ] [RuntimeExec.runCommand:294]  runCommand: Waiting for the process
[Thread-83] [ 2021-02-24 16:47:49.798 EST ] [StreamReader.run:62]  In StreamReader.run
[Thread-82] [ 2021-02-24 16:47:49.798 EST ] [StreamReader.run:62]  In StreamReader.run
[Thread-83] [ 2021-02-24 16:47:50.201 EST ] [StreamReader.run:66]  ERROR>protocol error: filename does not match request
...
[main] [ 2021-02-24 16:51:46.747 EST ] [UnixSystem.remoteCopyFile:848]  UnixSystem: /usr/bin/scp -p host02:'/etc/resolv.conf' /tmp/CVU_19.0.0.0.0_grid/scratch/resolv.conf_host02
[main] [ 2021-02-24 16:51:46.760 EST ] [RuntimeExec.runCommand:294]  runCommand: Waiting for the process
[Thread-1060] [ 2021-02-24 16:51:46.760 EST ] [StreamReader.run:62]  In StreamReader.run
[Thread-1059] [ 2021-02-24 16:51:46.760 EST ] [StreamReader.run:62]  In StreamReader.run
[main] [ 2021-02-24 16:51:47.148 EST ] [RuntimeExec.runCommand:296]  runCommand: process returns 1
[Thread-1060] [ 2021-02-24 16:51:47.149 EST ] [StreamReader.run:66]  ERROR>protocol error: filename does not match request
...
[main] [ 2021-02-24 16:51:47.548 EST ] [UnixSystem.remoteCopyFile:848]  UnixSystem: /usr/bin/scp -p host02:'/etc/netsvc.conf' /tmp/CVU_19.0.0.0.0_grid/scratch/nssw_conf_host02
[Thread-1066] [ 2021-02-24 16:51:47.559 EST ] [StreamReader.run:62]  In StreamReader.run
[main] [ 2021-02-24 16:51:47.559 EST ] [RuntimeExec.runCommand:294]  runCommand: Waiting for the process
[Thread-1065] [ 2021-02-24 16:51:47.560 EST ] [StreamReader.run:62]  In StreamReader.run
[main] [ 2021-02-24 16:51:47.965 EST ] [RuntimeExec.runCommand:296]  runCommand: process returns 1
[Thread-1066] [ 2021-02-24 16:51:47.973 EST ] [StreamReader.run:66]  ERROR>protocol error: filename does not match request

All three errors occurred when scp is executed to copy file from remote node host02 to local node.
Interesting thing is that all three commands are using single quotation marks to enclose remote file name, I guess Oracle programmer left them there by accident because there is no reason to use them. Let's verify if it is the culprit by manually running scp
[grid@host01]$ scp -p host02:"'/etc/resolv.conf'" /tmp/CVU_19.0.0.0.0_grid/scratch/resolv.conf_host02
protocol error: filename does not match request
[grid@host01]$
[grid@host01]$ scp -T -p host02:"'/etc/resolv.conf'" /tmp/CVU_19.0.0.0.0_grid/scratch/resolv.conf_host02
resolv.conf                                             96    51.3KB/s   00:00
[grid@host01]$

The scp fails with same error "protocol error: filename does not match request", but succeeds if extra option -T is used.

The -T option was introduced by OpenSSH 8.0 released in April 2019. In earlier version of OpenSSH, when copying files from a remote system to a local directory, scp did not verify that the file names that the server sent matched those requested by the client. This could allow a hostile server to create or clobber unexpected local files with attacker-controlled content. OpenSSH 8.0 fixed this security issue and scp, by default, verifies the file name on client side, and also introduced -T option to provide capacity to disable the verification.

Although OpenSSH officially claims that the fix is introduced in 8.0, runcluvfy.sh shipped with Grid Infrastructure 19.3 also fails with same reason on AIX with OpenSSH 7.5p1 and it is where the errors used in this article happened.

This version of runcluvfy.sh sends remote file name with single quotation marks, but remote server returns file name without quotation. Technically, they are same thing, but they are visually different. Therefore, old version scp worked because it did not verify them, but current scp fails it with "filename does not match".

It can be resolved by downloading new standalone version Cluster Verification Utility from My Oracle Support(Patch 30839369). Note: the executable is cluvfy instead of runcluvfy.sh. If download is not the option. As a temporary workaround, we can rename scp and create a new scp
# Rename the original scp
mv /usr/bin/scp /usr/bin/scp.bak
# Create a new file scp
echo "/usr/bin/scp.orig -T $*" > /usr/bin/scp
# Make the file executable
chmod a+rx /usr/bin/scp

Now, the errors are gone. After successfully installing GI, remember to restore original scp
# Delete interim scp
rm /usr/bin/scp
# Restore the original scp.
mv /usr/bin/scp.bak /usr/bin/scp

Sunday, February 28, 2021

19c gridSetup.sh failed with [INS-06006] Passwordless SSH connectivity not set up between the following nodes

When running 19.3 gridSetup.sh to configure Grid Infrastructure, following error occurs at configuring passwordless SSH connectivity between cluster nodes
[INS-06006] Passwordless SSH connectivity not set up between the following node(s): [host02].

Cause - Either passwordless SSH connectivity is not setup between specified node(s) or they are not reachable. Refer to the logs for more details.

Action - Refer to the logs for more details or contact Oracle Support Services.

More Details
PRVF-5311 : File "/tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out" either does not exist or is not accessible on node "host02". 

Manually test passwordless SSH connectivity between nodes
[grid@host01]$ ssh host02       <=  Connect to second node host02 from first node host01
[grid@host02]$                  <=  Connected to host02 successfully without password

[grid@host02]$ ssh host02       <=  Connect to first node host01 from second node host02
[grid@host01]$                  <=  Connected to host01 successfully without password

Apparently, the passwordless SSH connectivity has been configured properly. Re-run gridSetup.sh in debug mode to figure out what happened,
grid@host01]$ /u01/app/19.0.0/grid _1/gridSetup.sh -debug

Launching Oracle Grid Infrastructure Setup Wizard...

[main] [ 2021-02-24 17:29:53.105 EST ] [Version.isPre:757]  version to be checked 19.0.0.0.0 major version to check against 10
[main] [ 2021-02-24 17:29:53.106 EST ] [Version.isPre:768]  isPre.java: Returning FALSE

  <<Message truncated>>

[Worker 2] [ 2021-02-24 17:31:26.521 EST ] [Utils.getLocalHost:487]  Hostname retrieved: host01, returned: host01
[Worker 2] [ 2021-02-24 17:31:26.521 EST ] [Utils.getLocalHost:487]  Hostname retrieved: host01, returned: host01
[Worker 2] [ 2021-02-24 17:31:26.521 EST ] [UnixSystem.remoteCopyFile:848]  UnixSystem: /usr/bin/scp -p host02:'/tmp/GridSetupActions2021-02-24_05-29-41PM/CVU_19.0.0.0.0_grid/scratch/getFileInfo1638946.out' /tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out
[Worker 2] [ 2021-02-24 17:31:26.534 EST ] [RuntimeExec.runCommand:294]  runCommand: Waiting for the process
[Thread-442] [ 2021-02-24 17:31:26.534 EST ] [StreamReader.run:62]  In StreamReader.run 
[Thread-443] [ 2021-02-24 17:31:26.534 EST ] [StreamReader.run:62]  In StreamReader.run 
[Thread-443] [ 2021-02-24 17:31:26.855 EST ] [StreamReader.run:66]  ERROR>protocol error: filename does not match request
[Worker 2] [ 2021-02-24 17:31:26.855 EST ] [RuntimeExec.runCommand:296]  runCommand: process returns 1
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.runCommand:323]  RunTimeExec: error>
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.runCommand:326]  protocol error: filename does not match request
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.traceCmdEnv:516]  Calling Runtime.exec() with the command 
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.traceCmdEnv:518]  /usr/bin/scp 
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.traceCmdEnv:518]  -p 
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.traceCmdEnv:518]  host02:'/tmp/GridSetupActions2021-02-24_05-29-41PM/CVU_19.0.0.0.0_grid/scratch/getFileInfo1638946.out' 
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.traceCmdEnv:518]  /tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out 
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [RuntimeExec.runCommand:349]  Returning from RunTimeExec.runCommand
[Worker 2] [ 2021-02-24 17:31:26.856 EST ] [NativeSystem.rununixcmd:1345]  NativeSystem.rununixcmd: RetString 0|protocol error: filename does not match request :failed
[Worker 2] [ 2021-02-24 17:31:26.859 EST ] [ClusterConfig$ExecuteCommand.returnCommandToClient:3324]  returnCommandToClient; fillCount=1 is full=false
[Worker 2] [ 2021-02-24 17:31:26.859 EST ] [Semaphore.release:88]  SyncBufferFull:Release called by thread Worker 2 m_count=2

When validating/configuring passwordless SSH connectivity, it has to copy SSH key information between nodes using scp. From the trace we can find that, it uses scp with -p option to copy the files, and destination file on local server (host01) is sent to scp with file name:

  /tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out
  
And source file on remote server(host02) is sent to scp with file name:

  host02:'/tmp/GridSetupActions2021-02-24_05-29-41PM/CVU_19.0.0.0.0_grid/scratch/getFileInfo1638946.out'

Interesting thing is that the whole file path includes single quotation marks, I guess Oracle programmer left them there by accident because there is no reason to use them. Let's verify if it is the culprit by manually running scp
[grid@host01]$ scp -p host02:"'/tmp/GridSetupActions2021-02-24_05-29-41PM/CVU_19.0.0.0.0_grid/scratch/getFileInfo1638946.out'" /tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out
protocol error: filename does not match request
[grid@host01]$
[grid@host01]$ scp -T -p host02:"'/tmp/GridSetupActions2021-02-24_05-29-41PM/CVU_19.0.0.0.0_grid/scratch/getFileInfo1638946.out'" /tmp/GridSetupActions2021-02-24_05-29-41PM/host02.getFileInfo1638946.out
getFileInfo1638946.out                                             98    56.5KB/s   00:00
[grid@host01]$

The scp fails with same error "protocol error: filename does not match request", but succeeds if extra option -T is used.

The -T option was introduced by OpenSSH 8.0 released in April 2019. In earlier version of OpenSSH, when copying files from a remote system to a local directory, scp did not verify that the file names that the server sent matched those requested by the client. This could allow a hostile server to create or clobber unexpected local files with attacker-controlled content. OpenSSH 8.0 fixed this security issue and scp, by default, verifies the file name on client side, and also introduced -T option to provide capacity to disable the verification.

Although OpenSSH officially claims that the fix is introduced in 8.0, gridSetup.sh 19.3 also fails with same reason on AIX with OpenSSH 7.5p1 and it is where the errors used in this article happened.

Oracle gridSetup.sh 19.3 (base pubic release of 19c) sends remote file name with single quotation marks, but remote server returns file name without quotation. Techinally, they are same thing, but they are visually different. Therefore, old version scp worked because it did not verify them, but current scp fails it with "filename does not match".

Oracle Release Update 19.6 fixed this problem by removing the single quotation marks. Therefore, gridSetup.sh can be run successfully with -applyRU option to apply 19.6 or higher Realease Update before installing/configuring GI
[grid@r6-dart]$ ./gridSetup.sh -applyRU /u01/stage/30501910
Preparing the home to patch...
Applying the patch /u01/stage/grid/30501910...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2021-02-24_03-25-57PM/installerPatchActions_2021-02-24_03-25-57PM.log
...

Here, Oracle Grid Infrastructure Release Update 19.6 (Patch 30501910) is unzipped under directory /u01/stage/30501910.

If 19.3 is really needed for some reason, as a temporary workaround, we can rename scp and create a new scp
# Rename the original scp
mv /usr/bin/scp /usr/bin/scp.bak

# Create a new file scp
echo "/usr/bin/scp.orig -T $*" > /usr/bin/scp

# Make the file executable
chmod a+rx /usr/bin/scp

After successfully installing GI 19.3, remember restore original scp
# Delete interim scp
rm /usr/bin/scp

# Restore the original scp.
mv /usr/bin/scp.bak /usr/bin/scp

Tuesday, December 1, 2020

Create Oracle ASM disks using multipath devices and udev rules on Oracle Linux / Red Hat Enterprise Linux

 Since Oracle Automatic Storage Management Filter Driver (ASMFD) was released with Oracle Database 12c, DBA has very few chance to work with udev rules when creating ASM disks. 

Fortunately, Oracle Database 19.6 introduced a new feature, Zero-Downtime Grid Infrastructure patching which brings the requirement of udev rules back to DBA. When Oracle ASMFD is used for ASM disks, Oracle patching has to update the operating system driver for ASMFD, which cannot be done in zero-downtime mode. Therefore, in order to patch Grid Infrastructure in zero-downtime mode instead of rolling mode, udev rules becomes a perfect option.

Saturday, November 7, 2020

Zero-Downtime Grid Infrastructure Patching root.sh failed with CLSRSC-675: Cluster-wide Grid Plug and Play profile not found

 When running root.sh with zero-downtime GI patching option -transparent and -nodriverupdate, got following errors
2020/11/05 11:55:01 CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node.
Died at /u01/app/19.9.0/grid/crs/install/crsgpnp.pm line 3436.
The command '/u01/app/19.9.0/grid/perl/bin/perl -I/u01/app/19.9.0/grid/perl/lib -I/u01/app/19.9.0/grid/crs/install /u01/app/19.9.0/grid/srvm/admin/rhpmovegi.pl /u01/app/19.0.0/grid /u01/app/19.9.0/grid true' execution failed
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_11-55-01AM.log
Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_11-55-01AM.log
2020/11/05 11:55:01 CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node.
Died at /u01/app/19.9.0/grid/crs/install/crsgpnp.pm line 3436.
The command '/u01/app/19.9.0/grid/perl/bin/perl -I/u01/app/19.9.0/grid/perl/lib -I/u01/app/19.9.0/grid/crs/install /u01/app/19.9.0/grid/crs/install/rootcrs.pl  -transparent -nodriverupdate -dstcrshome /u01/app/19.9.0/grid -prepatch' execution failed

Sunday, May 17, 2020

12.2.0.1 init.ohasd does not start clusterware stack after April 2020 RU applied

The init.ohasd process does not start clusterware stack after the 12.2.0.1 Grid Infrastructure home is applied April 2020 Release Update. The scenario is not a typical Grid Infrastructure startup issue. It is a human error and happened when DBA, by accident, ran roothas.sh instead of rootcrs.sh while patching GI cluster home. Before knowing it is human error, the diagnose process is done as following.

Thursday, January 25, 2018

AIX Disk Only Visible to One ASM Instance of Grid Infrastructure

Oracle Grid Infrastructure 12.2.0.1
AIX 7.1


New disks are added to both nodes of Grid Infrastructure, and the disk permission is configured properly,
[grid@host01]$ ls -l /dev/*hdisk[56]
brw-------      1 root      system         21, 26 Jan 22 10:58 /dev/hdisk5
brw-------      1 root      system         21, 25 Jan 22 10:58 /dev/hdisk6
crw-rw----    1 grid     asmadmin     21, 26 Jan 24 16:07 /dev/rhdisk5
crw-rw----    1 grid     asmadmin     21, 25 Jan 24 16:06 /dev/rhdisk6

[grid@host02]$ ls -l /dev/*hdisk2[56]
brw-------      1 root      system          21, 25 Jan 22 11:08 /dev/hdisk5
brw-------      1 root      system          21, 31 Jan 22 11:08 /dev/hdisk6
crw-rw----    1 grid     asmadmin     21, 25 Jan 22 11:08 /dev/rhdisk5
crw-rw----    1 grid     asmadmin     21, 31 Jan 24 14:49 /dev/rhdisk6

Saturday, January 20, 2018

Upgrading Grid Infrastructure from 12.1 to 12.2 failed with [INS-20802]

Customer is upgrading two-node Oracle Grid Infrastructure from version 12.1.0.2 to version 12.2.0.1, and gridSetup.sh returned error [INS-20802] during Preparing for deploying Oracle Grid Infrastructure Management Repository.

Friday, December 15, 2017

ASM instance shows 'Down' in OEM 13.2 after upgraded to 12.2

Standalone Grid Infrastruture (Oracle Restart) is upgraded to 12.2 and everything works well. However, the ASM instance shows 'Down' in Enterprise Manager 13.2.

Checked 'Monitoring Configuration' of ASM target, all information is correct and 'Test Connection' succeeds.  We can log into ASM instance page in OEM to do management tasks (creat/drop diskgroup, etc). Everything looks good except the Target Status.

Log onto the host where ASM is running on, and run following commands from OEM agent home:

1. Get the target name of ASM instance
$ emctl config agent listtargets | grep ASM
[+ASM_host01.dbaplus.ca, osm_instance]



2. Find out the 'real time' results of the 'REsponse' metric (test connection from local agent to target)
$ emctl getmetric agent +ASM_host01.dbaplus.ca,osm_instance,Response
Oracle Enterprise Manager Cloud Control 12c Release 5 
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Status,oraerr
0
,Failed to connect: java.sql.SQLException: ORA-28040: No matching authentication protocol


Wow, '0' is returned with 'ORA-28040'. I believe many DBAs saw error 28040 when upgrading their databases to 12c. OEM agent (12c Release 5) is using old Oracle client (10g) to access ASM instance which is running on 12.2.

Solution:
Add following lines to $GRID_HOME/network/admin/sqlnet.ora
SQLNET.ALLOWED_LOGON_VERSION_SERVER=10
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=10
 

Test the connection again:
$emctl getmetric agent +ASM_host01.dbaplus.ca,osm_instance,Response
Oracle Enterprise Manager Cloud Control 12c Release 5 
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Status,oraerr
1
,
 


'1' is returned and indicates that the ASM  is up and running. 

And the ASM target shows 'Up' in OEM. 

Friday, October 2, 2015

Invalid password while logging 12c ASM instance

Customer complains that he cannot log into 12c ASM instance because of invalid username/password, but he is pretty sure username and password are correct.
$ . oraenv
ORACLE_SID = [+ASM] ? +ASM
The Oracle base remains unchanged with value /u01/app
$
$ sqlplus /nolog

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 2 10:26:38 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

SQL> connect sys as sysasm
Enter password:
ERROR:
ORA-01017: invalid username/password; logon denied

Try to log in through OS authentication and reset the password:
SQL> conn / as sysasm
Connected.

SQL> alter user sys identified by abc123;
alter user sys identified by abc123
                             *
ERROR at line 1:
ORA-01918: user 'INTERNAL' does not exist

SQL> select * from v$pwfile_users;

no rows selected

Looks like the password file is missing:
$ ls -l $ORACLE_HOME/dbs
total 18
-rw-rw----   1 oragrid  oracle      2180 Jul 24 13:32 ab_+ASM.dat
-rw-rw----   1 oragrid  oracle      1544 Oct  2 10:28 hc_+ASM.dat
-rw-r--r--   1 oragrid  oracle      2992 Feb  3  2012 init.ora

Ok, the password file is not there as before. And we know it can be stored in ASM diskgroup from 12c, check if it is the case:
$ srvctl config asm -a
ASM home: 
Password file:
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
$

The entry 'Password file' is empty, it is not expected configuration if password file does not exist in $ORACLE_HOME/dbs. Let us see if there is password file in ASM:
$ asmcmd
ASMCMD> ls -l
State    Type    Rebal  Name
MOUNTED  EXTERN  N      DATA/
MOUNTED  EXTERN  N      FRA/
ASMCMD> cd DATA
ASMCMD> ls -l
Type      Redund  Striped  Time             Sys  Name
                                            Y    ASM/
                                            Y    DB01/
PASSWORD  UNPROT  COARSE   JUL 14 09:00:00  N    orapwasm => +DATA/ASM/PASSWORD/pwdasm.256.885030391
ASMCMD> exit

The password file 'orapwasm' is in 'DATA' diskgroup, update Oracle Restart registry:
$ srvctl modify asm -pwfile +DATA/orapwasm
$ srvctl config asm -a
ASM home: 
Password file: +DATA/orapwasm
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:

try logging again:
$ sqlplus sys as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 16 11:16:04 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS SYSBA SYSDG SYSKM     CON_ID
------------------------------ ----- ----- ----- ----- ----- ----- ----------
SYS                            TRUE  TRUE  TRUE  FALSE FALSE FALSE          0
ASMSNMP                        TRUE  FALSE FALSE FALSE FALSE FALSE          0

It works.

Wednesday, June 10, 2015

Failed to retrieve the password file location used by ASM asm

After Oracle Restart (Grid Infrastructure standalone) is upgraded from 11g to 12c, srvctl get errors:
$ srvctl config asm
ASM home: 
PRCA-1057 : Failed to retrieve the password file location used by ASM asm
PRCR-1097 : Resource attribute not found: PWFILE

Check environment and password file:
$ echo $ORACLE_HOME
/u01/app/grid/product/12.1.0/grid

$ ls -l /u01/app/grid/product/12.1.0/grid/dbs
total 28
-rw-rw---- 1 grid oinstall 2783 Jun  9 14:27 ab_+ASM.dat
-rw-rw---- 1 grid oinstall 1544 Jun  9 14:27 hc_+ASM.dat
-rw-r--r-- 1 grid oinstall 2992 Feb  3  2012 init.ora
-rw-r----- 1 grid oinstall 8704 Jun  9 14:10 orapw+ASM
-rw-r----- 1 grid oinstall 1536 Jun  9 14:27 spfile+ASM.ora

The password file was created and ORACLE_HOME is set correctly. Checked with DBA, the upgrade was done successfully. There was no errors/exceptions during upgrade.
Oracle Support explains that:
12c ASM instance has a attribute called PWFILE(password file), but 11g ASM does not. Looks like the upgrade utility does not create the new attribute for an ASM instance which is upgraded from 11g.
Try to manually re-create the ora.asm resource with -pwfile option:
$ crsctl status resource ora.asm
NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on host01

$ srvctl stop asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified
$ srvctl stop asm -f
$ 
$ srvctl config asm
PRCR-1001 : Resource ora.asm does not exist
$ 
$ srvctl add asm -listener LISTENER -spfile /u01/app/grid/product/12.1.0/grid/dbs/spfile+ASM.ora -pwfile /u01/app/grid/product/12.1.0/grid/dbs/orapw+ASM

Test if it is fixed:
$ srvctl sconfig asm
ASM home: 
Password file: /u01/app/grid/product/12.1.0/grid/dbs/orapw+ASM
ASM listener: LISTENER
Spfile: /u01/app/grid/product/12.1.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

There is no error any more.

Monday, February 9, 2015

Apply GI PSU 12.1.0.2.2 while using Software-Only installation of GI12.1.0.2

A software-only installation only copies the Oracle Grid Infrastructure for a standalone server binaries to the specified location. After the installation, Oracle Grid Infrastructure (GI) is not configured and HAS (GI standalone)/ CRS(GI cluster) stack is not up yet. Therefore, "optach auto" introduced the the PSU document can not be used to apply the patch.

1. Start Oracle Universal Installer as the Oracle Restart software owner user (grid)
[grid@host01 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 82041 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4095 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-02-05_02-16-33PM. Please wait ...
[grid@host01 grid]$ You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2015-02-05_02-16-33PM.log
... ...

2. In the Select Installation Option screen, select the Install Oracle Grid Infrastructure Software Only option to install Oracle Restart and Oracle ASM software only. Click Next.

3. In the Select Product Languages screen, select one or more languages. Move the languages from the Available Languages list to the Selected Languages list. Click Next.

4. In the Privileged Operating System Groups screen, select the name of the operating system group you created for the OSDBA group, the OSASM group, and the Oracle ASM operator group OSOPER. If you create only the dba group, then you can use that group for all three privileged groups. If you created a separate asmadmin group, then use that value for the OSASM group. Click Next.

5. In the Specify Installation Location screen, enter the following information, and click Next:
Oracle Base: Enter the location for the Oracle base directory. Do not include spaces in the path. Here, I am using: /u01/app/grid
Software Location: Accept the default value or enter the directory path in which you want to install the software.The directory path must not contain
spaces. Here, I am using /u01/app/grid/product/12.1.0/grid12102

6. In the pop-up window, the warning message is for cluster installation, click Yes to continue if the GI is installed as Standalone. Otherwise, Oracle Home location has to be out of ORACLE_BASE.

7. The Create Inventory screen is displayed only if this is the first time you are installing Oracle software on your system. Change the path for the Inventory Directory (e.g. /u01/app/oraInventory), if required. Select oinstall for the oraInventory Group Name, if required. Click Next.

8. The Perform Prerequisite Checks screen checks if the minimum system requirements are met to perform the Oracle Grid Infrastructure installation. If all the system requirements are met, then you are directed to the Summary screen. If an installation fails, you can review the error.

9. Review the contents of the Summary screen, and click Install.

10. The Install Product screen displays the progress of the Oracle Grid Infrastructure installation. Oracle Universal Installer prompts you to run the root.sh script and, if required, the orainstRoot.sh script as the root user to complete the installation. During this process, the Execute Configuration Scripts window appears. Do not click OK until you successfully apply the patch (PSU 12.1.0.2.2).

11. If required, run orainstRoot.sh script as the root user
[root@host01 app]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

12. Install new version of OPatch
[grid@host01 tmp]$ cd /u01/app/grid/product/12.1.0/grid12102
[grid@host01 grid12102]$ OPatch/opatch version
OPatch Version: 12.1.0.1.3

OPatch succeeded.
[grid@host01 grid12102]$ mv OPatch OPatch.12.1.0.1.3
[grid@host01 grid12102]$ unzip /u01/app/media/p6880880_121010_Linux-x86-64.zip
Archive:  /u01/app/media/p6880880_121010_Linux-x86-64.zip
   creating: OPatch/
  inflating: OPatch/opatchdiag.bat
    ... ...
[grid@host01 grid12102]$ OPatch/opatch version
OPatch Version: 12.1.0.1.6

OPatch succeeded.

13. Extract the GI PSU 12.1.0.2.2 (Patch 19954978) into the empty stage directory as the GI software owner (grid), we will find four patched included in the PSU bundle: 19769473, 19769479, 19769480, 19872484
[grid@host01 grid12102]$ unzip -d /u01/app/media/ /u01/app/media/p19954978_121020_Linux-x86-64.zip
[grid@host01 grid12102]$ ls -l /u01/app/media/19954978
total 104
drwxr-xr-x. 5 grid oinstall  4096 Jan 16 04:30 19769473
drwxr-xr-x. 5 grid oinstall  4096 Jan 16 04:30 19769479
drwxrwxr-x. 4 grid oinstall  4096 Jan 16 04:30 19769480
drwxr-xr-x. 4 grid oinstall  4096 Jan 16 04:30 19872484
drwxr-xr-x. 2 grid oinstall  4096 Jan 16 04:30 automation
-rw-rw-r--. 1 grid oinstall  4987 Jan 16 05:53 bundle.xml
-rw-rw-r--. 1 grid oinstall 77081 Jan 19 12:32 README.html
-rw-r--r--. 1 grid oinstall     0 Jan 16 04:30 README.txt

14.  Apply patch to the newly installed 12.1.0.2 GI home as the GI software owner, since "opatch auto" cannot be used, we have to apply patches in the PSU bundle one by one using opatch apply,
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769479
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769473
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19872484
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769480
[grid@host01 ~]$ export ORACLE_HOME=/u01/app/grid/product/12.1.0/grid12102
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769479
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769479' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
   ... ...
Verifying the update...
Patch 19769479 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-33-35PM_1.log

OPatch succeeded.

[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769473
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769473' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
   ... ...
Patching component oracle.usm, 12.1.0.2.0...

Verifying the update...
Patch 19769473 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-41-04PM_1.log

OPatch succeeded.
[grid@host01 ~]$
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19872484
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19872484' to OH '/u01/app/grid/product/12.1.0/grid12102'
   ... ...
Patching component oracle.wlm.dbwlm, 12.1.0.2.0...

Verifying the update...
Patch 19872484 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-45-29PM_1.log

OPatch succeeded.
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769480
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769480' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
Patch 19769480: Optional component(s) missing : [ oracle.xdk, 12.1.0.2.0 ] , [ oracle.oraolap, 12.1.0.2.0 ]
   ... ...
OPatch succeeded.

15. Run root.sh as user root


[root@host01 bin]# /u01/app/grid/product/12.1.0/grid12102/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/grid/product/12.1.0/grid12102

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/grid/product/12.1.0/grid12102/perl/bin/perl -I/u01/app/grid/product/12.1.0/grid12102/perl/lib -I/u01/app/grid/product/12.1.0/grid12102    /crs/install /u01/app/grid/product/12.1.0/grid12102/crs/install/roothas.pl


To configure Grid Infrastructure for a Cluster execute the following command as grid user:
/u01/app/grid/product/12.1.0/grid12102/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed     through the response file that is available in the installation media.



16. Back to Oracle Universal Installer. In Execute Configuration Scripts screen, click Ok.
17. The Finish screen displays the installation status. Click Close to end the installation and exit Oracle Universal Installer.

References:
Supplemental Readme - Patch Installation and Deinstallation For 12.1.0.x.x GI PSU (Doc ID 1591616.1)
How to Apply a Grid Infrastructure Patch Before root script (root.sh or rootupgrade.sh) is Executed? (Doc ID 1410202.1)

Wednesday, January 14, 2015

Automate Startup/Shutdown of OSWatcher with Oracle HAS (GridInfrastructure Standalone)

Grid Infrastructure Standalone 11.2.0.3
Operation system AIX 6.1

Download Oracle OSWatcher (oswbb732.tar, current version 7.3.2) from My Oracle Support. Install OSWatcher:
$ mkdir /u01/app/grid/product/7.3.2
$ cd /u01/app/grid/product/7.3.2
$ tar xvf /tmp/oswbb732.tar
  ... ...
$ ls -l oswbb/*OSWbb*
-rwxr-xr-x    1 grid  oinstall         2385 Oct 07 2013  oswbb/startOSWbb.sh
-rwxr-xr-x    1 grid  oinstall          558 Apr 17 2014  oswbb/stopOSWbb.sh