Showing posts with label Grid Infrastructure & ASM. Show all posts
Showing posts with label Grid Infrastructure & ASM. Show all posts

Thursday, January 25, 2018

AIX Disk Only Visible to One ASM Instance of Grid Infrastructure

Oracle Grid Infrastructure 12.2.0.1
AIX 7.1


New disks are added to both nodes of Grid Infrastructure, and the disk permission is configured properly,
[grid@host01]$ ls -l /dev/*hdisk[56]
brw-------      1 root      system         21, 26 Jan 22 10:58 /dev/hdisk5
brw-------      1 root      system         21, 25 Jan 22 10:58 /dev/hdisk6
crw-rw----    1 grid     asmadmin     21, 26 Jan 24 16:07 /dev/rhdisk5
crw-rw----    1 grid     asmadmin     21, 25 Jan 24 16:06 /dev/rhdisk6

[grid@host02]$ ls -l /dev/*hdisk2[56]
brw-------      1 root      system          21, 25 Jan 22 11:08 /dev/hdisk5
brw-------      1 root      system          21, 31 Jan 22 11:08 /dev/hdisk6
crw-rw----    1 grid     asmadmin     21, 25 Jan 22 11:08 /dev/rhdisk5
crw-rw----    1 grid     asmadmin     21, 31 Jan 24 14:49 /dev/rhdisk6

Saturday, January 20, 2018

Upgrading Grid Infrastructure from 12.1 to 12.2 failed with [INS-20802]

Customer is upgrading two-node Oracle Grid Infrastructure from version 12.1.0.2 to version 12.2.0.1, and gridSetup.sh returned error [INS-20802] during Preparing for deploying Oracle Grid Infrastructure Management Repository.

Friday, December 15, 2017

ASM instance shows 'Down' in OEM 13.2 after upgraded to 12.2

Standalone Grid Infrastruture (Oracle Restart) is upgraded to 12.2 and everything works well. However, the ASM instance shows 'Down' in Enterprise Manager 13.2.

Checked 'Monitoring Configuration' of ASM target, all information is correct and 'Test Connection' succeeds.  We can log into ASM instance page in OEM to do management tasks (creat/drop diskgroup, etc). Everything looks good except the Target Status.

Log onto the host where ASM is running on, and run following commands from OEM agent home:

1. Get the target name of ASM instance
$ emctl config agent listtargets | grep ASM
[+ASM_host01.dbaplus.ca, osm_instance]



2. Find out the 'real time' results of the 'REsponse' metric (test connection from local agent to target)
$ emctl getmetric agent +ASM_host01.dbaplus.ca,osm_instance,Response
Oracle Enterprise Manager Cloud Control 12c Release 5 
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Status,oraerr
0
,Failed to connect: java.sql.SQLException: ORA-28040: No matching authentication protocol


Wow, '0' is returned with 'ORA-28040'. I believe many DBAs saw error 28040 when upgrading their databases to 12c. OEM agent (12c Release 5) is using old Oracle client (10g) to access ASM instance which is running on 12.2.

Solution:
Add following lines to $GRID_HOME/network/admin/sqlnet.ora
SQLNET.ALLOWED_LOGON_VERSION_SERVER=10
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=10
 

Test the connection again:
$emctl getmetric agent +ASM_host01.dbaplus.ca,osm_instance,Response
Oracle Enterprise Manager Cloud Control 12c Release 5 
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Status,oraerr
1
,
 


'1' is returned and indicates that the ASM  is up and running. 

And the ASM target shows 'Up' in OEM. 

Friday, October 2, 2015

Invalid password while logging 12c ASM instance

Customer complains that he cannot log into 12c ASM instance because of invalid username/password, but he is pretty sure username and password are correct.
$ . oraenv
ORACLE_SID = [+ASM] ? +ASM
The Oracle base remains unchanged with value /u01/app
$
$ sqlplus /nolog

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 2 10:26:38 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

SQL> connect sys as sysasm
Enter password:
ERROR:
ORA-01017: invalid username/password; logon denied

Try to log in through OS authentication and reset the password:
SQL> conn / as sysasm
Connected.

SQL> alter user sys identified by abc123;
alter user sys identified by abc123
                             *
ERROR at line 1:
ORA-01918: user 'INTERNAL' does not exist

SQL> select * from v$pwfile_users;

no rows selected

Looks like the password file is missing:
$ ls -l $ORACLE_HOME/dbs
total 18
-rw-rw----   1 oragrid  oracle      2180 Jul 24 13:32 ab_+ASM.dat
-rw-rw----   1 oragrid  oracle      1544 Oct  2 10:28 hc_+ASM.dat
-rw-r--r--   1 oragrid  oracle      2992 Feb  3  2012 init.ora

Ok, the password file is not there as before. And we know it can be stored in ASM diskgroup from 12c, check if it is the case:
$ srvctl config asm -a
ASM home: 
Password file:
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
$

The entry 'Password file' is empty, it is not expected configuration if password file does not exist in $ORACLE_HOME/dbs. Let us see if there is password file in ASM:
$ asmcmd
ASMCMD> ls -l
State    Type    Rebal  Name
MOUNTED  EXTERN  N      DATA/
MOUNTED  EXTERN  N      FRA/
ASMCMD> cd DATA
ASMCMD> ls -l
Type      Redund  Striped  Time             Sys  Name
                                            Y    ASM/
                                            Y    DB01/
PASSWORD  UNPROT  COARSE   JUL 14 09:00:00  N    orapwasm => +DATA/ASM/PASSWORD/pwdasm.256.885030391
ASMCMD> exit

The password file 'orapwasm' is in 'DATA' diskgroup, update Oracle Restart registry:
$ srvctl modify asm -pwfile +DATA/orapwasm
$ srvctl config asm -a
ASM home: 
Password file: +DATA/orapwasm
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:

try logging again:
$ sqlplus sys as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 16 11:16:04 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS SYSBA SYSDG SYSKM     CON_ID
------------------------------ ----- ----- ----- ----- ----- ----- ----------
SYS                            TRUE  TRUE  TRUE  FALSE FALSE FALSE          0
ASMSNMP                        TRUE  FALSE FALSE FALSE FALSE FALSE          0

It works.

Wednesday, June 10, 2015

Failed to retrieve the password file location used by ASM asm

After Oracle Restart (Grid Infrastructure standalone) is upgraded from 11g to 12c, srvctl get errors:
$ srvctl config asm
ASM home: 
PRCA-1057 : Failed to retrieve the password file location used by ASM asm
PRCR-1097 : Resource attribute not found: PWFILE

Check environment and password file:
$ echo $ORACLE_HOME
/u01/app/grid/product/12.1.0/grid

$ ls -l /u01/app/grid/product/12.1.0/grid/dbs
total 28
-rw-rw---- 1 grid oinstall 2783 Jun  9 14:27 ab_+ASM.dat
-rw-rw---- 1 grid oinstall 1544 Jun  9 14:27 hc_+ASM.dat
-rw-r--r-- 1 grid oinstall 2992 Feb  3  2012 init.ora
-rw-r----- 1 grid oinstall 8704 Jun  9 14:10 orapw+ASM
-rw-r----- 1 grid oinstall 1536 Jun  9 14:27 spfile+ASM.ora

The password file was created and ORACLE_HOME is set correctly. Checked with DBA, the upgrade was done successfully. There was no errors/exceptions during upgrade.
Oracle Support explains that:
12c ASM instance has a attribute called PWFILE(password file), but 11g ASM does not. Looks like the upgrade utility does not create the new attribute for an ASM instance which is upgraded from 11g.
Try to manually re-create the ora.asm resource with -pwfile option:
$ crsctl status resource ora.asm
NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on host01

$ srvctl stop asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified
$ srvctl stop asm -f
$ 
$ srvctl config asm
PRCR-1001 : Resource ora.asm does not exist
$ 
$ srvctl add asm -listener LISTENER -spfile /u01/app/grid/product/12.1.0/grid/dbs/spfile+ASM.ora -pwfile /u01/app/grid/product/12.1.0/grid/dbs/orapw+ASM

Test if it is fixed:
$ srvctl sconfig asm
ASM home: 
Password file: /u01/app/grid/product/12.1.0/grid/dbs/orapw+ASM
ASM listener: LISTENER
Spfile: /u01/app/grid/product/12.1.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

There is no error any more.

Monday, February 9, 2015

Apply GI PSU 12.1.0.2.2 while using Software-Only installation of GI12.1.0.2

A software-only installation only copies the Oracle Grid Infrastructure for a standalone server binaries to the specified location. After the installation, Oracle Grid Infrastructure (GI) is not configured and HAS (GI standalone)/ CRS(GI cluster) stack is not up yet. Therefore, "optach auto" introduced the the PSU document can not be used to apply the patch.

1. Start Oracle Universal Installer as the Oracle Restart software owner user (grid)
[grid@host01 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 82041 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4095 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-02-05_02-16-33PM. Please wait ...
[grid@host01 grid]$ You can find the log of this install session at: /u01/app/oraInventory/logs/installActions2015-02-05_02-16-33PM.log
... ...

2. In the Select Installation Option screen, select the Install Oracle Grid Infrastructure Software Only option to install Oracle Restart and Oracle ASM software only. Click Next.

3. In the Select Product Languages screen, select one or more languages. Move the languages from the Available Languages list to the Selected Languages list. Click Next.

4. In the Privileged Operating System Groups screen, select the name of the operating system group you created for the OSDBA group, the OSASM group, and the Oracle ASM operator group OSOPER. If you create only the dba group, then you can use that group for all three privileged groups. If you created a separate asmadmin group, then use that value for the OSASM group. Click Next.

5. In the Specify Installation Location screen, enter the following information, and click Next:
Oracle Base: Enter the location for the Oracle base directory. Do not include spaces in the path. Here, I am using: /u01/app/grid
Software Location: Accept the default value or enter the directory path in which you want to install the software.The directory path must not contain
spaces. Here, I am using /u01/app/grid/product/12.1.0/grid12102

6. In the pop-up window, the warning message is for cluster installation, click Yes to continue if the GI is installed as Standalone. Otherwise, Oracle Home location has to be out of ORACLE_BASE.

7. The Create Inventory screen is displayed only if this is the first time you are installing Oracle software on your system. Change the path for the Inventory Directory (e.g. /u01/app/oraInventory), if required. Select oinstall for the oraInventory Group Name, if required. Click Next.

8. The Perform Prerequisite Checks screen checks if the minimum system requirements are met to perform the Oracle Grid Infrastructure installation. If all the system requirements are met, then you are directed to the Summary screen. If an installation fails, you can review the error.

9. Review the contents of the Summary screen, and click Install.

10. The Install Product screen displays the progress of the Oracle Grid Infrastructure installation. Oracle Universal Installer prompts you to run the root.sh script and, if required, the orainstRoot.sh script as the root user to complete the installation. During this process, the Execute Configuration Scripts window appears. Do not click OK until you successfully apply the patch (PSU 12.1.0.2.2).

11. If required, run orainstRoot.sh script as the root user
[root@host01 app]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

12. Install new version of OPatch
[grid@host01 tmp]$ cd /u01/app/grid/product/12.1.0/grid12102
[grid@host01 grid12102]$ OPatch/opatch version
OPatch Version: 12.1.0.1.3

OPatch succeeded.
[grid@host01 grid12102]$ mv OPatch OPatch.12.1.0.1.3
[grid@host01 grid12102]$ unzip /u01/app/media/p6880880_121010_Linux-x86-64.zip
Archive:  /u01/app/media/p6880880_121010_Linux-x86-64.zip
   creating: OPatch/
  inflating: OPatch/opatchdiag.bat
    ... ...
[grid@host01 grid12102]$ OPatch/opatch version
OPatch Version: 12.1.0.1.6

OPatch succeeded.

13. Extract the GI PSU 12.1.0.2.2 (Patch 19954978) into the empty stage directory as the GI software owner (grid), we will find four patched included in the PSU bundle: 19769473, 19769479, 19769480, 19872484
[grid@host01 grid12102]$ unzip -d /u01/app/media/ /u01/app/media/p19954978_121020_Linux-x86-64.zip
[grid@host01 grid12102]$ ls -l /u01/app/media/19954978
total 104
drwxr-xr-x. 5 grid oinstall  4096 Jan 16 04:30 19769473
drwxr-xr-x. 5 grid oinstall  4096 Jan 16 04:30 19769479
drwxrwxr-x. 4 grid oinstall  4096 Jan 16 04:30 19769480
drwxr-xr-x. 4 grid oinstall  4096 Jan 16 04:30 19872484
drwxr-xr-x. 2 grid oinstall  4096 Jan 16 04:30 automation
-rw-rw-r--. 1 grid oinstall  4987 Jan 16 05:53 bundle.xml
-rw-rw-r--. 1 grid oinstall 77081 Jan 19 12:32 README.html
-rw-r--r--. 1 grid oinstall     0 Jan 16 04:30 README.txt

14.  Apply patch to the newly installed 12.1.0.2 GI home as the GI software owner, since "opatch auto" cannot be used, we have to apply patches in the PSU bundle one by one using opatch apply,
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769479
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769473
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19872484
$ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769480
[grid@host01 ~]$ export ORACLE_HOME=/u01/app/grid/product/12.1.0/grid12102
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769479
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769479' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
   ... ...
Verifying the update...
Patch 19769479 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-33-35PM_1.log

OPatch succeeded.

[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769473
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769473' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
   ... ...
Patching component oracle.usm, 12.1.0.2.0...

Verifying the update...
Patch 19769473 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-41-04PM_1.log

OPatch succeeded.
[grid@host01 ~]$
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19872484
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19872484' to OH '/u01/app/grid/product/12.1.0/grid12102'
   ... ...
Patching component oracle.wlm.dbwlm, 12.1.0.2.0...

Verifying the update...
Patch 19872484 successfully applied
Log file location: /u01/app/grid/product/12.1.0/grid12102/cfgtoollogs/opatch/opatch2015-02-09_14-45-29PM_1.log

OPatch succeeded.
[grid@host01 ~]$ $ORACLE_HOME/OPatch/opatch apply -oh /u01/app/grid/product/12.1.0/grid12102 -local /u01/app/media/19954978/19769480
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
   ... ...
Applying interim patch '19769480' to OH '/u01/app/grid/product/12.1.0/grid12102'
Verifying environment and performing prerequisite checks...
Patch 19769480: Optional component(s) missing : [ oracle.xdk, 12.1.0.2.0 ] , [ oracle.oraolap, 12.1.0.2.0 ]
   ... ...
OPatch succeeded.

15. Run root.sh as user root


[root@host01 bin]# /u01/app/grid/product/12.1.0/grid12102/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/grid/product/12.1.0/grid12102

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/grid/product/12.1.0/grid12102/perl/bin/perl -I/u01/app/grid/product/12.1.0/grid12102/perl/lib -I/u01/app/grid/product/12.1.0/grid12102    /crs/install /u01/app/grid/product/12.1.0/grid12102/crs/install/roothas.pl


To configure Grid Infrastructure for a Cluster execute the following command as grid user:
/u01/app/grid/product/12.1.0/grid12102/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed     through the response file that is available in the installation media.



16. Back to Oracle Universal Installer. In Execute Configuration Scripts screen, click Ok.
17. The Finish screen displays the installation status. Click Close to end the installation and exit Oracle Universal Installer.

References:
Supplemental Readme - Patch Installation and Deinstallation For 12.1.0.x.x GI PSU (Doc ID 1591616.1)
How to Apply a Grid Infrastructure Patch Before root script (root.sh or rootupgrade.sh) is Executed? (Doc ID 1410202.1)

Wednesday, January 14, 2015

Automate Startup/Shutdown of OSWatcher with Oracle HAS (GridInfrastructure Standalone)

Grid Infrastructure Standalone 11.2.0.3
Operation system AIX 6.1

Download Oracle OSWatcher (oswbb732.tar, current version 7.3.2) from My Oracle Support. Install OSWatcher:
$ mkdir /u01/app/grid/product/7.3.2
$ cd /u01/app/grid/product/7.3.2
$ tar xvf /tmp/oswbb732.tar
  ... ...
$ ls -l oswbb/*OSWbb*
-rwxr-xr-x    1 grid  oinstall         2385 Oct 07 2013  oswbb/startOSWbb.sh
-rwxr-xr-x    1 grid  oinstall          558 Apr 17 2014  oswbb/stopOSWbb.sh

Create an Oracle HAS(Grid Infrastructure Standalone) Action Script file startOSWbb.sh:
#!/bin/sh
#
# ACTION_SCRIPT for Oracle HAS to manage OSWatcher
#
OSWBB_HOME=/u01/app/grid/product/7.3.2/oswbb
OSWBB_LOG=/u01/app/grid/local/logs/oswbb/alert_OSWbb.log
OSWBB_ARCHIVE_DEST=/u01/app/grid/local/logs/oswbb/archive
SNAPSHOT_INTERVAL=30   # in seconds
RETENTION_POLICY=720   # in hours
COMPRESS_COMMAND=NONE  # compress utility command such as zip, gzip, etc
#
case "$1" in
  'start')
     cd $OSWBB_HOME
     echo " " >> $OSWBB_LOG
     TMP=`ps -ef | grep OSWatcher  | grep -v grep |wc -l`
     if [ $TMP -eq 1 ]; then  # Only one process (OSWatcher or OSWatcherFM) running is not enough
        ./stopOSWbb.sh        # Stop orphan process
     fi
     echo "******************************************************" >> $OSWBB_LOG
     echo "...Starting OSWbb at `date` " >> $OSWBB_LOG
     ./startOSWbb.sh $SNAPSHOT_INTERVAL $RETENTION_POLICY $COMPRESS_COMMAND $OSWBB_ARCHIVE_DEST >> $OSWBB_LOG &
     if [ $? -eq 0 ]; then
        sleep 60    # Wait for all processes to be started
        exit 0
     else
        exit 1
     fi
     ;;
  'stop')
     cd $OSWBB_HOME
     echo "...Stopping OSWbb  at `date` " >> $OSWBB_LOG
     ./stopOSWbb.sh
     exit $?
     ;;
  'check')
    TMP=`ps -ef | grep OSWatcher  | grep -v grep |wc -l`
    if [ $TMP -eq 2 ]; then  # Both OSWatcher and OSWatcherFM (2 processes) running
       exit 0
    else
       exit 1
    fi
    ;;
  'clean')
     cd $OSWBB_HOME
     echo "...Cleaning/Stopping OSWbb  at `date` " >> $OSWBB_LOG
     ./stopOSWbb.sh
     exit $?
     ;;
esac

Note:
1. startOSWbb.sh has to be started in background by appending sign '&' at the end of the command line.
2. the output of startOSWbb.sh needs to be redirected to a file

Add a resource to HAS
crsctl add resource ora.OSWbb -type local_resource -attr "ACTION_SCRIPT=/u01/app/grid/local/bin/startOSWbb.sh,CHECK_INTERVAL=60,RESTART_ATTEMPTS=2"

Start resource ora.OSWbb
$ crsctl start resource ora.OSWbb
CRS-2672: Attempting to start 'ora.OSWbb' on 'host02'
CRS-2674: Start of 'ora.OSWbb' on 'host02' failed
CRS-4000: Command Start failed, or completed with errors.

Check OHASD agent log for errors:
$ tail /u01/app/grid/product/11.2.0/gi11204/log/host02/agent/ohasd/scriptagent_grid/scriptagent_grid.log
2015-01-14 10:00:49.240: [    AGFW][2057]{0:0:14962} Agent received the message: RESOURCE_START[ora.OSWbb host02 1] ID 4098:444107
2015-01-14 10:00:49.240: [    AGFW][2057]{0:0:14962} Preparing START command for: ora.OSWbb host02 1
2015-01-14 10:00:49.240: [    AGFW][2057]{0:0:14962} ora.OSWbb host02 1 state changed from: UNKNOWN to: STARTING
2015-01-14 10:00:49.242: [    AGFW][1543]{0:0:14962} Entering script entry point...
2015-01-14 10:00:49.242: [ora.OSWbb][1543]{0:0:14962} [start] Executing action script: /u01/app/grid/local/bin/startOSWbb.sh[start]
2015-01-14 10:00:49.304: [    AGFW][1543]{0:0:14962} Command: start for resource: ora.OSWbb host02 1 completed with status: SUCCESS
2015-01-14 10:00:49.305: [CLSFRAME][1] TM [MultiThread] is changing desired thread # to 3. Current # is 2
2015-01-14 10:00:49.306: [    AGFW][1543]{0:0:14962} Entering script entry point...
2015-01-14 10:00:49.306: [ora.OSWbb][1543]{0:0:14962} [check] Executing action script: /u01/app/grid/local/bin/startOSWbb.sh[check]
2015-01-14 10:00:49.307: [    AGFW][2057]{0:0:14962} Agent sending reply for: RESOURCE_START[ora.OSWbb host02 1] ID 4098:444107
2015-01-14 10:00:49.365: [    AGFW][2057]{0:0:14962} ora.OSWbb host02 1 state changed from: STARTING to: OFFLINE
2015-01-14 10:00:49.365: [    AGFW][2057]{0:0:14962} Agent sending last reply for: RESOURCE_START[ora.OSWbb host02 1] ID 4098:444107
2015-01-14 10:00:49.365: [    AGFW][2057]{0:0:14962} Agent has no resources to be monitored, Shutting down ..
2015-01-14 10:00:49.365: [    AGFW][2057]{0:0:14962} Agent sending message to PE: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:25
2015-01-14 10:00:49.370: [    AGFW][2057]{0:0:14962} Agent is shutting down.
2015-01-14 10:00:49.370: [ USRTHRD][2057]{0:0:14962} Script agent is exiting..

Did not find helpful message, check OSWatcher log:
$ cat  /u01/app/grid/local/logs/oswbb/alert_OSWbb.log

******************************************************
The archive directory you specified for parameter 4 in startOSWbb.sh:/u01/app/grid/local/logs/oswbb/archive does not exist. Please create this directory and rerun ./startOSWbb.sh

******************************************************
...Starting OSWbb at Wed Jan 14 10:00:49 EST 2015
The archive directory you specified for parameter 4 in startOSWbb.sh:/u01/app/grid/local/logs/oswbb/archive does not exist. Please create this directory and rerun ./startOSWbb.sh

OSWatcher archive directory does not exist, manually create it:
$ mkdir /u01/app/grid/local/logs/oswbb/archive

Re-start resource ora.OSWbb
$ crsctl start resource ora.OSWbb
CRS-2672: Attempting to start 'ora.OSWbb' on 'host02'
CRS-2676: Start of 'ora.OSWbb' on 'host02' succeeded

Resource ora.OSWbb started successfully, make sure OSWatcher is running:
$ ps -ef | grep OSWatcher
 grid  7405596  5046352   0 10:13:16  pts/6  0:00 grep OSWatcher
 grid  7078198 11469280   0 10:11:23      -  0:00 /bin/sh ./OSWatcherFM.sh 720 /u01/app/grid/local/logs/oswbb/archive
 grid 11469280        1   0 10:10:54      -  0:00 /bin/sh ./OSWatcher.sh 30 720 NONE /u01/app/grid/local/logs/oswbb/archive

Note: Keyword 'OSWatcher' is reserved by OSWatcher stopping script to determine if the utility is running. DO NOT using string 'OSWatcher' anywhere (directory name, file name, command name, etc).

Test resource ora.OSWbb  stop function:
$ crsctl stop resource ora.OSWbb
CRS-2673: Attempting to stop 'ora.OSWbb' on 'host02'
CRS-2677: Stop of 'ora.OSWbb' on 'host02' succeeded
$ ps -ef | grep OSWatcher
 oragrid 12058682  5046352   0 15:15:50  pts/6  0:00 grep OSWatcher

Test resource ora.OSWbb  restart function (check & start functions):
$ ps -ef | grep OSWatcher
 oragrid 11010394  5046352   0 15:18:33  pts/6  0:00 grep OSWatcher
$ crsctl start resource ora.OSWbb
CRS-2672: Attempting to start 'ora.OSWbb' on 'host02'
CRS-2676: Start of 'ora.OSWbb' on 'host02' succeeded
$  ps -ef | grep OSWatcher
 oragrid  7864326        1   0 15:18:42      -  0:00 /bin/sh ./OSWatcher.sh 30 720 NONE /u01/app/grid/local/logs/oswbb/archive
 oragrid  8388952  7864326   0 15:19:10      -  0:00 /bin/sh ./OSWatcherFM.sh 720 /u01/app/grid/local/logs/oswbb/archive
 oragrid 11665602  5046352   0 15:19:53  pts/6  0:00 grep OSWatcher

OSWatcher is running, kill one of two running processes:
$ kill 8388952
$
$ ps -ef | grep OSWatcher
 oragrid  7405662  5046352   0 15:20:26  pts/6  0:00 grep OSWatcher
 oragrid  7864326        1   0 15:18:42      -  0:00 /bin/sh ./OSWatcher.sh 30 720 NONE /u01/app/grid/local/logs/oswbb/archive

Process OSWatcherFM.sh is killed. Now wait for 1 minute and see what happens:
$ ps -ef | grep OSWatcher
 oragrid 14614732  6816014   0 15:21:11      -  0:00 /bin/sh ./OSWatcherFM.sh 720 /u01/app/grid/local/logs/oswbb/archive
 oragrid  6816014        1   0 15:20:42      -  0:00 /bin/sh ./OSWatcher.sh 30 720 NONE /u01/app/grid/local/logs/oswbb/archive
 oragrid 10355082  5046352   0 15:22:23  pts/6  0:00 grep OSWatcher
$

Wow! The processes are restarted.