Showing posts with label Enterprise Manager. Show all posts
Showing posts with label Enterprise Manager. Show all posts

Sunday, November 18, 2018

Patch or Upgrade Single instance Database and Oracle Restart Using OEM 13c Fleet Maintenance

Oracle Enterprise Manager (OEM) 13c Database Fleet Maintenance is a centralized mechanism for patching/upgrading of Oracle homes and maintain consistency. It allows OEM Cloud Control administrators to patch database, Oracle Restart (Standalone Grid Infrastructure) or Grid Infrastructure with minimal downtime.

Tuesday, August 14, 2018

Restrict OEM agent 13.2 to use TLS protocol TLSv1.2

Oracle Enterprise Manager 13.2 agent, by default, is configured to use TLS protocol TLSv1.0, TLSv1.1 and TLSv1.2,all versions of TLS when OEM 13.2 released. To restrict agent to use only TLSv1.2 as following,

Thursday, March 8, 2018

Manually Delete Agent and Targets Using EMCLI in OEM 13c

Although it is recommended to remove targets through OEM console interface, command line utility is more helpful/convenient in some scenarios.

For example, the target host is crashed and cannot be restored, and a new host with new name is built for replacement. Therefore,
the old host and all targets running on this host have to be remove from OEM. Depends on number of targets, it could be a pain to remove
them through console. At this time, manually running EMCLI command becomes the best way.

Following command removed the agent running on server host01 and all targets monitored by this agent,
[oracle@oms1]$ $EMCLI_HOME/bin/emcli login -username=sysman
Enter password :

Login successful
[oracle@oms1]$ $EMCLI_HOME/bin/emcli delete_target -name="host01.dbaplus.ca:3872" -type="oracle_emd" -delete_monitored_targets -async
Target "host01.dbaplus.ca:3872:oracle_emd" deleted successfully
[oracle@oms1]$


We can also run EMCLI command to remove a specific target,
$EMCLI_HOME/bin/emcli delete_target -name="<target-name>" -type="<target-type>"


To remove database (instance)  TESTDB,
$EMCLI_HOME/bin/emcli delete_target -name="TESTDB" -type="oracle_database"


To remove ASM instance target +ASM_host01.dbaplus.ca,
$EMCLI_HOME/bin/emcli delete_target -name="+ASM_host01.dbaplus.ca" -type="osm_instance"


To Oracle listener LISTENER_host01.dbaplus.ca,
$EMCLI_HOME/bin/emcli delete_target -name="LISTENER_host01.dbaplus.ca" -type="oracle_listener"

Saturday, October 28, 2017

Alert Log Filter Expression Editable when view-mode in Monitoring Template of OEM 13.2

OMS 13.2.0.0.171017
DB Plug-in 13.2.2.0.170930
EM Agent 13.2.0.0.170930

A customer asked for helping to apply patch 24334660 on the Enterprise Manager 13.2. Before starting patching, check the Oracle Home patching level as following,
[oracle@oms1]$ /u01/app/oracle/product/em/middleware/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 13.9.1.3.0
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/em/middleware
Central Inventory : /u01/app/oracle/oraInventory
   from           : /u01/app/oracle/product/em/middleware/oraInst.loc
OPatch version    : 13.9.1.3.0
OUI version       : 13.9.1.0.0
OUI location      : /u01/app/oracle/product/em/middleware/oui
Log file location : /u01/app/oracle/product/em/middleware/cfgtoollogs/opatch/opatch2017-10-28_10-43-03AM_1.log


OPatch detects the Middleware Home as "/u01/app/oracle/product/em/middleware"

<<Output truncated>>

Patch  26625183     : applied on Fri Oct 20 16:32:23 EDT 2017
Unique Patch ID:  21574333
Patch description:  "EMBP Patch Set Update 13.2.0.0.171017"
   Created on 8 Oct 2017, 20:17:42 hrs PST8PDT
   Bugs fixed:
     26782785, 25341210, 26090681, 26680573, 25110496, 24953203, 24459731
     26364865, 25861385, 26027847, 26542313, 20511097, 25967196, 25230688
     24701883, 25326446, 25217253, 25534065, 23506806, 25244726, 25179524
     25136456, 22143858, 25354849, 25615522, 25674055, 25219896, 25255429
     23251874, 24334660, 24944461, 23743109, 24426862, 25701178, 25268183
     23549064, 25506784, 25497731, 25497622, 24789016, 24742172, 25355038
     25163555, 25044101, 24608207, 23619694, 24753427, 24734911, 23705752
     24917276, 24614251, 22901546, 23196054, 21470098

<<Output truncated>>


Apparently, the OMS home has been applied PSU 13.2.0.0.171017, which already included the fix of the bug. Customer performs following steps to reproduce his issue,

In Monitoring Templates window, click the name of desired template and get into template details window, then click the Glasses icon in 'Edit' column besides metric 'Generic Alert Log Error'. In the new windows, the 'Alert Log Filter Expression' can be changed, but the new value cannot be saved (retained) after click OK. The value goes back to old value when come into this window again. It seems a bug/defect here. Actually, the window is a view-mode (read only) window with title 'View Advanced Settings: Generic Alert Log Error'. Yes, it gives you opportunity to type words in view-mode window, but does not change anything, a funny behaviour.

It is not a correct way to edit a metric in a monitoring template by clicking the template name. Instead, select the record (row) of the template and click 'Edit' button. In 'Edit Monitoring Template' window, the metric can be changed permanently.

Tuesday, August 15, 2017

Birthday Attacks against TLS ciphers Used by OEM 13.2 Agent

Oracle Enterprise Manager 13.2 uses following cipher suites as default value for SSL communication,
 SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA and SSL_RSA_WITH_3DES_EDE_CBC_SHA
 
Which support ciphers as listed by script CipherScan.bsh (script details can be found at the end of this post),
[oracle@host01]$ ./CipherScan.bsh host01.dbaplus.ca:3872
 Following Cipher(s) is/are supported on server host01.dbaplus.ca:3872
     ECDHE-RSA-AES128-SHA256
     ECDHE-RSA-AES128-SHA
     DHE-RSA-AES128-SHA256
     DHE-RSA-AES128-SHA
     AES128-SHA256
     AES128-SHA
     DES-CBC3-SHA

Here, cipher DES-CBC3-SHA could introduce a security threat of 'Birthday Attack against TLS ciphers with 64bit block size vulnerability'. Remote attackers can obtain cleartext data via a birthday attack against a long-duration encrypted session of all versions of SSL/TLS protocol supporting cipher suites which use DES, 3DES, IDEA or RC2 as the symmetric encryption cipher in CBC mode.

It can be disabled/removed by configuring agent's properties. Before changing the agent property, one of following method can be used to check current setting,
* Run command: <AGENT_INST_HOME>/bin/emctl getproperty agent -name SSLCipherSuites or
* Check agent property file by running:  grep -i <AGENT_INST_HOME>/sysman/config/emd.properties
 
[oracle@host01]$ cd /u01/software/em/agent/agent_inst
[oracle@host01]$ ./bin/emctl getproperty agent -name SSLCipherSuites
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SSLCipherSuites is unset; default value is SSL_RSA_WITH_RC4_128_MD5:SSL_RSA_WITH_RC4_128_SHA:SSL_RSA_WITH_3DES_EDE_CBC_SHA
[oracle@host01]$
[oracle@host01]$ grep -i 'SSLCipherSuites' ./sysman/config/emd.properties
[oracle@host01]$

The property is using default value and not set yet.
Remove weak cipher suite by setting SSLCipherSuites property of agent,
[oracle@host01]$ cd /u01/software/em/agent/agent_inst                      
[oracle@host01]$ ./bin/emctl setproperty agent -name SSLCipherSuites -value TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:RSA_WITH_AES_256_CBC_SHA256
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
EMD setproperty succeeded

If the Agent is running on AIX platform, use the cipher SSL_RSA_WITH_AES_128_CBC_SHA only.
[oracle@host01]$ ./bin/emctl setproperty agent -name SSLCipherSuites -value SSL_RSA_WITH_AES_128_CBC_SHA


Check current value,
[oracle@host01]$ cd /u01/software/em/agent/agent_inst
[oracle@host01]$ ./bin/emctl getproperty agent -name SSLCipherSuites
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SSLCipherSuites=TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:RSA_WITH_AES_256_CBC_SHA256
[oracle@host01]$
[oracle@host01]$ grep -i ./sysman/config/'SSLCipherSuites' emd.properties
SSLCipherSuites=TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:RSA_WITH_AES_256_CBC_SHA256
[oracle@host01]$

Restart agent,
[oracle@host01]$ /u01/software/em/agent/agent_13.2.0.0.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
[oracle@host01]$
[oracle@host01]$ /u01/software/em/agent/agent_13.2.0.0.0/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Starting agent ............... started.


Verify the supported ciphers,
[oracle@host01]$ ./CipherScan.bsh host01.dbaplus.ca:3872
 Following Cipher(s) is/are supported on server host01.dbaplus.ca:3872
     AES128-SHA

There is no threat (weak cipher) any more.
 
Appendex.  Script file CipherScan.bsh
#!/usr/bin/bash
# -----------------------------------------------
# Scan available Cipher on given server with port
# Usage:
#    CipherScan.bsh  <SERVER_IP>:<PORT>
#
# For example,
#    ./CipherScan.bsh  host01.dbaplus.ca:3872
# -----------------------------------------------
SERVER=$1
CIPHER_SUPPORTED=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')
echo -e "\n Following Cipher(s) is/are supported on server $SERVER \n"
for cipher in ${CIPHER_SUPPORTED[@]}
do
    result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1)
    if [[ "$result" =~ "Connection refused" ]] ; then
        echo  $result
        break
    fi
    if ! [[ "$result" =~ ":error:" ]] ; then
          echo "     "$cipher
    fi
done

Thursday, January 26, 2017

OMS version not checked yet while emctl upload agent

After OEM managed host was bounced, the agent status shows 'DOWN' in OEM. It happened because old version agent which pointed to old OMS server was started.

Find out if OEM agent is running on target,
$ ps -ef | grep emagent
  oracle 63111344 41484628   0 14:10:15  pts/0  0:00 grep emagent
  oracle  7143666        1   0 13:55:39  pts/4  0:00 /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/perl/bin/perl /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/bin/emwd.pl agent /u01/app/oracle/product/12.1.0/agent/agent_inst/sysman/log/emagent.nohup
  oracle 26280338  7143666   0 13:55:51  pts/4  0:06 /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/jdk/bin/java -Xmx160M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -Xgcpolicy:gencon -Dwatchdog.pid=7143666 -cp /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/jdbc/lib/ojdbc5.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/ucp/lib/ucp.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/modules/oracle.http_client_11.1.1.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/lib/xmlparserv2.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/lib/jsch.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/lib/optic.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/modules/oracle.dms_11.1.1/dms.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/modules/oracle.odl_11.1.1/ojdl.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/modules/oracle.odl_11.1.1/ojdl2.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/sysman/jlib/log4j-core.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/jlib/gcagent_core.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain

There is an instance of OEM agent running from Oracle Home '/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0'. Test the communication between managed host and the Management Service,
$ /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)

$ /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/bin/emctl pingOMS
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: Failure connecting to https://oem.dbaplus.ca:1159/empbs/upload , err Connection refused

Command 'emctl upload agent' returned error 'OMS version not checked yet', and 'emctl pingOMS' is complaining that OMS server cannot be connected. However, current OMS server is oms.dbaplus.ca and not oem.dbaplus.ca which is retired. Apparently, bouncing host automatically started an old (obsolete) agent which worked with old OMS server.

Let's fix it,
$ cat /etc/oragchomelist

/u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0:/u01/app/oracle/product/12.1.0/agent/agent_inst

/u01/app/oracle/product/em/agent/agent_13.2.0.0.0:/u01/app/oracle/product/em/agent/agent_inst

The first entry in file /etc/oragchomelist makes OS start old agent (version 12.1.0.4), remove it from /etc/oragchomelist and restart host. After host is up, check agent status,
$ cat /etc/oragchomelist
/u01/app/oracle/product/em/agent/agent_13.2.0.0.0:/u01/app/oracle/product/em/agent/agent_inst

$ ps -ef | grep emagent
  oracle 57082074 41484628   0 16:21:18  pts/0  0:00 grep emagent
  oracle 63766548        1   0 15:00:53  pts/0  0:00 /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/perl/bin/perl /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/bin/emwd.pl agent /u01/app/oracle/product/em/agent/agent_inst/sysman/log/emagent.nohup
  oracle 29950448 63766548   0 15:01:00  pts/0  1:20 /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/jdk/bin/java -Xmx183M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -Xgcpolicy:gencon -Dwatchdog.pid=63766548 -cp /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/jdbc/lib/ojdbc7.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/ucp/lib/ucp.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/jsch-0.1.53.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/com.oracle.http_client.http_client_12.1.3.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/oracle.xdk_12.1.3/xmlparserv2.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/oracle.dms_12.1.3/dms.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/oracle_common/modules/oracle.odl_12.1.3/ojdl2.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/lib/optic.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/sysman/jlib/log4j-core.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/jlib/gcagent_core.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oracle/product/em/agent/agent_13.2.0.0.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain

$ /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

$ /u01/app/oracle/product/em/agent/agent _13.2.0.0.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/product/em/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/product/em/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0
Core JAR Location      : /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/jlib
Agent Process ID       : 11665618
Parent Process ID      : 13238612
Agent URL              : https://host01.dbaplus.ca:3872/emd/main/
Local Agent URL in NAT : https://host01.dbaplus.ca:3872/emd/main/
Repository URL         : https://oms.dbaplus.ca:4903/empbs/upload
Started at             : 2017-01-26 14:25:33
Started by user        : oracle
Operating System       : AIX version 7.1 (ppc64)
Number of Targets      : 23
Last Reload            : (none)
Last successful upload                       : 2017-01-26 14:27:22
Last attempted upload                        : 2017-01-26 14:27:22
Total Megabytes of XML files uploaded so far : 0.62
Number of XML files pending upload           : 734
Size of XML files pending upload(MB)         : 0.31
Available disk space on upload filesystem    : 59.62%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2017-01-26 14:26:26
Last successful heartbeat to OMS             : 2017-01-26 14:26:26
Next scheduled heartbeat to OMS              : 2017-01-26 14:27:28

---------------------------------------------------------------
Agent is Running and Ready

Agent is running on managed host, and it also shows 'UP' in OEM.

Monday, September 21, 2015

Using Corrective Action to capture and email database status

As DBA, were you asked questions like:
I was blocked by other database sessions last night, could you please check who blocked me and what he was doing at that time?
My query ran into an temp tablespace space error, could you please check who was using large temporary space at that moment?
They are not hard questions, but you will find that it is not easy to answer them unless you have proactive consideration.

Here, I am going to show how easy it is to do that using Corrective Action in Oracle Enterprise Manager Cloud Control 12c.

1. Setting Up a Mail Server for Notifications
Log into OEM, from the [Setup] menu, select [Notifications], then select [Notification Methods]



2. Setting Up E-mail for Yourself
From [username] drop-down menu, select [Enterprise Manager Password & E-mail]. Click [Add Another Row] to create a new e-mail entry field in the E-mail Addresses table. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from  Enterprise Manager will be sent to the e-mail addresses you specify.



3. Set up a Notification Schedule 
Once you have defined your e-mail notification addresses, you will need to define a notification schedule.
From [Setup] menu, select [Notifications], then select [My Notification Schedule]. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails


4.  Creating Incident Rules for the events you want to monitor
From the [Setup] menu, select [Incidents], then select [Incident Rules]. Click [Create Rule Set...]


In [Create Rule Set] page,  specify the [Name], [Description], and the [Targets] to which the rules set should apply. Then click [Create...] to display [Select Type of Rule to Create] window, select [Incoming Events and updates to events], click [Continue].


In [Create New Rule: Select Events] page, select [Specific events of type Metric Alert], click [Add] to display [Select Specific Metric Alert] page, select [User Block] for metric and appropriate options for [Severity] and [Corrective action status], then click [Ok]



Back to [Create New Rule: Select Events] page, click [Next] to display [Create New Rule: Add Actions] page. Click [Add] to display
[Add Actions] page, specify your username for [E-mail To] Notificatios, then click [Continue] to return to [Create New Rule: Add Actions] page. Click [Next] to specify Name and Description for new rule, then click [Next] to display [Create New Rule: Review] page, click [Continue]



Back to [Create Rule Set] page, click [Save] to return to [Incident Rules - All Enterprise Rules] page.



5.     Create Corrective Actions

In database home page, from [Oracle Database] menu, select [Monitoring], then click [Metric and Collection Settings] to display [Metric and Collection Settings]

In [Metric and Collection Settings], click [Disabled] in the [Collection Schedule] field after metric [User Block] to enable metric collection. Then click edit image after metric [Blocking Session Count] to display [Edit Advanced Settings: Blocking Session Count].
In [Edit Advanced Settings: Blocking Session Count], select [All others] then click [Edit] button to display [Edit Advanced Settings: Blocking Session Count for All others]

In  [Edit Advanced Settings: Blocking Session Count for All others], enter 2 for [Warning Threshold] and 5 for [Critical Threshold], then click [Add] button beside [Warning].

In [Add Corrective Action] page, select [SQL Script] from dropbox then click [Continue] button to display [Create 'SQL Script' Corrective Action].

In [Create 'SQL Script' Corrective Action]
* Enter Name and Description in tab [Gerneral]
* Enter SQL statements in tab [Parameters], here, SQL statements are what you want to run immediately after the alert is generated.
* Enter database credential and host credential information.



Then click [Continue] button and back to [Edit Advanced Settings: Blocking Session Count] page, then click [Continue], then click [Continue] again and back to [Metric and Collection Settings], then click [Ok] button to save changes.

6.    Make sure Notification Email include message of Corrective Actions' status

From [Setup] menu, select [Notifications], then select [Customize Email Formats]


Click [Customize] to bring up following window and make sure Corrective Actions' section is there (It is default configuration)


Now, you will receive a email when blocked sessions are more than 2 sessions and the query result of SQL statements created in Corrective Action will be included in the email.


Thursday, July 2, 2015

Agent status shows 'Availability Evaluation Error' on EM 12c Console

Background
During deploying agent 12.1.0.4 to new host for EM 12.1.0.4, the local firewall was enabled by accident and blocked incoming access. The deployment is successful. However, OEM shows error:
'Unable to connect to the agent at https://host01.dbaplus.ca:3872/emd/main/ [Connection establishment timed out])'

After stopping firewall on the new host, agent status is shown 'Availability Evaluation Error' on EM 12c Console.
Diagnose
On the new host:
$ /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/product/12.1.0/agent/agent_inst
Agent Log Directory    : /u01/app/oracle/product/12.1.0/agent/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0
Agent Process ID       : 8366
Parent Process ID      : 8280
Agent URL              : https://host01.dbaplus.ca:3872/emd/main/
Local Agent URL in NAT : https://host01.dbaplus.ca:3872/emd/main/
Repository URL         : https://omshost.dbaplus.ca:1159/empbs/upload
Started at             : 2015-07-02 11:29:46
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.17.1.el6uek.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 21.46%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-07-02 11:30:49
Last successful heartbeat to OMS             : 2015-07-02 11:30:49
Next scheduled heartbeat to OMS              : 2015-07-02 11:31:49

---------------------------------------------------------------
Agent is Running and Ready

We can find that 'Last successful upload' is '(none)', it is supposed to be a valid date & time. Try to manually upload:
$ /u01/app/oracle/product/12.1.0/agent/core/12.1.0.4.0/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It does not help. 'Last successful upload' is still '(none)' and agent status is still shown 'Availability Evaluation Error'.
Try to re-synchronize the agent on OMS server:
$ . oraenv
ORACLE_SID = [oracle] ? OMS
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/12.1.0/OMS/oms
The Oracle base has been set to 
$ which emcli
/u01/app/oracle/product/12.1.0/OMS/oms/bin/emcli
$ 
$ emcli  login -username=SYSMAN
Enter password 

Login successful
$ emcli sync
Synchronized successfully
$ emcli resyncAgent -agent=co arp.corp.toronto.ca:3872
Resync job RESYNC_20150702142805361 successfully submitted

Now agent is shown UP on EM 12c console.
Reference
EM 12c: emctl status agent Agent 'unavailable with metric collection error' 'Cleanup of previous target with the same name () is in progress' Reported in MGMT_TARGETS (Doc ID 1950874.1)

Friday, February 13, 2015

Integrating BI Publisher with Enterprise Manager using the configureBIP Script

As we know, Oracle has deprecated Information Publisher (IP) Reports for Enterprise Manager 12c and introduce new report tool BI Publisher (BIP) Enterprise Reports. Here, I am going to configure BIP after OEM 12c (12.1.0.4) has been installed and used for a while.

Configure environment and set ORACLE_HOME to OMS installation path:
$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/oms12104/oms
$ export PATH=$ORACLE_HOME/bin:$PATH
$ which emctl
/u01/app/oracle/product/12.1.0/oms12104/oms/bin/emctl

Backup OEM using command: emctl exportconfig oms
$ emctl exportconfig oms -dir /u01/app/oracle/OMS_BACKUP -oms_only -keep_host
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
ExportConfig started...
Backup directory is /u01/app/oracle/OMS_BACKUP
Performing OMS backup...
Exporting emoms properties...
Exporting secure properties...

Export has determined that the OMS is not fronted
by an SLB. You have chosen to export the local
hostname. The exported data may ONLY be imported
on a host with the same hostname. Please see the
EM Advanced Configuration Guide for more details.

Warning: Could not export files for module bip: Unable to read file /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/embip.properties for exporting
Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /u01/app/oracle/OMS_BACKUP/opf_OMS_20150210_111433.bka

The export file contains sensitive data.
You must keep it secure.

ExportConfig completed successfully!

Configure BIP using script configureBIP
$ cd $ORACLE_HOME/bin
$
$ ./configureBIP
Configuring BI Publisher Version "11.1.1.7.0" to work with Enterprise Manager
Logging started at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log.
Before this command is run, a backup of Enterprise Manager should be performed using the :emctl exportconfig oms: command. Have you made a valid backup of Enterprise Manager (yes/no) [no] ? yes
Enter sysdba user name (sys):sys
Enter sysdba user password:
Enter Administration Server user password:
Configuring BI Publisher in Oracle Home located in /u01/app/oracle/product/12.1.0/oms12104/Oracle_BI1 ...
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 0
Percent Complete: 10
Percent Complete: 30
Percent Complete: 50
Percent Complete: 50
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host01.dbaplus.ca)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=OEMREP)))
Connected As : sys
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile : /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/emBIPLATFORM.log
Component schemas created:
Component Status Logfile
Business Intelligence Platform Success /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/biplatform.log

Repository Creation Utility - Create : Operation Completed
Successfully created SYSMAN_BIPLATFORM schema...
Enter an integer between 9701 and 49152 for the BI Publisher HTTP server port. (9701):
Enter an integer between 9702 and 49152 for the BI Publisher HTTPS server port. (9702):
Extending domain with BI Publisher. This operations can take some time. Do not interrupt this command while it is running...
Locking Enterprise Manager ...
OMS Console is locked. Access the console over HTTPS ports.
Restart OMS.
Restarting Enterprise Manager ...
Stopping Enterprise Manager, this can take some time ...
Failed to stop Enterprise Manager. Diagnostic code 1.
See log at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log for details.
Error extending domain
See log at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log for details.

OMG, a bad luck! Check log file for what happened,
$ tail -20 /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log

[2015-02-10T11:30:51.192-05:00] [sysman] [NOTIFICATION] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 17] [userId: oracle] [ecid: 0000KhoL3KH0rmjrh0r2E41KqYwd000004,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.ProcessOutputReader] [SRC_METHOD: run] Stopping WebTier...[[

]]
[2015-02-10T11:30:55.221-05:00] [sysman] [NOTIFICATION] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 17] [userId: oracle] [ecid: 0000KhoL3KH0rmjrh0r2E41KqYwd000004,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.ProcessOutputReader] [SRC_METHOD: run] Error Occurred: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space[[

]]
[2015-02-10T11:30:55.222-05:00] [sysman] [NOTIFICATION] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 17] [userId: oracle] [ecid: 0000KhoL3KH0rmjrh0r2E41KqYwd000004,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.ProcessOutputReader] [SRC_METHOD: run] Please check /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details[[

]]
[2015-02-10T11:30:55.265-05:00] [sysman] [ERROR] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 10] [userId: oracle] [ecid: 0000KhoI3ci0rmjrh0r2E41KqYwd000000,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA] [SRC_METHOD: lockAndBounceEM] Failed to stop Enterprise Manager. Diagnostic code 1.
[2015-02-10T11:30:55.273-05:00] [sysman] [ERROR] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 10] [userId: oracle] [ecid: 0000KhoI3ci0rmjrh0r2E41KqYwd000000,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA] [SRC_METHOD: extendDomainWithBIP] See log at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log for details.
[2015-02-10T11:30:55.274-05:00] [sysman] [ERROR] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 10] [userId: oracle] [ecid: 0000KhoI3ci0rmjrh0r2E41KqYwd000000,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA] [SRC_METHOD: handleFatalError] Error extending domain
[2015-02-10T11:30:55.275-05:00] [sysman] [ERROR] [] [oracle.sysman.bipca] [host: host01.dbaplus.ca] [nwaddr: 10.10.0.34] [tid: 10] [userId: oracle] [ecid: 0000KhoI3ci0rmjrh0r2E41KqYwd000000,0] [SRC_CLASS: oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA] [SRC_METHOD: handleFatalError] Fatal error:[[
java.lang.Exception: See log at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150210111722.log for details.
at oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA.extendDomainWithBIP(BIPCA.java:2370)
at oracle.sysman.sdkImpl.core.ip.model.bipca.BIPCA.main(BIPCA.java:1531)

At the begining of the error, I found messages: 'opmnctl', 'Not enough space', 'Failed to stop Enterprise Manager'. What a familiar look! Yes, it is familiar. Let's try to fix it following "Got ‘Not enought space’ while ‘emctl stop oms’". Then, restart the process. Of course, I have to clean/rollback the failed configuration first before re-running configureBIP.

Clean/rollback failed configureBIP run

Stop BIP server
$ emctl stop oms -bip_only
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
BI Publisher Server is Down

Unregister BIP from Enterprise Manager
$ emcli login -username=sysman
Enter password :

Login successful
$ emcli sync
Synchronized successfully

$ emcli unregister_bipublisher -force
Warning: BI Publisher was not registered for use with Enterprise Manager, no action taken.

BIP was not registered with Enterprise Manager yet, but it does not really matter, just go ahead.
Drop schema created by script configureBIP from Enterprise Manager repository database
$ sqlplus sys@OEMREP as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Fri Feb 13 08:39:01 2015

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters and Automatic Storage Management options

SQL> drop user SYSMAN_BIPLATFORM cascade;

User dropped.

SQL> select comp_id,COMP_NAME,owner,version,status from system.schema_version_registry where comp_id = 'BIPLATFORM';

COMP_ID COMP_NAME OWNER VERSION STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ -----------
BIPLATFORM OracleBI and EPM SYSMAN_BIPLATFORM 11.1.1.7.0 VALID

SQL> delete from system.schema_version_registry where comp_id = 'BIPLATFORM';

1 row deleted.

SQL> commit;

Commit complete.

SQL> select comp_id,COMP_NAME,owner,version,status from system.schema_version_registry where comp_id = 'BIPLATFORM';

no rows selected

Re-run script configureBIP to configure BIP with Enterprise Manager
$ ./configureBIP
Configuring BI Publisher Version "11.1.1.7.0" to work with Enterprise Manager
Logging started at /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/bipca_20150213084523.log.
Before this command is run, a backup of Enterprise Manager should be performed using the :emctl exportconfig oms: command. Have you made a valid backup of Enterprise Manager (yes/no) [no] ? yes
Enter sysdba user name (sys):
Enter sysdba user password:
Enter Administration Server user password:
Configuring BI Publisher in Oracle Home located in /u01/app/oracle/product/12.1.0/oms12104/Oracle_BI1 ...
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 0
Percent Complete: 10
Percent Complete: 30
Percent Complete: 50
Percent Complete: 50
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host01.dbaplus.ca)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=OEMREP)))
Connected As : sys
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile : /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/emBIPLATFORM.log
Component schemas created:
Component Status Logfile
Business Intelligence Platform Success /u01/app/oracle/product/12.1.0/oms12104/oms/cfgtoollogs/bip/biplatform.log

Repository Creation Utility - Create : Operation Completed
Successfully created SYSMAN_BIPLATFORM schema...
Enter an integer between 9701 and 49152 for the BI Publisher HTTP server port. (9701):
Enter an integer between 9702 and 49152 for the BI Publisher HTTPS server port. (9702):
Extending domain with BI Publisher. This operations can take some time. Do not interrupt this command while it is running...
Locking Enterprise Manager ...
OMS Console is locked. Access the console over HTTPS ports.
Restart OMS.
Restarting Enterprise Manager ...
Stopping Enterprise Manager, this can take some time ...
Starting Enterprise Manager. This operation can take some time. Do not interrupt this command while it is running.
OMS Started Successfully
BI Publisher server named :BIP: running at https://host01.dbaplus.ca:9702/xmlpserver.
Registering BI Publisher with Enterprise Manager and deploying reports...
Performing automatic backup of Enterprise Manager using the command :emctl exportconfig oms:.
Successfully backed up Enterprise Manager. The backup file is located in the INSTANCE_HOME sysman backup directory.
Successfully setup BI Publisher with Enterprise Manager

Now, we can use BI Publisher to do report instead of Information Publisher.

Tuesday, January 6, 2015

OEM Cloud Control 12c: OMS 12.1.0.4.1 PSU installation

OEM: Cloud Control 12.1.0.4
OS: Solaris 10 Sparc 64bit

Download PSU p19218456_121040_Generic.zip and OPatch p6880880_111000_SOLARIS64.zip from my oracle support.

1. Upgrade OPatch
$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 11.1.0.11.0

OPatch succeeded.
$ cd $ORACLE_HOME
$ pwd
/u01/app/oracle/product/12.1.0/em12104/oms

$ mv OPatch Opatch.11.1.0.11.0

$ unzip /u01/app/media/p6880880_111000_SOLARIS64.zip
... ...

$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 11.1.0.12.3

OPatch succeeded.

2. Inventory of Oracle Home before patching
$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.12.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/em12104/oms
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/em12104/oms/oraInst.loc
OPatch version : 11.1.0.12.3
OUI version : 11.1.0.12.0
Log file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/opatch2015-01-02_13-28-19PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/product/12.1.0/em12104"

Lsinventory Output file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/lsinv/lsinventory2015-01-02_13-28-19PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):

EM Platform (OMS) 12.1.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch 19136663 : applied on Fri Oct 03 11:00:22 EDT 2014
Unique Patch ID: 18088442
Created on 25 Sep 2014, 04:29:53 hrs PST8PDT
Bugs fixed:
19136663
--------------------------------------------------------------------------------
OPatch succeeded.

3. Ensure that we do not have any issues with configuration and binary prerequisite checks
$ unzip -d /u01/app/setup /u01/app/media/p19218456_121040_Generic.zip
... ...
$ cd /u01/app/setup/19218456
$ $ORACLE_HOME/OPatch/opatchauto apply -analyze
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation. All rights reserved.

OPatchauto version : 11.1.0.12.3
OUI version : 11.1.0.12.0
Running from : /u01/app/oracle/product/12.1.0/em12104/oms
Log file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/opatch2015-01-02_13-29-10PM_1.log

OPatchauto log file: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatchauto/19218456/opatch_oms_2015-01-02_13-29-11PM_analyze.log

Please enter OMS weblogic admin server URL(t3s://host01.dbaplus.ca:7101):>
Please enter OMS weblogic admin server username:> weblogic
Please enter OMS weblogic admin server password:>

Configuration Validation: Success

Running apply prerequisite checks for sub-patch(es) "19218456" and Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms"...
Please monitor OPatch log file: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/19218456_Jan_02_2015_13_29_10/ApplyPrereq2015-01-02_13-29-37PM_2.log
Sub-patch(es) "19218456" are successfully analyzed for Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms"

Complete Summary
================

All log file names referenced below can be accessed from the directory "/u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/2015-01-02_13-29-10PM_SystemPatch_19218456_1"

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

Oracle Home Name Sub-patches Log file
---------------- ----------- --------
oms12c1 19218456 19218456_ApplyPrereq2015-01-02_13-29-37PM_2.log

Log file location: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatchauto/19218456/opatch_oms_2015-01-02_13-29-11PM_analyze.log
OPatchauto succeeded.

4. Stop OMS
$ which emctl
/u01/app/oracle/product/12.1.0/em12104/oms/bin/emctl
$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping WebTier...
Error Occurred: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
Please check /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

There is not enough SWAP free space. Check article "Got ‘Not enought space’ while ‘emctl stop oms’" for details.
Step ahead after reclaiming SWAP space:
$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

5. Install PSU
$ cd /u01/app/setup/19218456
$ $ORACLE_HOME/OPatch/opatchauto apply
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation. All rights reserved.

OPatchauto version : 11.1.0.12.3
OUI version : 11.1.0.12.0
Running from : /u01/app/oracle/product/12.1.0/em12104/oms
Log file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/opatch2015-01-02_14-24-17PM_1.log

OPatchauto log file: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatchauto/19218456/opatch_oms_2015-01-02_14-24-18PM_deploy.log

Please enter OMS weblogic admin server URL(t3s://host01.dbaplus.ca:7101):>
Please enter OMS weblogic admin server username:> weblogic
Please enter OMS weblogic admin server password:>

Configuration Validation: Success

Running apply prerequisite checks for sub-patch(es) "19218456" and Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms"...
Please monitor OPatch log file: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/19218456_Jan_02_2015_14_24_17/ApplyPrereq2015-01-02_14-24-41PM_2.log
Sub-patch(es) "19218456" are successfully analyzed for Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms"

To continue, OPatchauto will do the following:
[Patch and deploy artifacts] : Apply sub-patch(es) [ 19218456 ] to Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms";
Apply RCU artifact with patch "/u01/app/oracle/product/12.1.0/em12104/oms/.patch_storage/19218456_Oct_14_2014_03_22_39/original_patch"

Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch "19218456" to Oracle Home "/u01/app/oracle/product/12.1.0/em12104/oms"...
Please monitor OPatch log file: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/19218456_Jan_02_2015_14_24_17/apply2015-01-02_14-25-16PM_4.log

Updating repository with RCU reference file "/u01/app/oracle/product/12.1.0/em12104/oms/.patch_storage/19218456_Oct_14_2014_03_22_39/original_patch"

Complete Summary
================

All log file names referenced below can be accessed from the directory "/u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/2015-01-02_14-24-17PM_SystemPatch_19218456_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

Oracle Home Name Sub-patches Log file
---------------- ----------- --------
oms12c1 19218456 19218456_apply2015-01-02_14-25-16PM_4.log

Deployment summary:
-------------------

The following artifact(s) have been successfully deployed:

Artifacts Log file
--------- --------
SQL rcu_applypatch_original_patch_2015-01-02_14-27-25PM.log


Log file location: /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatchauto/19218456/opatch_oms_2015-01-02_14-24-18PM_deploy.log

OPatchauto succeeded.

6 Start OMS
$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...

Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

7 Validate ORACLE_HOME
$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.12.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/em12104/oms
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/em12104/oms/oraInst.loc
OPatch version : 11.1.0.12.3
OUI version : 11.1.0.12.0
Log file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/opatch2015-01-02_14-33-53PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/product/12.1.0/em12104"

Lsinventory Output file location : /u01/app/oracle/product/12.1.0/em12104/oms/cfgtoollogs/opatch/lsinv/lsinventory2015-01-02_14-33-53PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

EM Platform (OMS) 12.1.0.4.0
There are 1 products installed in this Oracle Home.

Interim patches (2) :

Patch 19218456 : applied on Fri Jan 02 14:27:10 EST 2015
Unique Patch ID: 17982946
Patch description: "ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.1 PSU (GENERIC)"
Created on 14 Oct 2014, 03:22:39 hrs PST8PDT
Bugs fixed:
18599481, 17407918, 18932882, 18805935, 19258481, 19376082, 19323634
18938561, 17812648, 18725891, 18945007, 18721562, 18359621, 18490432
19218456, 18490468, 19137411, 17843191, 18710124, 18516298, 18490455
18256983, 18999400, 19329809, 19300103, 18351725, 18521731, 18485889
19458672, 18490444, 18446798, 18765545, 18389542, 18509136

Patch 19136663 : applied on Fri Oct 03 11:00:22 EDT 2014
Unique Patch ID: 18088442
Created on 25 Sep 2014, 04:29:53 hrs PST8PDT
Bugs fixed:
19136663

--------------------------------------------------------------------------------

OPatch succeeded.

8 References:

Readme - Oracle Enterprise Manager 12c Release 4 Patch Set Update 1 (12.1.0.4.1) for Oracle Management Service

Monday, January 5, 2015

Got 'Not enought space' while 'emctl stop oms'

Got 'Not enought space' message while stopping oms (OEM Cloud Control 12.1.0.4 running on Solaris 10 Sparc 64 bit):
$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
Error Occurred: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
Please check /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

Check log for details:
$ tail /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details
2015-01-02 14:00:35,466 [main] ERROR wls.OMSController main.253 - OMSController failed for stop oms
2015-01-02 14:00:35,469 [main] ERROR wls.OMSController main.254 - OMSController Error: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmn
ctl": error=12, Not enough space
java.io.IOException: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:369)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1228)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1217)
at oracle.sysman.emctl.util.WebTierUtil.stopWebTier(WebTierUtil.java:430)
at oracle.sysman.emctl.commands.StopCommand.stopOMS(StopCommand.java:204)
at oracle.sysman.emctl.commands.StopCommand.execute(StopCommand.java:114)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:234)
Caused by: java.io.IOException: error=12, Not enough space
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 9 more

No extra helpful message was found. What kind of space is not enough, OMS Home, /tmp or SWAP?
$ df -k
Filesystem kbytes used avail capacity Mounted on
/devices 0 0 0 0% /devices
... ...
swap 8609640 4896048 3713592 57% /tmp
swap 3713656 64 3713592 1% /var/run
rootpool/oracle 209715200 102092072 107623127 49% /u01
... ...
$ swap -s
total: 33445792k bytes allocated + 2772072k reserved = 36217864k used, 3711289k available

There is >90GB (102092072k) space free for mount point /u01 which can be used by OMS instance, and >3GB(3713592k) space left for /tmp.
It is not reasonable to say the space is not enough. However, is 3.6GB (3711289k) available (free) SWAP space enough for stopping OMS?
I'd like to say yes unless I found MOS Doc 1452151.1. The document is talking about same issues happened while OMS 12c is being started and
mentions 4-5GB of minimum free SWAP space for starting OMS. Could it be same requirement for stopping OMS? Make sense?

Before we enlarge SWAP, we found that /tmp and /var/run are sharing SWAP space. Here, we can ignore /var/run because it is tiny (64k).

Tried to delete garbage files from /tmp and check available SWAP:
$ swap -s
total: 31272184k bytes allocated + 2552208k reserved = 33824392k used, 6271264k available

Now, we have almost 6G SWAP available space and re-run 'emctl stop oms', it works!

Refrerences:

EM 12c Emctl Start OMS Returns Error Occurred: Cannot Run Program "<Middleware_Home>/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not Enough Space (Doc ID 1452151.1)