Pages

Monday, January 5, 2015

Got 'Not enought space' while 'emctl stop oms'

Got 'Not enought space' message while stopping oms (OEM Cloud Control 12.1.0.4 running on Solaris 10 Sparc 64 bit):
$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
Error Occurred: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
Please check /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

Check log for details:
$ tail /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details
2015-01-02 14:00:35,466 [main] ERROR wls.OMSController main.253 - OMSController failed for stop oms
2015-01-02 14:00:35,469 [main] ERROR wls.OMSController main.254 - OMSController Error: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmn
ctl": error=12, Not enough space
java.io.IOException: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:369)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1228)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1217)
at oracle.sysman.emctl.util.WebTierUtil.stopWebTier(WebTierUtil.java:430)
at oracle.sysman.emctl.commands.StopCommand.stopOMS(StopCommand.java:204)
at oracle.sysman.emctl.commands.StopCommand.execute(StopCommand.java:114)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:234)
Caused by: java.io.IOException: error=12, Not enough space
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 9 more

No extra helpful message was found. What kind of space is not enough, OMS Home, /tmp or SWAP?
$ df -k
Filesystem kbytes used avail capacity Mounted on
/devices 0 0 0 0% /devices
... ...
swap 8609640 4896048 3713592 57% /tmp
swap 3713656 64 3713592 1% /var/run
rootpool/oracle 209715200 102092072 107623127 49% /u01
... ...
$ swap -s
total: 33445792k bytes allocated + 2772072k reserved = 36217864k used, 3711289k available

There is >90GB (102092072k) space free for mount point /u01 which can be used by OMS instance, and >3GB(3713592k) space left for /tmp.
It is not reasonable to say the space is not enough. However, is 3.6GB (3711289k) available (free) SWAP space enough for stopping OMS?
I'd like to say yes unless I found MOS Doc 1452151.1. The document is talking about same issues happened while OMS 12c is being started and
mentions 4-5GB of minimum free SWAP space for starting OMS. Could it be same requirement for stopping OMS? Make sense?

Before we enlarge SWAP, we found that /tmp and /var/run are sharing SWAP space. Here, we can ignore /var/run because it is tiny (64k).

Tried to delete garbage files from /tmp and check available SWAP:
$ swap -s
total: 31272184k bytes allocated + 2552208k reserved = 33824392k used, 6271264k available

Now, we have almost 6G SWAP available space and re-run 'emctl stop oms', it works!

Refrerences:

EM 12c Emctl Start OMS Returns Error Occurred: Cannot Run Program "<Middleware_Home>/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not Enough Space (Doc ID 1452151.1)

No comments: