While starting system components using opmnctl, it fails with below error.
$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
/orasw/MW_HOME/Oracle_BI1/opmn/bin/opmn: unexpected exit: code 512
opmnctl startall: opmn failed to start.
Reason
Check OPMN logs to find the issue.
cd $INSTANCE_HOME/diagnostics/logs/OPMN/opmn
$ ls -lrt
total 10072
-rw-r----- 1 oracle oinstall 0 Sep 27 2011 debug.log
-rw-r----- 1 oracle oinstall 432 Jan 13 07:06 opmn.out
-rw-r----- 1 oracle oinstall 257891 Jan 22 00:32 logquery~5.log
-rw-r----- 1 oracle oinstall 269575 Jan 22 00:32 logquery~4.log
-rw-r----- 1 oracle oinstall 273531 Jan 22 00:32 logquery~3.log
-rw-r----- 1 oracle oinstall 279896 Jan 22 00:32 logquery~2.log
-rw-r----- 1 oracle oinstall 290644 Jan 22 00:32 logquery~1.log
-rw-r----- 1 oracle oinstall 8894259 Feb 5 00:41 opmn.log
In the log we found that OPMN Local Port and Remote ports are already in use by some other process.
[2013-02-04T04:48:36][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready. Request handling enabled.
[2013-02-04T04:49:50][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready. Request handling enabled.
[2013-02-05T00:41:07][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)
Solution:
1. Go to $INSTANCE_HOME/config/OPMN//opmn.xml and check the ports.
In our case it was 9500 and 9501 ports
2. Check who is using these ports using below command.
$ netstat -an |grep 9500
tcp 0 0 ::ffff:10.241.37.9:9500 :::* LISTEN
tcp 0 0 ::ffff:10.241.37.9:9500 ::ffff:10.241.37.9:60741 ESTABLISHED
tcp 0 0 ::ffff:10.241.37.9:60741 ::ffff:10.241.37.9:9500 ESTABLISHED
$ netstat -an |grep 9501
tcp 0 0 :::9501 :::* LISTEN
So both ports are being used.
3. Either terminate processes using these ports or change ports in opmn.xml with ports not in use.
In our case, 9502 and 9503 weren't in use so replaced with these ports in opmn.xml.
4.Start system components now.
cd $INSTANCE_HOME/bin
$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
$ ./opmnctl status
Processes in Instance: instance1
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
coreapplication_obiccs1 | OracleBIClusterCo~ | 31392 | Alive
coreapplication_obisch1 | OracleBIScheduler~ | 31394 | Alive
coreapplication_obijh1 | OracleBIJavaHostC~ | 31393 | Alive
coreapplication_obips1 | OracleBIPresentat~ | 31391 | Alive
coreapplication_obis1 | OracleBIServerCom~ | 31395 | Alive