OBIEE 11g startup issue :fails to start system components.

While starting system components using opmnctl, it fails with below error.

$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
/orasw/MW_HOME/Oracle_BI1/opmn/bin/opmn: unexpected exit: code 512
opmnctl startall: opmn failed to start.

Reason

Check OPMN logs to find the issue.

cd $INSTANCE_HOME/diagnostics/logs/OPMN/opmn

$ ls -lrt

total 10072

-rw-r----- 1 oracle oinstall       0 Sep 27  2011 debug.log

-rw-r----- 1 oracle oinstall     432 Jan 13 07:06 opmn.out

-rw-r----- 1 oracle oinstall  257891 Jan 22 00:32 logquery~5.log

-rw-r----- 1 oracle oinstall  269575 Jan 22 00:32 logquery~4.log

-rw-r----- 1 oracle oinstall  273531 Jan 22 00:32 logquery~3.log

-rw-r----- 1 oracle oinstall  279896 Jan 22 00:32 logquery~2.log

-rw-r----- 1 oracle oinstall  290644 Jan 22 00:32 logquery~1.log

-rw-r----- 1 oracle oinstall 8894259 Feb  5 00:41 opmn.log

In the log we found that OPMN Local Port and Remote ports are already in use by some other process.

[2013-02-04T04:48:36][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-04T04:49:50][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-05T00:41:07][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

Solution:

1. Go to $INSTANCE_HOME/config/OPMN//opmn.xml and check the ports.

In our case it was 9500 and 9501 ports

2. Check who is using these ports using below command.

$ netstat -an |grep 9500

tcp        0      0 ::ffff:10.241.37.9:9500     :::*                        LISTEN

tcp        0      0 ::ffff:10.241.37.9:9500     ::ffff:10.241.37.9:60741    ESTABLISHED

tcp        0      0 ::ffff:10.241.37.9:60741    ::ffff:10.241.37.9:9500     ESTABLISHED

$ netstat -an |grep 9501

tcp        0      0 :::9501                     :::*                        LISTEN

So both ports are being used.

3. Either terminate processes using these ports or change ports in opmn.xml with ports not in use.

In our case, 9502 and 9503 weren't in use so replaced with these ports in opmn.xml.

4.Start system components now.

cd $INSTANCE_HOME/bin

$ ./opmnctl startall

opmnctl startall: starting opmn and all managed processes...

$ ./opmnctl status

Processes in Instance: instance1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

coreapplication_obiccs1          | OracleBIClusterCo~ |   31392 | Alive

coreapplication_obisch1          | OracleBIScheduler~ |   31394 | Alive

coreapplication_obijh1           | OracleBIJavaHostC~ |   31393 | Alive

coreapplication_obips1           | OracleBIPresentat~ |   31391 | Alive

coreapplication_obis1            | OracleBIServerCom~ |   31395 | Alive