Tuesday, 5 February 2013

OBIEE 11g startup issue :fails to start system components.

While starting system components using opmnctl, it fails with below error.

$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
/orasw/MW_HOME/Oracle_BI1/opmn/bin/opmn: unexpected exit: code 512
opmnctl startall: opmn failed to start.


Check OPMN logs to find the issue.

cd $INSTANCE_HOME/diagnostics/logs/OPMN/opmn

$ ls -lrt
total 10072
-rw-r----- 1 oracle oinstall       0 Sep 27  2011 debug.log
-rw-r----- 1 oracle oinstall     432 Jan 13 07:06 opmn.out
-rw-r----- 1 oracle oinstall  257891 Jan 22 00:32 logquery~5.log
-rw-r----- 1 oracle oinstall  269575 Jan 22 00:32 logquery~4.log
-rw-r----- 1 oracle oinstall  273531 Jan 22 00:32 logquery~3.log
-rw-r----- 1 oracle oinstall  279896 Jan 22 00:32 logquery~2.log
-rw-r----- 1 oracle oinstall  290644 Jan 22 00:32 logquery~1.log
-rw-r----- 1 oracle oinstall 8894259 Feb  5 00:41 opmn.log

In the log we found that OPMN Local Port and Remote ports are already in use by some other process.

[2013-02-04T04:48:36][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states
[2013-02-04T04:49:50][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.
[2013-02-04T04:49:50][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states
[2013-02-05T00:41:07][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.
[2013-02-05T00:41:07][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)


1. Go to $INSTANCE_HOME/config/OPMN//opmn.xml and check the ports.
In our case it was 9500 and 9501 ports

2. Check who is using these ports using below command.

$ netstat -an |grep 9500
tcp        0      0 ::ffff:     :::*                        LISTEN
tcp        0      0 ::ffff:     ::ffff:    ESTABLISHED
tcp        0      0 ::ffff:    ::ffff:     ESTABLISHED

$ netstat -an |grep 9501
tcp        0      0 :::9501                     :::*                        LISTEN

So both ports are being used.

3. Either terminate processes using these ports or change ports in opmn.xml with ports not in use.

In our case, 9502 and 9503 weren't in use so replaced with these ports in opmn.xml.

4.Start system components now.


$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...

$ ./opmnctl status

Processes in Instance: instance1
ias-component                    | process-type       |     pid | status
coreapplication_obiccs1          | OracleBIClusterCo~ |   31392 | Alive
coreapplication_obisch1          | OracleBIScheduler~ |   31394 | Alive
coreapplication_obijh1           | OracleBIJavaHostC~ |   31393 | Alive
coreapplication_obips1           | OracleBIPresentat~ |   31391 | Alive
coreapplication_obis1            | OracleBIServerCom~ |   31395 | Alive


  1. interesting piece of information, I had come to know about your web-page from my friend pramod, jaipur,i have read atleast eight posts of yours by now, and let me tell you, your blog gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you! Thanks a million once again, Regards,obiee online training

  2. Appreciation for nice Updates, I found something new and folks can get useful info about BEST obiee ONLINE TRAINING

  3. Hai From the above post I understood that you had a good knowledge on Oracle 11g DBA.. I too had good knowledge on Oracle 11g DBA because i have Oracle 11g DBA online training in 123trainings at hyderabad.. They will provide excellent training.

  4. very useful really good information thanks for posting such a good information it will hepls the people a lot keep it up , Regards, obiee training in hyderabad

  5. I appreciate you sharing this article. Really thank you! Much obliged.
    This is one awesome blog article. Much thanks again.

    oracle online training
    sap sd online training
    hadoop online training

  6. Pretty good post. I just came across your site and wanted to say that I’ve really enjoyed reading your posts. In any case I’ll be subscribing to your feed and I hope you will keep a good work!Cheer!

    sap online training
    sap fico online training
    dotnet online training

  7. I was reading your blog this morning and noticed that you have a awesome
    resource page. I actually have a similar blog that might be helpful or useful
    to your audience.

    sap sd and crm online training
    sap online tutorials
    sap sd tutorial
    sap sd training in ameerpet