OWB Control Center Service Fails to Start after OWB cloning.

We cloned 10.2.0.2 database and OWB. When we tried to start the control center service after host name change in WB_RT_SERVICE_NODES(present in  repository owner schema) table, we got error as below

SQL> @start_service.sql
Not available
Diagnostics:
service startup failure using command "/orasw/app/oracle/product/owb10201/owb/b
in/unix/run_service.sh -manual 1 /orasw/app/oracle/product/owb10201 BIREPOWN sl
c04kvm.us.oracle.com 1521 BI221SF1" reason ORA-29532: Java call terminated by u
ncaught Java exception: java.io.IOException: service early exit: code=0 : err=
: out=2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - OCI fix performed

2013/03/24-23:30:44-PDT [1AC1FE4] Thin driver connection time - 126 millisecond
(s)
2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - property user.timezo
ne value is PST8PDT
2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - conn
ection.timezone not set. Defaulting to value SERVICE
2013/03/24-23:30:44-PDT [1

PL/SQL procedure successfully completed.

Reason:

The OS open file limit for the Oracle user is set either set to unlimited or less value. In our case it was

-bash-3.2$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 124064
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 124064
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Solution:

1.Change open files limit to 65536 in /etc/security/limits.conf for oracle user  with root as below

oracle     soft   nofile    131072
oracle    hard   nofile    65536

2. Login with new session and start the service.

-bash-3.2$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Mon Mar 25 00:48:26 2013

Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> conn birepown/birepown
Connected.
SQL> @stop_service.sql
Not Available

PL/SQL procedure successfully completed.

SQL> @start_service.sql
Available

PL/SQL procedure successfully completed.

SQL> @show_service.sql
Available

PL/SQL procedure successfully completed.

SQL> exit

Using network link to import data in Oracle

Using network link to import data in Oracle

1.Create database link ,in destination database, pointing to source database from where you want import the data as below.

In our case:
Source Schema =dwadm
Destination Schema=zz_dwadm

SQL> conn zz_dwadm/zz_dwadm
Connected.
SQL> create database link QA1DB_TO_DEVDB connect to dwadm identified by dwadm using 'QA1DB';

Database link created.

SQL> exit
Disconnected from Oracle

2.Import the table TEST_TABLE from dwadm into zz_dwadm using below command

impdp directory=dump_dir network_link=QA1DB_TO_DEVDB tables=dwadm.TEST_TABLE logfile=impdp.log remap_schema=dwadm:zz_dwadm


Import: Release 11.2.0.1.0 - Production on Wed Mar 13 09:40:30 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Username: zz_dwadm/zz_dwadm

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Error while connecting to OWB control center in RAC or EXADATA

We got below error while connecting to OWB repository  in EXADATA, from any node other than node from which we installed repository

2013/02/06:02:46    OMB01118: Could not connect to repository! ORA-29540: class oracle/wh/runtime/server/KeyManagement does not exist
ORA-06512: at "OWBSYS.GETROLESTARTED", line 31
ORA-06512: at line 1

2013/02/06:02:46    Unable to connect to repository.

Reason: 

You OWB repository is not configured properly. Probably, you might have missed to register other nodes in OWB while configuring repository. 

Solution: 

Refer the Doc ID 459961.1 for further details.

To cleanup OWB repository and creating OWBSYS Schema with Repository Objects

To cleanup OWB repository

Go to $ORACLE_HOME/owb/UnifiedRepos


cd /u01/app/oracle/product/11.2.0.3/dbhome_1/owb/UnifiedRepos

[oracle@slcm02db01 UnifiedRepos]$ ls -lrt clean_owbsys.sql
-rw-r--r-- 1 oracle oinstall 163 Sep 26  2008 clean_owbsys.sql

2.Connect to owbsys and Stop Oracle Warehouse Builder Control Center service.
SQL> conn owbsys/owbsys
connected
SQL> @?/owb/rtp/sql/stop_service.sql


3.Connect to database and execute the clean_owbsys.sql script to drop all the objects in the existing OWBSYSas

[oracle@slcm02db01 UnifiedRepos]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Mon Feb 4 00:36:03 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @clean_owbsys.sql
User dropped.


User dropped.


Role dropped.


Role dropped.


Role dropped.

SQL>

Creating an OWBSYS Schema with Repository Objects

1.connect to the database as sysdba and run the cat_owb.sql script to set up OWBSYS in a tablespace in your database.
For example, this command sets up OWBSYS in the USERS tablespace

SQL> @?/owb/UnifiedRepos/cat_owb.sql USERS
Enter Tablespace Name for OWBSYS user:

Package created.


Package body created.

Create user OWBSYS with default tablespace USERS ...
old   1: CREATE USER OWBSYS IDENTIFIED BY OWBSYS DEFAULT TABLESPACE &owbsys_tbs
new   1: CREATE USER OWBSYS IDENTIFIED BY OWBSYS DEFAULT TABLESPACE USERS

User created.

User OWBSYS has been created.

PL/SQL procedure successfully completed.


Grant succeeded.

Configuring the Control Center Service

If you are installing OWB 11.2 on an older Oracle Database release, or if you
integrating OWB with Oracle Business Intelligence Standard Edition , you must run the reset_owbcc_home.sql script to ensure that the
Control Center Service functions correctly. Steps are as below.

1.Connect to the database as sysdba and run the reset_owbcc_home.sql script to force the Control Center Service to
run from Oracle Warehouse Builder 11.2 installation.

SQL> @?/owb/UnifiedRepos/reset_owbcc_home.sql
Enter the full path of the Oracle home for the OWB Control Center install.
If you are installing in a Windows environment, please ensure that the case of the path exactly matches the Oracle install path including the drive letter.

OWB Control Center Home:
/u01/app/oracle/product/11.2.0.3/dbhome_1/owb

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.

SQL>

Unlocking the OWBSYS and OWBSYS_AUDIT Accounts

connect to the database as sysdba and execute below commands.

alter user owbsys identified by password account unlock;
alter user owbsys_audit identified by password account unlock;

Setting the OWB Home for Remote Installations
==============================================
You must create a database directory OWB_REMOTE_ADMIN that is accessed by remote
installations, so they can read the server's OWB_HOME/owb/bin/admin directory.

Connect to the database as sysdba and set the OWB_REMOTE_HOME variable using script remote_owb_install.sql as below

SQL> @?/owb/UnifiedRepos/remote_owb_install.sql
Oracle Home for the Remote OWB software installation
/u01/app/oracle/product/11.2.0.3/dbhome_1

Directory created.


Grant succeeded.


Grant succeeded.

SQL>

Creating the First Workspace in the Repository

To use a repository, you must define an initial workspace, and the owner of that
workspace. The following instructions use the Repository Assistant.

To create a workspace:
1. At the command prompt, navigate to the UNIX bin directory:
cd $ORACLE_HOME/owb/bin/unix

2. Start the repository assistant.
 ./reposinst.sh

3. On the Welcome screen of the Repository assistant, click Next and follow the instructions.

OBIEE 11g startup issue :fails to start system components.

While starting system components using opmnctl, it fails with below error.

$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
/orasw/MW_HOME/Oracle_BI1/opmn/bin/opmn: unexpected exit: code 512
opmnctl startall: opmn failed to start.

Reason

Check OPMN logs to find the issue.

cd $INSTANCE_HOME/diagnostics/logs/OPMN/opmn

$ ls -lrt

total 10072

-rw-r----- 1 oracle oinstall       0 Sep 27  2011 debug.log

-rw-r----- 1 oracle oinstall     432 Jan 13 07:06 opmn.out

-rw-r----- 1 oracle oinstall  257891 Jan 22 00:32 logquery~5.log

-rw-r----- 1 oracle oinstall  269575 Jan 22 00:32 logquery~4.log

-rw-r----- 1 oracle oinstall  273531 Jan 22 00:32 logquery~3.log

-rw-r----- 1 oracle oinstall  279896 Jan 22 00:32 logquery~2.log

-rw-r----- 1 oracle oinstall  290644 Jan 22 00:32 logquery~1.log

-rw-r----- 1 oracle oinstall 8894259 Feb  5 00:41 opmn.log

In the log we found that OPMN Local Port and Remote ports are already in use by some other process.

[2013-02-04T04:48:36][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-04T04:49:50][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-05T00:41:07][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

Solution:

1. Go to $INSTANCE_HOME/config/OPMN//opmn.xml and check the ports.

In our case it was 9500 and 9501 ports

2. Check who is using these ports using below command.

$ netstat -an |grep 9500

tcp        0      0 ::ffff:10.241.37.9:9500     :::*                        LISTEN

tcp        0      0 ::ffff:10.241.37.9:9500     ::ffff:10.241.37.9:60741    ESTABLISHED

tcp        0      0 ::ffff:10.241.37.9:60741    ::ffff:10.241.37.9:9500     ESTABLISHED

$ netstat -an |grep 9501

tcp        0      0 :::9501                     :::*                        LISTEN

So both ports are being used.

3. Either terminate processes using these ports or change ports in opmn.xml with ports not in use.

In our case, 9502 and 9503 weren't in use so replaced with these ports in opmn.xml.

4.Start system components now.

cd $INSTANCE_HOME/bin

$ ./opmnctl startall

opmnctl startall: starting opmn and all managed processes...

$ ./opmnctl status

Processes in Instance: instance1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

coreapplication_obiccs1          | OracleBIClusterCo~ |   31392 | Alive

coreapplication_obisch1          | OracleBIScheduler~ |   31394 | Alive

coreapplication_obijh1           | OracleBIJavaHostC~ |   31393 | Alive

coreapplication_obips1           | OracleBIPresentat~ |   31391 | Alive

coreapplication_obis1            | OracleBIServerCom~ |   31395 | Alive

OWB repository creation fails with error as ORA-29540: class oracle/wh/runtime/server/Util does not exist

OWB repository creation fails with error as ORA-29540: class oracle/wh/runtime/server/Util does not exist

Solution:

It seems that you must have set wrong OWB Control Center Home so reset using reset_owbcc_home.sql script located at $ORACLE_HOME/owb/UnifiedRepos as below.

SQL> @reset_owbcc_home.sql
Enter the full path of the Oracle home for the OWB Control Center install.
If you are installing in a Windows environment, please ensure that the case of the path exactly matches the Oracle install path including the drive letter.

OWB Control Center Home:
/u01/app/oracle/product/11.2.0.3/dbhome_1/owb

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.

SQL>
SQL> !pwd
/u01/app/oracle/product/11.2.0.3/dbhome_1/owb/UnifiedRepos

SQL> SHO USER;
USER is "SYS"

Goldengate: ggsci: error while loading shared libraries: libnnz11.so: cannot open shared object file

We got below error while connecting to GG after installing it.

[gbuora@slc03qmb GG]$ ./ggsci
./ggsci: error while loading shared libraries: libnnz11.so: cannot open shared object file: No such file or directory

Solution:

1. Go to GG installed directory

eg: cd /scratch/GG

2. create softlink for below files located at $ORACLE_HOME/lib as below.

libnnz11.so
libclntsh.so.11.1

ln -s /scratch/app/gbuora/product/11.2.0/dbhome_1/lib/libnnz11.so
ln -s /scratch/app/gbuora/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1


lrwxrwxrwx 1 gbuora dba        59 Feb  3 23:07 libnnz11.so -> /scratch/app/gbuora/product/11.2.0/dbhome_1/lib/libnnz11.so
lrwxrwxrwx 1 gbuora dba        65 Feb  3 23:08 libclntsh.so.11.1 -> /scratch/app/gbuora/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

3. Connect to GG now.

[gbuora@slc03qmb GG]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.1 OGGCORE_11.2.1.0.1_PLATFORMS_120423.0230_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr 23 2012 08:32:14

Copyright (C) 1995, 2012, Oracle and/or its affiliates. All rights reserved.



GGSCI (slc03qmb) 1> exit
[gbuora@slc03qmb GG]$ pwd
/scratch/GG