# srvctl config nodeapps -a

Network exists: 1/10.1.223.0/255.255.255.0/lan0, type static
VIP exists: /cruz-vip/10.1.223.73/10.1.223.0/255.255.255.0/lan0, hosting node cruz
VIP exists: /davis-vip/10.1.223.72/10.1.223.0/255.255.255.0/lan0, hosting node davis

Stop Instance,ASM,Nodeapps resources

$srvctl stop instance -d romeo -i romeo2
$srvctl stop asm -n romeo1
$srvctl stop nodeapps -n romeo1

Verify the VIP Address is no longer running by using below command

crsctl status  resource  -t

Modify VIP Address or VIP hostname on nodeapps by using srvctl command (Run as root)

#srvctl modify nodeapps -n [-o ] [-A ]
Where
-n < node_name> - Node name.
-o - Oracle Home for the cluster software (CRS-Home).
-A The node level VIP address (/netmask[/if1[|if2|...]]).

Example:- Modify the VIP Address to 10.1.223.72

#srvctl modify nodeapps -n davis -A 10.1.223.72/255.255.255.0/eth0

Modify VIP hostname from ”davis-vip” to “davis-v”

#srvctl modify nodeapps -n  -A davis-v/255.255.255.0/eth0

Verify the change by running below command

$srvctl config nodeapps -n davis –a

Start all resources

$srvctl start nodeapps -n davis
$srvctl start asm -n davis

$srvctl start instance -d romeo –i romeo1

1. List the nodes in your cluster by running the following command as root on one node:
cruz:# ./olsnodes
davis
cruz

2. Stop Oracle Clusterware by running the following command as root on all of the nodes:
# crsctl stop crs

If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as root on all of the nodes:
# crsctl stop crs -f

3. Run the following command to list the backup files:

ocrconfig -showbackup

cruz:# ./ocrconfig -showbackup
davis     2012/01/25 15:49:11     /u01/app/11.2.0.2/grid/cdata/ractestscan/backup00.ocr
davis     2012/01/25 11:49:09     /u01/app/11.2.0.2/grid/cdata/ractestscan/backup01.ocr
davis     2012/01/25 07:49:07     /u01/app/11.2.0.2/grid/cdata/ractestscan/backup02.ocr
davis     2012/01/24 11:49:00     /u01/app/11.2.0.2/grid/cdata/ractestscan/day.ocr
davis     2012/01/18 03:48:04     /u01/app/11.2.0.2/grid/cdata/ractestscan/week.ocr

4. If you are restoring OCR to a cluster file system or network file system, then run the following command as root to restore OCR

# ocrconfig -restore /u01/app/11.2.0.2/grid/cdata/ractestscan/backup00.ocr

5. Begin to start Oracle Clusterware by running the following command as root on all of the nodes:
# crsctl start crs

6. Verify OCR integrity of all of the cluster nodes that are configured as part of your cluster by running the following CVU command:

# cluvfy comp ocr -n all -verbose

RESTORING VOTING DISK

1.  Make sure that the OCR is restored and valid using ocrcheck
2. Run the following command as root from only one node to start the Oracle Clusterware stack in exclusive mode, which does not require voting files to be present or usable:
# crsctl start crs -excl

3. Run the crsctl query css votedisk command to retrieve the list of voting files currently defined, similar to the following:

## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE a9f61832112b4f33bf98988caac790f6 (/u02/ocr/vdsk) []
Located 1 voting disk(s).

4. run the following command using the File Universal Identifier (FUID) obtained in the previous step:

$ crsctl delete css votedisk FUID

5.  Add a voting disk, as follows:

$ crsctl add css votedisk path_to_voting_disk

6. Stop the Oracle Clusterware stack as root:
# crsctl stop crs
If the Oracle Clusterware stack is running in exclusive mode, then use the -f option to force the shutdown of the stack.

7. Restart the Oracle Clusterware stack in normal mode as root:
# crsctl start crs

***********************************************************************
NI cryptographic checksum mismatch error: 12599.

VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 14-JUL-2010 11:39:22
Tracing not turned on.
Tns error struct:
ns main err code: 12599

TNS-12599: TNS:cryptographic checksum mismatch
ns secondary err code: 2526
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Wed Jul 14 11:48:14 2010
***********************************************************************

Workaround:

on the target database side, set SQLNET.ENCRYPTION_TYPES_SERVER= (3DES168)  in sqlnet.ora

Patches for bug 9953045 and  bug 12360281 fix this problem.

- bug 9953045   is to fix the issue for repository connection made from OMS
- bug 12360281 is to fix target database connections made from OMS

Download patches from Oracle

p9953045_111010_Generic.zip

p12360281_111010_Generic.zip

Starting background process RSMN
Fri Dec 16 16:55:40 2011
RSMN started with pid=28, OS id=24795
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Fri Dec 16 16:55:40 2011
ALTER DATABASE MOUNT /* db agent *//* {4:8516:36317} */

to fix the error run the following command using srvctl

srvctl setenv database -d romeo -t “ORACLE_BASE=/u01/app/oracle”

Once that was executed, I bounced this particular instance using:

srvctl stop instance -d romeo -i romeo2
srvctl start instance -d romeo -i romeo2

this resolved the issue

Starting background process RSMN
Wed Jan 25 12:39:47 2012
RSMN started with pid=28, OS id=16293
ORACLE_BASE from environment = /u01/app/oracle
Wed Jan 25 12:39:48 2012
ALTER DATABASE MOUNT /* db agent *//* {2:43873:47317} */

4 nodes Oracle 11.2.0.2 on HP-UX 11.31 hostA1 hostB1 hhostC1 hostD1 database romeo

Target

4 nodes Oracle 11.2.0.2 on HP-UX 11.31  hostA2 hostB2 hostC2 hostD2  database Juliet

1. On the source host, backup the source using RMAN. Sample here is shown using control file to disk.

$ORACLE_HOME/bin/rman

oracle@sausalito:/u01/app/oracle/product/11.2.0/db_1/dbs > rman target / catalog rman/rman@rmancpr

Recovery Manager: Release 11.2.0.2.0 – Production on Fri Jan 20 09:08:25 2012

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ROMEO(DBID=498205723)

connected to recovery catalog database

run {
backup format ‘/u1/df_%t_%s_%p’ database
include current controlfile
plus archivelog;
release channel d1;
}

2.  Copy all the backupset files to one of the target hosts to the same location of backup

$scp /rmanbackup/oradata/romeo/df*   hostA2: /rmanbackup/oradata/romeo/

 Create a pfile from the source database and move it to the target system

SQL> create pfile=’/tmp/initjuliet.ora’ from spfile ;

Move initjuliet.ora to the target system $ORACLE_HOME/dbs

3. Edit initjuliet.ora  and change

*.cluster_database=false

And any references of romeo to Juliet

Add convert name if names or paths are different

*.db_file_name_convert = (‘/u02/oradata/romeo’,'/u02/oradata/juliet’)

*.log_file_name_convert= (‘/u02/oradata/romeo’,'/u02/oradata/juliet’)

Create a password file using orapwd on the target host

orapwd file=orapwjuliet password=xxxxxxxx

4. On the target host create a TNSNAME alias pointing to the source database

ROMEO =

  (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = maverick)(PORT = 1521))

    (CONNECT_DATA =

      (SERVER = DEDICATED)

      (SERVICE_NAME = romeo)

    )

  )

5. Startup the Juliet instance in nomount  mode

6. Start the rman duplicate

 rman target sys/xxxxxxx@romeo auxiliary /  catalog rman/rman@rmancpr

 RMAN> run {

allocate channel D1 device type disk;

allocate auxiliary channel D2 device type disk;

DUPLICATE TARGET DATABASE TO juliet;

}

If successfully you should see the following extract at the end

 

datafile 2 switched to datafile copy

input datafile copy RECID=1 STAMP=772991657 file name=/u03/oradata/dupdb/sysaux01.dbf

datafile 3 switched to datafile copy

input datafile copy RECID=2 STAMP=772991657 file name=/u03/oradata/dupdb/undotbs01.dbf

datafile 4 switched to datafile copy

input datafile copy RECID=3 STAMP=772991657 file name=/u03/oradata/dupdb/users01.dbf

datafile 5 switched to datafile copy

input datafile copy RECID=4 STAMP=772991657 file name=/u03/oradata/dupdb/undotbs02.dbf

datafile 6 switched to datafile copy

input datafile copy RECID=5 STAMP=772991657 file name=/u03/oradata/dupdb/undotbs03.dbf

datafile 7 switched to datafile copy

input datafile copy RECID=6 STAMP=772991657 file name=/u03/oradata/dupdb/undotbs04.dbf

 contents of Memory Script:

{

   Alter clone database open resetlogs;

}

executing Memory Script

 database opened

Finished Duplicate Db at 19-JAN-2012 15:54:40

released channel: D1

released channel: D2

7. Add a new temporary file into the temporary table space

 alter tablespace temp add tempfile ‘/u02/oradata/dupdb/temp01.dbf’ size 2048m reuse ;

8. Make the database RAC aware

Each node of the cluster has its own oracle home n u01 and shared filesystem u02 and u03  and therefore needed to make the following changes enable RAC

Shutdown database on hostA2 and move the spfile from $ORACLE_HOME/dbs to /u03/oradata/Juliet

This is shared storage which all the 4 hosts can access

srvctl modify database -d juliet -o /u01/app/oracle/product/11.2.0/db_1 -p /u03/oradata/dupdb/spfilejuliet.ora

srvctl add instance -d julite -i juliet1 –n hostA2

srvctl add instance -d julite -i juliet2 –n hostB2

srvctl add instance -d julite -i juliet3 –n hostC2

srvctl add instance -d julite -i juliet4 –n hostD2

move the password file from hostA2  $ORACLE_HOME/dbs  to /u03/oradata/Juliet

create links  from each host  to the password file in  /u03/oradata/Juliet

hostA2 : $ORACLE_HOME/dbs ln –s /u03/oradata/Juliet/orapwjuliet  orapwdupdb1

hostB2 : $ORACLE_HOME/dbs ln –s /u03/oradata/Juliet/orapwjuliet  orapwdupdb2

hostC2 : $ORACLE_HOME/dbs ln –s /u03/oradata/Juliet/orapwjuliet  orapwdupdb3

hostD2 : $ORACLE_HOME/dbs ln –s /u03/oradata/Juliet/orapwjuliet  orapwdupdb4

on each host create the following init.ora 

hostA2  initjuliet1.ora   containing SPFILE=’/u03/oradata/dupdb/spfiledupdb.ora’

hostB2  initjuliet1.ora   containing SPFILE=’/u03/oradata/dupdb/spfiledupdb.ora’

hostC2  initjuliet1.ora   containing SPFILE=’/u03/oradata/dupdb/spfiledupdb.ora’

hostD2  initjuliet1.ora   containing SPFILE=’/u03/oradata/dupdb/spfiledupdb.ora’

add the following entries on each host /etc/oratab

hostA2        

juliet1:/u01/app/oracle/product/11.2.0/db_1:N     

hostB2

juliet2:/u01/app/oracle/product/11.2.0/db_1:N     

hostC2

juliet3:/u01/app/oracle/product/11.2.0/db_1:N     

hostD2

juliet4:/u01/app/oracle/product/11.2.0/db_1:N     

On hostA2 where the database was created

Mount the database

Export ORACLE_SID=juliet1

SQL> connect / as sysdba

SQL> startup mount

SQL> alter system set cluster_database=TRUE scope=spfile sid=’*’

SQL> shutdown immediate

Start up instances or databases using srvctl

srvctl start instance –d Juliet –I juliet1

Srvctl start instance –d Juliet –I juliet2

Srvctl start instance –d Juliet –I juliet3

Srvctl start instance –d Juliet –I juliet4

9   Add the database to OEM

 You might get the following issues in OEM

 Service romeoXDB and romeo  and any services that existed in romeo is down alert in GRID Control

 Refer to the following article for a resolution and how to avoid the problem

 http://managingoracle.blogspot.com/2011/01/service-xdb-is-down-alert-in-grid.html

 To resolve the problem  if  the services are not relevant  delete them

 SQL> exec dbms_service.delete_service(‘romeoXDB’) ;

SQL> exec dbms_service.delete_service(‘romeo’) ;

useful web links

http://oraexplorer.com/2007/11/rman-duplicate/

http://www.colestock.com/blogs/2007/09/how-to-duplicate-rac-database-using.html

http://www.oracleracexpert.com/2009/12/duplicate-rac-database-using-rman.html

http://surachartopun.com/2009/11/duplicate-without-connection-to-target.html

C:\Documents and Settings\SYSMAN>emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation. All rights reserved.


Agent Version : 10.2.0.2.0
OMS Version : 10.2.0.2.0
Protocol Version : 10.2.0.2.0
Agent Home : D:\oracle\agent10g
Agent binaries : D:\oracle\agent10g
Agent Process ID : 4008
Agent URL : <a href="http://usnapipca20.nalco.one.net:3872/emd/main/">http://USNAPIPCA20.Nalco.One.Net:3872/emd/main/</a>
Repository URL : <a href="http://usnapoems11.nalco.one.net:1159/em/upload/">http://usnapoems11.nalco.one.net:1159/em/upload/</a>
Started at : 2007-06-07 13:55:38
Started by user : SYSTEM
Last Reload : 2007-06-07 13:57:53
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 173
Size of XML files pending upload(MB) : 30.52
Available disk space on upload filesystem : 11.07%
Last attempted heartbeat to OMS : 2007-06-07 17:12:30
Last successful heartbeat to OMS : unknown

Agent is Running and Ready

to fix it do the following

- emctl stop agent
- delete all files in $AGENT_HOME/sysman/emd/upload and $AGENT_HOME/sysman/emd/state
- emctl clearstate agent (this should delete all state files but sometimes it is not enough, but I experienced that it is better to use this command after deletion of the files)
- emctl secure agent (and specify the password on demand)
- emctl start agent

oracle@sausalito:/u01/app/oracle/product/11.2.0/agent/agent11g > emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/11.2.0/agent/agent11g/sausalito.cpp-group.com
Agent binaries    : /u01/app/oracle/product/11.2.0/agent/agent11g
Agent Process ID  : 24039
Parent Process ID : 24033
Agent URL         : https://sausalito.cpp-group.com:3872/emd/main
Repository URL    : https://uk-yo-dbmon01.cpp-group.com:4900/em/upload
Started at        : 2011-11-23 11:16:23
Started by user   : oracle
Last Reload       : 2011-11-23 11:16:23
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :      210
Size of XML files pending upload(MB)         :    50.52
Available disk space on upload filesystem    :    60.23%
Collection Status                            : Disabled by Upload Manager
Last successful heartbeat to OMS             : 2011-11-23 11:37:37
—————————————————————
Agent is Running and Ready

When I tried to upload manually i received the following message

oracle@sausalito:/u01/app/oracle/product/11.2.0/agent/agent11g/sysman/emd > emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
—————————————————————
EMD upload error: Failed to upload file A0000005.xml: HTTP error.
Response received: ERROR-400|Data will be rejected for upload from agent ‘https://sausalito.cpp-group.com:3872/emd/main’, max size limit for direct load exceeded [8324170/5242880]

To fix the problem i  removed the file from $AGENT_HOME//u01/app/oracle/product/11.2.0/agent/agent11g/sausalito.iii-group.com/sysman/emd/upload

I then restarted the agent

oracle@sausalito:/u01/app/oracle/product/11.2.0/agent/agent11g/sausalito.cpp-group.com/sysman/emd/upload > emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/11.2.0/agent/agent11g/sausalito.cpp-group.com
Agent binaries    : /u01/app/oracle/product/11.2.0/agent/agent11g
Agent Process ID  : 27951
Parent Process ID : 27945
Agent URL         : https://sausalito.cpp-group.com:3872/emd/main
Repository URL    : https://uk-yo-dbmon01.cpp-group.com:4900/em/upload
Started at        : 2011-11-23 11:58:22
Started by user   : oracle
Last Reload       : 2011-11-23 11:58:22
Last successful upload                       : 2011-11-23 11:59:02
Total Megabytes of XML files uploaded so far :     0.48
Number of XML files pending upload           :        3
Size of XML files pending upload(MB)         :     1.18
Available disk space on upload filesystem    :    60.27%
Last successful heartbeat to OMS             : 2011-11-23 11:58:36
—————————————————————
Agent is Running and Ready

Also refer to the following document from metalink , should should resolve the problem from happening again

Agent Upload to OMS Fails with “ERROR-400|Data will be rejected for upload from agent … max size limit for direct load exceeded” [ID 359097.1]

recover standby database

1-Locate the central inventory
cat /etc/oraInst.loc or /var/opt/oracle/oraInst.loc (Platform specific)

2-Take a back up of central inventory or rename in all the nodes.
mv oraInventory oraInventory.orig

3-cd $ORACLE_HOME/oui/bin

4-Execute runInstaller with attachHome option for OraCrsHome.

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="<10g Ora_Crs_Home Path>" ORACLE_HOME_NAME="<Name of oracleCRSHome>" LOCAL_NODE='node1' CLUSTER_NODES=node1,node2 CRS=true

for ex:
./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/app/oracle/product/10.2.0/crs_1" ORACLE_HOME_NAME="OraCrs10g_home" LOCAL_NODE='rac1' CLUSTER_NODES=rac1,rac2 CRS=true

5-Execute runInstaller with attachHome option for OraDbHome
./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="<10g Oracle_Home Path>" ORACLE_HOME_NAME="<Name of oracleHome>" LOCAL_NODE='node1' CLUSTER_NODES=node1,node2

for ex:
./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/app/oracle/product/10.2.0/db_1" ORACLE_HOME_NAME="OraDb10g_home1" LOCAL_NODE='rac1' CLUSTER_NODES=rac1,rac2 
6-Verify the inventory creation in all the nodes.

7-Verify and compare XML files size inside oraInventory/ContentsXML across all the nodes.

backup up the database

rman "target / nocatalog"
RUN
{
ALLOCATE CHANNEL chan_name TYPE DISK;
BACKUP DATABASE FORMAT 'some_backup_directory%U' TAG before_upgrade;
BACKUP CURRENT CONTROLFILE FORMAT 'controlfile location and name';
}

Prepare the new oracle home

Copy the spfile and password file from $OLD_ORACLE_HOME/dbs to $NEW_ORACLE_HOME

If you are upgrading a cluster database , then set the CLUSTER_DATABASE initialization parameter to false

Upgrade the database

1. Shutdown the instance

SQL> SHUTDOWN IMMEDIATE ;

2. Amend the entry in /etc/oratab for the database you are upgrading to point to the new home
RMANCATPR:/u01/app/oracle/product/11.2.0.3/db_1:N

3. The following environment variables point to the Oracle Database 11g Release 2 (11.2.0.3) directories:
ORACLE_HOME
PATH

4. Log in to the system as the owner of the Oracle Database 11g Release 2 (11.2.0.3) Oracle home directory

5. At a system prompt, change to the ORACLE_HOME/rdbms/admin directory

6. Start SQL*Plus.

7. Connect to the database instance as a user with SYSDBA privileges.

8. Start the instance by issuing the following command:

SQL> STARTUP UPGRADE

9. Set the system to spool results to a log file for later verification of success:
SQL> spool /tmp/upgrade.log

10. Run the Pre-Upgrade Information Tool by executing the utlu112i.sql script:
SQL> @utlu112i.sql

11. Run the catupgrd.sql script:
SQL> @catupgrd.sql

12. Restart the instance to reinitialize the system parameters for normal operation.
SQL> STARTUP

13. Run utlu112s.sql, the Post-Upgrade Status Tool, which provides a summary of the upgrade at the end of the spool log.
Run utlu112s.sql to display the results of the upgrade as follows:
SQL> @utlu112s.sql

14. Run catuppst.sql, located in the ORACLE_HOME/rdbms/admin directory, to perform upgrade actions that do not require
the database to be in UPGRADE mode:
SQL> @rdbms/admin/catuppst.sql

15. Run utlrp.sql to recompile any remaining stored PL/SQL and Java code.
SQL> @utlrp.sql

Verify that all expected packages and classes are valid:
SQL> SELECT count(*) FROM dba_invalid_objects;
SQL> SELECT distinct object_name FROM dba_invalid_objects;

16. Exit SQL*Plus.

17. If you are upgrading a cluster database from releases 10.2, 11.1, or 11.2.0.1, then upgrade the database configuration in Oracle Clusterware using the following command:

$ srvctl upgrade database -d db-unique-name -o oraclehome
where db-unique-name is the database name assigned to it (not the instance name), and oraclehome is the Oracle home location in which the database is being upgraded.

Rerunning the Upgrade

You can rerun the upgrade with the catupgrd.sql script as described in the following steps.

To rerun the upgrade

1.Shut down the database as follows:

SQL> SHUTDOWN IMMEDIATE
2.Restart the database in UPGRADE mode:

SQL> STARTUP UPGRADE
3.Set the system to spool results to a log file for later verification of success:

SQL> SPOOL upgrade.log
4.Rerun catupgrd.sql:

SQL> @catupgrd.sql
Note:

You can rerun the catupgrd.sql script as many times as necessary. The first time you run the script, there should be no error messages returned. If you rerun the script, then the ORA-00001 message is displayed. You can safely ignore this message.
5.Rerun utlu112s.sql:

SQL> @utlu112s.sql

 
Cancelling the Upgrade

If you completed the steps in “Backing Up the Database” to back up your database,
then the easiest way to cancel the upgrade is to restore that backup as described in the following procedure.

To cancel the upgrade by restoring the previous backup

1.Log in to the system as the owner of the Oracle home directory of the previous release.

2.Sign on to RMAN:

rman “target / nocatalog”
3.Issue the following RMAN commands:

STARTUP NOMOUNT
RUN
{
    RESTORE CONTROLFILE FROM ‘save_controlfile_location’;
    ALTER DATABASE MOUNT;
    RESTORE DATABASE FROM TAG before_upgrade
    ALTER DATABASE OPEN RESETLOGS;
}