Starting with Oracle Database 19c Release Update (RU) 19.13, Oracle announced that Oracle RAC database instances can keep running and accessible to the database users during Oracle Grid Infrastructure patching. My patching practice is done in following environment
* Two nodes Grid Infrastructure 19.13 running on Oracle Linux 7
* Host name of nodes are node1 and node2
* GI RU Database Release Update : 19.15.0.0.220419 is applied in out-of-place(OOP) mode
Grid Infrastructure 19.8 running on Linux x86-64 with no ACFS/AFD configured.
[grid@node1 (LMT:+ASM) ~ ] $ $ORACLE_HOME/OPatch/opatch lsinventory -inactive | grep Patch
Oracle Interim Patch Installer version 12.2.0.1.27
OPatch version : 12.2.0.1.27
Patch 31780966 : applied on Tue Nov 03 17:19:41 GMT 2020
Unique Patch ID: 23770942
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)"
Patch 31773437 : applied on Tue Nov 03 17:18:31 GMT 2020
Unique Patch ID: 23768260
Patch description: "ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)"
Patch 31772784 : applied on Tue Nov 03 17:17:13 GMT 2020
Unique Patch ID: 23854876
Patch description: "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch 31771877 : applied on Tue Nov 03 17:13:36 GMT 2020
Unique Patch ID: 23869227
Patch description: "Database Release Update : 19.9.0.0.201020 (31771877)"
Patch 31335188 : applied on Wed Aug 26 16:23:04 BST 2020
Unique Patch ID: 23561916
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)"
Patch 31305087 : applied on Wed Aug 26 16:21:27 BST 2020
Unique Patch ID: 23640701
Patch description: "OCW RELEASE UPDATE 19.8.0.0.0 (31305087)"
Patch 31304218 : applied on Wed Aug 26 16:19:49 BST 2020
Unique Patch ID: 23548486
Patch description: "ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)"
Patch 31281355 : applied on Wed Aug 26 16:09:18 BST 2020
Unique Patch ID: 23688465
Patch description: "Database Release Update : 19.8.0.0.200714 (31281355)"
Patch 29585399 : applied on Thu Apr 18 08:36:24 BST 2019
Unique Patch ID: 22840393
Patch description: "OCW RELEASE UPDATE 19.3.0.0.0 (29585399)"
Patch 29517247 : applied on Thu Apr 18 08:36:02 BST 2019
Unique Patch ID: 22840392
Patch description: "ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)"
Patch 29517242 : applied on Thu Apr 18 08:35:39 BST 2019
Unique Patch ID: 22862832
Patch description: "Database Release Update : 19.3.0.0.190416 (29517242)"
Patch 29401763 : applied on Thu Apr 18 08:35:16 BST 2019
Unique Patch ID: 22759421
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)"
OPatch succeeded.
[grid@node1 (LMT:+ASM) ~ ] $ $ORACLE_HOME/OPatch/opatch lsinventory | grep Patch
Oracle Interim Patch Installer version 12.2.0.1.27
OPatch version : 12.2.0.1.27
Patch 33239955 : applied on Tue Mar 08 13:54:03 GMT 2022
Unique Patch ID: 24385395
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (33239955)"
Patch 33208123 : applied on Tue Mar 08 13:51:30 GMT 2022
Unique Patch ID: 24445425
Patch description: "OCW RELEASE UPDATE 19.13.0.0.0 (33208123)"
Patch 33208107 : applied on Tue Mar 08 13:48:51 GMT 2022
Unique Patch ID: 24433220
Patch description: "ACFS RELEASE UPDATE 19.13.0.0.0 (33208107)"
Patch 33192793 : applied on Tue Mar 08 13:44:49 GMT 2022
Unique Patch ID: 24462514
Patch description: "Database Release Update : 19.13.0.0.211019 (33192793)"
Patch 32585572 : applied on Tue Mar 08 13:42:14 GMT 2022
Unique Patch ID: 24103449
Patch description: "DBWLM RELEASE UPDATE 19.0.0.0.0 (32585572)"
OPatch succeeded.
[grid@node1 (LMT:+ASM1) ~ ] $ crsctl query crs releasepatch
Oracle Clusterware release patch level is [2966572961] and the complete list of patches [32585572 33192793 33208107 33208123 33239955 ] have been applied on the local node. The release patch string is [19.13.0.0.0].
[grid@node1 (LMT:+ASM1) ~ ] $ crsctl query crs softwarepatch
Oracle Clusterware patch level on node node1 is [2966572961].
Create new Oracle Grid Infrastructure (GI) 19c home and prepare Oracle GI Release Update (RU) 19.15 for patching
[root@node1]# mkdir -p /u01/app/19.15.0/grid
[root@node1]# chown -R grid:oinstall /u01/app/19.15.0
As the grid user on first node, download the Oracle Grid Infrastructure 19c image file and extract the files into new created GI home
[grid@node1]$ cd /u01/app/19.15.0/grid
[grid@node2]$ unzip -q /oraclesw/oracle_software/grid/LINUX.X64_193000_grid_home.zip
As grid user on first node, download Oracle GI RU 19.15.0.0.220419 and extract the files into stage/temporary directory
[grid@node1]$ mkdir /oraclesw/patches
[grid@node1]$ cd /oraclesw/patches
[grid@node1]$ unzip -q /oraclesw/patches/p33803476_190000_Linux-x86-64.zip
As grid user on first node, download and install latest version of OPatch (12.2.0.1.30) into new GI home
[grid@node1]$ cd /u01/app/19.15.0/grid
[grid@node1]$ mv OPatch OPatch.old
[grid@node1]$ unzip -q /oraclesw/patches/p6880880_190000_Linux-x86-64_12.2.0.1.30.zip
Apply the patch and configure the software .
cd /u01/app/19.15.0/grid
./gridSetup.sh -switchGridHome -applyRU /oraclesw/patches/33803476
Note: The installation option -applyOneoff can also be used to apply any oneoff patches as part of the installation process.
Eg: ./gridSetup.sh -switchGridHome -applyRU /oraclesw/patches/33803476 -applyOneOffs /oraclesw/patches/31668882
- In the ‘Select Configuration Option’ screen, select the ‘Set Up Software Only’ option to perform a software-only installation of Oracle Grid Infrastructure for a standalone server. Click Next.
- In the ‘Cluster Node Information’ screen, click Add button to add public host names of all cluster nodes(Node1 & Node2). Click Next.
- Respond to the prompts as needed to set up Oracle Grid Infrastructure
if you get an error
ERROR: The home is not clean. This home cannot be used since there was a failed OPatch execution in this home. Use a different home to proceed.
rm -rf /u01/app/19.15.0/grid/*
Start again , Unzip the software and update OPatch as described above
If you also get the following error
[INS-32826] The software home (/u01/app/19.15.0/grid) is already registered in the central inventory. Refer to patch readme instructions on how to apply. This was the issue <HOME NAME="tempGIName" LOC="/u01/app/19_4_0_0/grid" TYPE="O" IDX="3"/>
To solve this as I could not find a solution was to remove the entry from inventory.xml ***NOTE THIS IS NOT RECOMMENDED AS IT WILL LEAD TO CORRUPTION ***
But this worked for me
The order option is you could try to Detach the home But this did not work
$ <New_GI_HOME>/oui/bin/runInstaller -silent -waitforcompletion\
-detachHome ORACLE_HOME='tempGIName' -local
The other option is to recreate the inventory
backed up the content of /u01/app/oraInventory/ before emptying it on both nodes
<OLD_GI_HOME>/oui/bin/runInstaller/runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/app/19.0.0/grid" ORACLE_HOME_NAME="OraGI19Home1" LOCAL_NODE="node1" CLUSTER_NODES="{node1,node2}" CRS=true
<OLD_GI_HOME>/oui/bin/runInstaller/runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/u01/app/oracle/product/19.0.0/dbhome_1" ORACLE_HOME_NAME="OraDB19Home1"" LOCAL_NODE="node1" CLUSTER_NODES="{node1,node2}"
if all is successfully you will see the following screen
Run the root.sh for the install software only.
Switching the Grid Infrastructure Home
Run the gridSetup.sh from the target home
/u01/app/19.15.0/grid/gridSetup.sh -switchGridHome
It will launch the GUI
During this phase,you should not select the automated root.sh option.We have to run manually with additional option listed below.
When prompted, run “root.sh -transparent -nodriverupdate” instead of root.sh.
[root@node1 ~]# /u01/app/19.15.0.0/grid/root.sh -transparent -nodriverupdate
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/19.15.0.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19c/grid/lib:/u01/app/19.15.0.0/grid/lib:'
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node1/crsconfig/rootcrs_node1_2020-02-02_07-30-48PM.log
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node1/crsconfig/rootcrs_node1_2020-02-02_07-30-48PM.log
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node1/crsconfig/crs_prepatch_node1_2020-02-02_07-30-49PM.log
2020/04/20 19:30:50 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node1/crsconfig/crs_postpatch_node1_2020-02-02_07-30-51PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3253194715].
2020/04/20 19:31:10 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2020/04/20 19:31:10 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [3253194715].
2020/04/20 19:39:11 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/04/20 19:39:12 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
Run the root.sh -transparent -nodriverupdate on other nodes
On node2.
[root@node2 ~]# /u01/app/19.15.0.0/grid/root.sh -transparent -nodriverupdate
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/19.9.0.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19c/grid/lib:/u01/app/19.15.0.0/grid/lib:'
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node2/crsconfig/rootcrs_node2_2020-02-02_07-42-45PM.log
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node2/crsconfig/rootcrs_node2_2020-02-02_07-42-45PM.log
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node2/crsconfig/crs_prepatch_node2_2020-02-02_07-42-46PM.log
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node2/crsconfig/crs_prepatch_node2_2020-02-02_07-42-48PM.log
2020/04/20 19:43:08 CLSRSC-347: Successfully unlock /u01/app/19.9.0.0/grid
2020/04/20 19:43:10 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/19.15.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node2/crsconfig/crs_postpatch_node2_2020-02-02_07-43-11PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [3253194715].
2020/04/20 19:43:40 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd_dummy.service'
2020/04/20 19:47:08 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3633918477]
Validating logfiles...done
Patch 30557433 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply__MGMTDB_CDBROOT_2020Feb02_19_53_44.log (no errors)
Patch 30557433 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply__MGMTDB_PDBSEED_2020Feb02_19_56_03.log (no errors)
Patch 30557433 apply (pdb GIMR_DSCREP_10): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30557433/23305305/30557433_apply__MGMTDB_GIMR_DSCREP_10_2020Feb02_19_56_02.log (no errors)
SQL Patching tool complete on Sun Feb 2 19:58:43 2020
2020/04/20 20:00:57 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/04/20 20:01:02 CLSRSC-672: Post-patch steps for patching GI home successfully completed.
To clean up the old home post switch as clusterware user execute the following on any node:
## please replace $OLD_HOME with the path of pre-upgrade clusterware home
export ORACLE_HOME=$OLD_HOME
## detach OLD_HOME
$OLD_HOME/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=$OLD_HOME
## confirm $OLD_HOME is removed from central inventory:
$NEW_HOME/OPatch/opatch lsinventory -all
## remove files in OLD_HOME manually on all nodes:
/bin/rm -rf $OLD_HOME
unset ORACLE_HOME
If it fails for any reason, as clusterware user execute the following on all nodes:
export ORACLE_HOME=$OLD_HOME
## detach OLD_HOME
$OLD_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_HOME
## confirm $OLD_HOME is removed from central inventory:
$NEW_HOME/OPatch/opatch lsinventory -all
## remove files in OLD_HOME manually on all nodes:
/bin/rm -rf $OLD_HOME
unset ORACLE_HOME
Rollback
There are a few options to rollback from NEW_HOME to OLD_HOME
1. change ownership to the old home, run switch again and root.sh
chown -R grid:oinstall /u01/app/19c/grid
./gridSetup.sh -switchGridHome
./root.sh -transparent -nodriverupdate
2. return to the old version you may need to recreate another new home and switch to that.
3. restore from a full OS backup before the switch occurred.
Mike Desouza's Blog 