VirtualObjectives

Oracle Rolling Patch Procedure - RAC

This procedure is for Oracle RAC only.
It describes how to perform a rolling point release upgrade of Oracle.
The rolling upgrade will allow one node to be upgraded whilst the other node is running.
There is no downtime at all since at least one node is running at any one time.

The following procedure is an example of how to upgrade a two node RAC environment running on Windows servers running 10G.

Thoroughly read the file README.html contained in the Oracle patch.
It will detail the prerequisites, upgrade steps, and post upgrade steps.

Perform the following on node2 first and then on node1.
Remember, make sure at least one node is running at all times.

1 Start

You need to RDP into the server as an administrative session.

Copy the upgrade binaries to the server into a standard location like an "Install" folder.

You need to check the version of OPatch and display patches already applied. You may need to upgrade OPatch if the upgrade instructions specify an minimum required version.

Oracle Home       : e:\some\oracle\path\database
Central Inventory : C:\Program Files\Oracle\Inventory
   from           : n/a
OPatch version    : 10.n.n.n.n
OUI version       : 10.n.n.n.n
OUI location      : e:\some\oracle\path\database\oui

2 Backup Files

Backup the Oracle %ORACLE_HOME% directory.

Copy the %ORACLE_HOME% directory to a safe temporary location.
Copy the directory C:\Program Files\oracle\ to a safe temporary location.

3 Stopping Oracle

Perform a clean shutdown of all Oracle services (databases, Oracle listeners, Oracle agents, and so on.) that run from the Oracle Home where you installed the patch. This involves stopping OCR related resources, Cluster Ready Services (CRS) Services and all Oracle Services.

Stop ALL Oracle services running on the current node in the following order:

Explicitly stop the Distributed Transaction Coordinator service (a Windows service) if it is running.

Check Services to see if there any other services starting with "Oracle" which is still running and stop them.

4 Drivers

The upgrade notes for RAC may include the need to update drivers like orafencedrv.sys and ocfs.sys
It will state a reboot is required, but this can be done at the end.

Rename file orafencedrv.sys located under %SystemRoot%\system32\drivers\ to orafencedrv.sys.OLD
Rename file ocfs.sys located under %SystemRoot%\system32\drivers\ to ocfs.sys.OLD

Copy into %SystemRoot%\system32\drivers\ the file orafencedrv.sys supplied in the patch.
Copy into %SystemRoot%\system32\drivers\ the file ocfs.sys.2k864 supplied in the patch and rename to ocfs.sys
(You will need the select the correct file depending on whether your server is 32/64-bit and version of Windows)

5 Patching

Set your Oracle Home

Add the location of OPatch to the PATH:

Go to the Oracle patch location:

Before patching each node, OPatch will prompt for user input. At this point please make sure that all the services running on that node are stopped.

Email address/User Name: [Enter]
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
Is the local system ready for patching? [y|n] Y

At this point, Oracle is being patched.
The following prompt shows the computer name of the next node to be patched.
You will need to select NO, you don't want to continue.

The local system has been patched.  You can restart Oracle instances on it.

Patching in rolling mode.

The node 'ComputerName2' will be patched next.
Is the node ready for patching? [y|n] N
User Responded with: N
ApplySession exits on request
You may exit the patching session and patch remaining nodes later from
 an un-patched node.  Do you want to continue?
Do you want to proceed? [y|n] N
User Responded with: N
Patching aborted on node "ComputerName2", remaining nodes are
OPatch failed to propagate the patch to the remote node(s).
Run the patch again on each of the remote node(s) using -local option.
Details: Patching aborted on node "ComputerName2", remaining nodes are
ApplySession failed: OUI-67081:OPatch failed to propagate the patch to
 the remote node(s).
Run the patch again on each of the remote node(s) using -local option.
Details: Patching aborted on node "ComputerName2", remaining nodes are
System intact, OPatch will not attempt to restore the system
----------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch nnnnnnnn is a superset of the patch(es) [ 
 nnnnnnnnnnnn ] in the Oracle Home
----------------------------------------------------------------------------

OPatch failed with error code = 73

Save the output to a file and review it for errors.
Notice the warning and the error code 73. This is normal, and you would see similar warnings which should match your testing.

6 Restart the Node

Restart the server hosting node 2 and start Oracle if it does not start automatically.
Check Oracle alert log for errors.

Check the status of CRS:

Ensure that all is working correctly on node 2 before proceeding.

Repeat for Node 1

Now, go back and repeat Steps 1 to 6 for the next RAC node (node 1 in this example)

Apply Fixes for Bugs

Oracle's patch file README.html may contain added fixes.
These can be performed after the patches (above) are applied to all the nodes.
The fixes are usually required to be done on each node, not just one, however, the documentation will specify in detail.

Database Patching - Perform on One Node Only

Now, once all nodes are online, you need to apply various database patches and registration of patches applied.
Oracle's documentation will be more specific about what steps are needed.

Connect to the database from one server only using SQL*Plus as SYSDBA and run catcpu.sql as follows:

Check the log file for any errors. The log file is found in %ORACLE_HOME%\cfgtoollogs\catbundle and is named catbundle_WINBUNDLE__APPLY_.log, where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS.

Check the Oracle alert logs on all nodes for issues.
That's it. This does not need to be repeated on any other RAC node.

Issues

Review the Oracle documentation (README.html) for back out procedures.
If there are errors, refer to the section called "Known Issues".

If catcpu.sql reports any Invalid Objects, compile the invalid objects as follows.

(For large numbers of objects, this compilation step can take some time. If you are applying this patch through the Oracle Enterprise Manager console, you will be prompted to run this compilation script.)

To check for invalid objects, execute the following statement: