Exadata compute nodes patching

Exadata compute nodes patching

Imp points:

Patching a database node takes 45 minutes to one hour

You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

Starting with release 12.2.1.1.0, Exadata software updates for the Exadata database server can be applied only through the patchmgr utility. The dbserver.patch.zip file contains a special version of patchmgr used to orchestrate dbnodeupdate in updating Exadata Databases Nodes.

The patchmgr utility for updating Exadata database servers is not the same as the patchmgr script shipped with the Exadata software update.

Pre-requisites:

1. Note the output of below commands from each db node:
imageinfo
crsctl stat res –t

2. Check and note the ilom ip of each db node. connect to ilom of each db node and verify that each ilom server is running fine.

3. If you are planning to update all Exadata database servers at once, it is a requirement to run the update utility from a Linux node outside the group of Exadata database servers being updated. This is because the update utility cannot update the Exadata database server it is currently running on. If you have no other systems running Oracle Linux or Oracle Solaris you can run the update utility from one of the Exadata database servers. In such cases be sure that the Exadata database server where the update utility is running is not listed in the dbs_group file you specify.

4. You need to set up ssh equivalence for the root user from the driving node to the root user of all Exadata database servers that will be updated.

5. Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1. verify that the patchmgr utility is of latest version.

6. Download the latest Exachk from My Oracle Support note 1070954.1.

7. Run ExaChk tool

[root@dm01 ]# ./exachk

8. Run prerequisite check with the –nomodify_at_prereq flag weeks before doing the patching.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/

p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3

-nomodify_at_prereq &


If it find any unknown rpm package which is conflicting with the patch, so it will show error like below:

exadb01: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb01:/var/log/cellos/unkown_packages-rpt.070519194937.txt for more details.


Imp Contents of this file:

ALERT : Custom packages found (see above)

# These custom packages MUST be removed before proceeding a major Oracle Linux upgrade.

# Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed - OR -

# Run /var/log/cellos/remove_unkown_packages.070519194937.sh manually as root before re-running prereq check.

#

# RECOMMENDED : Let dbnodeupdate.sh remove the packages right after the mandatory backup by using the -R flag

# WARNING : Removing custom packages may impact functionality

# NOTE : When removed functionality is still required after the update, reinstall the rpm after the update

#################################################################################


Note:

By default, prerequisite check warnings such as active NFS mounts result in a failed check. If you want to allow active NFS mounts, then starting in release 12.1.2.1.1 you can use the -allow_active_network_mounts flag.

9. Fix the issues found in these two reports and run them again for no issues.

Perform a “backup only” run using the - backup flag.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -backup -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3 -allow_active_network_mounts &

Note:
The -backup action backs up the active root and /boot file system only and is sufficient for rolling back (failed) updates.

Re-running the update utility with the -backup flag (or in default updating mode) will overwrite existing backups.

Intervention steps:
1. Perform a “backup only” run using the - backup flag.
2. Run ExaChk tool
3. Run prerequisite check with the –nomodify_at_prereq flag weeks before doing the patching.
4. Check and fix the critical issues found in them and run them again for no issues.
5. Remove any blocking rpms and re-run the prerequisite check to validate if all pre-requisite are passed.
6. Run the pre-requisite checks with out the –nomodify_at_prereq flag to allow the removal of conflicting rpms using ISO.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/ p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3 &

7. • Perform the update. Use the –nobackup flag to skip the backup because you already made a “backup only” run.

[root@dm01 ]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip -target_version 12.1.2.2.3.160720 -allow_active_network_mounts –nobackup &

Note:
You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

For permforming the update in rolling fashion, use –rolling option. Below is the example:

[root@myclustercel01 ~]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/SAVE/p28666206_*_Linux-x86-64.zip -target_version 18.1.9.0.0.181006 -allow_active_network_mounts -rolling &


8. After the update, Reinstall any non-Exadata rpms that you removed before the Exadata update if they are needed.

9. Run Exachk. Check the issues and fix them.

10. Run imageinfo on each node. Verify that each node is running with the expected version.

11. Run crsctl stat res –t on each node. Compare its output with the prechecks output.


Rollback Steps:

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -rollback -target_version 12.1.2.3.0.160207


Note:
Having only one inactive system partition limits the rollback options to only the previous active image.

Firmware updates are not rolled back when rolling back to a previous image.

The Oracle Exadata System Software releases support later firmware releases. After rolling back, run the following commands to apply older firmware versions if needed:

/etc/init.d/lsidiag stop
/etc/init.d/lsi_mrdsnmpd stop
/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

The last command only applies to releases 11.2.3.3.0 or later.