Upgrade Grid Infrastructure from 12c To 19C

The upgrade will be done in rolling mode.

PRECHECKS:

1. Note the output of below commands on each node.

crsctl stat res -t
crsctl check crs


2. Check activeversion, softwareversion and releaseversion:

crsctl query crs activeversion
crsctl query crs softwareversion
crsctl query crs releaseversion

activeversion:

This is the version of grid across all nodes of the cluster. If the rolling upgrade is not completed across all nodes of the cluster, it will show the previous version of grid.

releaseversion:

It shows the grid version installed in the binaries of the local node.

softwareversion [node_name]:

It gives the leverage to check the grid version of the remote node. It tell the grid version that has been successfully started on the specified node which is mentioned in the command. If not specified it shows the same for the local node.

crsctl query crs softwareversion [node_name]

If you do not provide a node name, then Oracle Clusterware displays the version of Oracle Clusterware running on the local server.

Example

The crsctl query crs softwareversion command returns output similar to the following:

Oracle Clusterware version on node [node1] is [11.2.0.2.0]


3. Apply required patches to your existing 12c grid.

4. Download the software and unzip on your first node.

5. Run the orachk tool as grid owner ( oracle)

orachk tool will generate a report for recommendation, that need to be taken care before upgrading.

export GRID_HOME=/u01/oracle/app/grid19c
cd /u01/oracle/app/grid19c/suptools/orachk/
./orachk –u -o pre



Analyze the html report for any recommendations.

6. Run cluvfy as grid owner ( oracle )



cd /u01/oracle/app/grid19c

syntax – >
./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome -dest_crshome -dest_version 19.0.0.0.0 -fixup -verbose

i.e
./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /drcrs/app/oracle/product/grid12c -dest_crshome /u01/oracle/app/grid19c -dest_version 19.0.0.0.0 -fixup -verbose



In case any error reported, fix them before proceeding further.



DRY RUN PHASE

It may restart the cluster nodes so take downtime if possible to run it.

Dry run phase will not do any changes to the existing grid setup. It will just check the system readiness.

As per oracle note: Dry run does below activities:

· Validates storage and network configuration for the new release

· Checks if the system meets the software and hardware requirements for the new release.

· Checks for the patch requirements and apply necessary patches before starting the upgrade.

· Writes system configuration issues or errors in the gridSetupActions<timestamp>.log log file



— Run as grid owner ( oracle)

unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID

cd /u01/oracle/app/grid19c
./gridSetup.sh –dryRunForUpgrade



Follow the Gui instructions.

The warnings for public network interface can be ignored.

It will ask to run rootupgrade.sh on local node only. Not to be run on the remote node.

The dry run upgrade is successful. Lets proceed with actual upgrade.

ACTUAL UPGRADE:

Now we will proceed with the actual upgrade in a rolling mode .

— Run as grid owner ( oracle )
unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID

cd /u01/oracle/app/grid19c
./gridSetup.sh

Follow the Gui instructions

Or You can execute it using responsefile:

run gridsetup.sh script with the resonse file :

./gridSetup.sh -executeConfigTools -responseFile /u01/crsapp/grid19c/install/response/gridinstall_5_07_2019.rsp


This will skip the already executed tasks and complete the pending configuration.

It will ask to run rootupgrade.sh first on the local node. Then it can be executed parallely in subsequent nodes except the last node. Run this script in the end on the last node.



The rootupgrade.sh does the following:

1. Sets oracle_owner, oracle_home.

2. Put oraenv and coraenv in /usr/local/bin/ directory.

3. Add entires in /etc/oratab file.

4. Relinking will be done.

5. Check and validate the the crs confiruration which is present in /u01/oracle/app/grid19c/crs/install/crsconfig_params file.

6. Stop the TFA process.



Once rootupgade.sh script execution completed on both nodes. Proceed to resume.

We have successful upgraded the grid to 19c version.



POST CHECK :

7. Check activeversion, softwareversion and releaseversion:

crsctl query crs activeversion

crsctl query crs softwareversion
crsctl query crs releaseversion






oracle@node1:~$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [19.0.0.0.0]



oracle@node1:~$ crsctl query crs softwareversion

Oracle Clusterware version on node [node1] is [19.0.0.0.0]

oracle@node2:~$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [19.0.0.0.0]

oracle@node2:~$ crsctl query crs softwareversion

Oracle Clusterware version on node [node2] is [19.0.0.0.0]



8. Run the below commands on each node and compare the output of below commands with the prechecks.



crsctl stat res -t

crsctl check crs



TROUBLESHOOTING:

1. If rootupgrade.sh script failed on local node:



In case rootupgrade.sh script failed on local node either due to any error or system got rebooted during that time, Then analyze the error and fix it . Once fixed, resume the ugprade with below step.

–run rootupgrade.sh script again on node 1:

cd /u01/crsapp/grid19c
/u01/crsapp/grid19c/rootupgrade.sh

–run rootupgrade.sh script again on node 2

cd /u01/crsapp/grid19c
/u01/crsapp/grid19c/rootupgrade.sh

Direct 12c database Upgrade high level steps

Direct 12c database Upgrade high level steps

1. Backup the source database. Enable flashback if possible and create a guaranteed restore point.

2. Direct upgrade to 12c is possible from 10.2.0.5, 11.1.0.7 and 11.2.0.2 so please check the source db version and if:

a. It is < 10.2.0.5, then upgrade it to 10.2.0.5 first.

b. If it is 11.1.0.7, then upgrade it to 11.1.0.7 first.

c. If it is 11.2.0.2 then upgrade it to 11.2.0.2 first.

Also, 12c database cannot be downgraded to 10g. it can only be downgraded to 11g.

3. Install the 12c binaries in a separate ORACLE_HOME directory.

/home/oracle/stage/database/runInstaller -silent -responseFile /home/oracle/stage/database/response/db_install.rsp

4. Run the pre upgrade tool. It is present in new ORACLE_HOME. You can run it from new ORACLE_HOME. 

$ORACLE_HOME/rdbms/admin/preupgrd.sql

5. It will create the preupgrade fixup and postupgrade fix up scripts. Run them as suggested by this tool.

6. Purge recycle bin. DB control is no longer used in 12c. it is removed during the upgrade but to save the downtime, it recommends to remove it. it is also recommended by preupgrade tool.

7. Disable cron jobs and jobs scheduled outside the database until the upgrade is completed.

8. Shutdown the source db and listener.

9. Copy the parameter file, password file, network files in new ORACLE_HOME.

10. From Oracle Database 12c home, start up the database using STARTUP UPGRADE and execute catctl.pl

SQL> startup UPGRADE

$ORACLE_HOME/perl/bin/perl catctl.pl catupgrd.sql

11. Run post upgrade fixup scripts.

12. Run the Post-Upgrade Status tool to show a summary and status of the upgrade. If there are any errors or any component is INVALID, then troubleshoot to fix the error.

SQL> @@?/rdbms/admin/utlu121s.sql

13. 12c using the timezone with version 18. So it is not that it will recommended to update it post upgrade.

14. Recompile any invalid objects. You may specify parallelism for the script as a parameter.

SQL> @@?/rdbms/admin/utlrp 6

15. Run the utluiobj.sql (Upgrade Invalid Objects tool) script to identify new invalid objects after the upgrade. This script will work only if you have run the preupgrd.sql script before the upgrade. It outputs the difference between the invalid objects that exist after the upgrade and invalid objects that existed prior to the upgrade. Fix up the new invalid objects.

SQL > @?/rdbms/admin/utluiobj.sql

16. Start the listener from the new oracle home.

17. Enable back Database Vault if required.

18. Enable back the cron jobs.

19. Drop restore point if it is taking up huge space.

SELECT NAME,SCN,TIME,GUARANTEE_FLASHBACK_DATABASE,STORAGE_SIZE/1024/1024/1024 STORAGE_SIZE_GB,DATABASE_INCARNATION# FROM V$RESTORE_POINT;

drop restore point <restore Point name>;

20. Ask app team and all other stakeholder teams for confirmation if all is ok to set the compatible parameter to 12.1.0 to enable its features.

SQL> ALTER SYSTEM SET compatible = '12.1.0' SCOPE=spfile;








Preupgrade tool:

If $ORACLE_BASE is defined, the generated scripts and log files are saved in $ORACLE_BASE/cfgtoollogs/db_unique_name/preupgrade directory. If $ORACLE_BASE is not defined, then the generated scripts and log files are created in $ORACLE_HOME/cfgtoollogs/db_unique_name/preupgrade.

It will check the following and reports issues if found:

a) If the COMPATIBLE parameter is at 11.0.0 or higher. So for 10g databases, it cannot be downgraded. It will need to be restore from backup if needed for downgrade.

b) db parameters. If there is a need to remove hidden parameters, depreciated parameters, and underscore events.

c) components of the database

d) size and free space in system , sysaux, undo, temp tablespaces. If SYS and SYSTEM users have SYSTEM as their default tablespace.

e) resources count like processes,

f) users and roles with the same name e.g. AUDSYS, AUDIT_VIEWER etc.

g) DB control is no longer used in 12c. it is removed during the upgrade but to save the downtime, it recommends to remove it before using below:

- Stop EM Database Control:

$> emctl stop dbconsole

- Connect to the Database using the SYS account AS SYSDBA:

SET ECHO ON;

SET SERVEROUTPUT ON;

@emremove.sql

h) invalid objects,

i) gather stats

EXECUTE dbms_stats.gather_dictionary_stats;

Please create stats on fixed objects after the upgrade using the command:

EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;


j) If the JOB_QUEUE_PROCESSES value is set too low.

k) Purge recyclebin

l) Timezone. 12c using the timezone with version 18. So it is not that it will recommended to update it post upgrade using below:

-- Fix Summary:

-- Update the timezone using the DBMS_DST package after upgrade is complete.

dbms_preup.run_fixup_and_report('OLD_TIME_ZONES_EXIST');

END;

and creates two pre and post upgrade fixup scripts:

Pre-Upgrade Fixup Script (run in source database environment):

/u01/app/oracle/cfgtoollogs/ocad11/preupgrade/preupgrade_fixups.sql

Post-Upgrade Fixup Script (run shortly after upgrade):

/u01/app/oracle/cfgtoollogs/ocad11/preupgrade/postupgrade_fixups.sql

Fixup scripts must be reviewed prior to being executed.

These scripts fixup only the trivial issues which are non-impacting and which doesn’t require DBA attention. it does not fix the issues that could damage the database. For example, dropping Enterprise Manager Database Control, resizing tablespaces, dropping users and roles, gathering statistics, and so on must be fixed manually by the DBA. It will let you know to manually fix those issues before the upgrade.

m) If Database Vault is enabled. It may recommend disabling, because it is a requirement to disable before the upgrade and enable after the upgrade if needed.

n) If any files are in backup mode or in media-recovery-needed state.

o) If the standby database is in sync with the primary.

Catctl.pl step:

Set the below parameters to the new ORACLE_HOME

ORACLE_HOME

ORACLE_SID

LD_LIBRARY_PATH

PATH

ORACLE_BASE

$ cd $ORACLE_HOME/rdbms/admin

$ sqlplus "/ as sysdba"



SQL> startup UPGRADE

$ORACLE_HOME/perl/bin/perl catctl.pl -n 6 -l $ORACLE_BASE/admin/$ORACLE_SID/upgrade catupgrd.sql



-n is for parallelism. 4 is the default no of parallelism.

-l is for explicit log location directory.



The db gets shutdown the upgrade completes. Check the upgrade logs and make sure that there is no error occurred in it.

Exadata Ibswitches patching

Imp points:
Starting with release 11.2.3.3.0, the patchmgr utility is used to upgrade and downgrade the InfiniBand switches.
IB Switch patch is delievered with Exadata storage patch.
IB Switch patches are released semi annually to annually.
IB Switch can be patched in Rolling fashion only.


Environment
Exadata Half Rack X4-2
4 Compute nodes, 7 Storage cells and 2 IB Switches
Current IB Switch Version 2.2.7-1



Steps:
1. Identify the number of switches in clusters.

[root@dm01dbadm01 ~]# ibswitches
Switch : 0x002128469b8aa0a0 ports 36 “SUN DCS 36P QDR dm01sw-iba01 10.209.41.246” enhanced port 0 lid 5 lmc 0
Switch : 0x002128469b97a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ibb01 10.209.41.247” enhanced port 0 lid 4 lmc 0


2. Identify the current IB switch software version on all the Switches
[root@dm01db01 patch_18.1.12.0.0.190111]# ssh dm01sw-iba01 version
SUN DCS 36p version: 2.2.7-1
Build time: Aug 4 2017 12:20:53
SP board info:
Manufacturing Date: 2014.05.20
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010



3. Log in to Exadata Compute node 1 as root user and navigate the Exadata Storage Software staging area

[root@dm01dbadm01 ESS_121220]# cd /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111
[root@dm01dbadm01 patch_18.1.12.0.0.190111]# pwd
/u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111

4. Execute the following to perform the IB Switch precheck
[root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade -ibswitch_precheck

5. Upgrade the IB Switches using the following command:
[root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade


6. Verify that all the IB Switches are upgraded to latest version.
[root@dm01db01 ~]# ssh dm01sw-ibb01 version
SUN DCS 36p version: 2.2.11-2
Build time: Aug 27 2018 11:18:39
SP board info:
Manufacturing Date: 2014.05.19
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
[root@dm01db01 ~]#

[root@dm01db01 ~]# ssh dm01sw-iba01 version
SUN DCS 36p version: 2.2.11-2
Build time: Aug 27 2018 11:18:39
SP board info:
Manufacturing Date: 2014.05.20
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

Exadata compute nodes patching

Exadata compute nodes patching

Imp points:

Patching a database node takes 45 minutes to one hour

You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

Starting with release 12.2.1.1.0, Exadata software updates for the Exadata database server can be applied only through the patchmgr utility. The dbserver.patch.zip file contains a special version of patchmgr used to orchestrate dbnodeupdate in updating Exadata Databases Nodes.

The patchmgr utility for updating Exadata database servers is not the same as the patchmgr script shipped with the Exadata software update.

Pre-requisites:

1. Note the output of below commands from each db node:
imageinfo
crsctl stat res –t

2. Check and note the ilom ip of each db node. connect to ilom of each db node and verify that each ilom server is running fine.

3. If you are planning to update all Exadata database servers at once, it is a requirement to run the update utility from a Linux node outside the group of Exadata database servers being updated. This is because the update utility cannot update the Exadata database server it is currently running on. If you have no other systems running Oracle Linux or Oracle Solaris you can run the update utility from one of the Exadata database servers. In such cases be sure that the Exadata database server where the update utility is running is not listed in the dbs_group file you specify.

4. You need to set up ssh equivalence for the root user from the driving node to the root user of all Exadata database servers that will be updated.

5. Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1. verify that the patchmgr utility is of latest version.

6. Download the latest Exachk from My Oracle Support note 1070954.1.

7. Run ExaChk tool

[root@dm01 ]# ./exachk

8. Run prerequisite check with the –nomodify_at_prereq flag weeks before doing the patching.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/

p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3

-nomodify_at_prereq &


If it find any unknown rpm package which is conflicting with the patch, so it will show error like below:

exadb01: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb01:/var/log/cellos/unkown_packages-rpt.070519194937.txt for more details.


Imp Contents of this file:

ALERT : Custom packages found (see above)

# These custom packages MUST be removed before proceeding a major Oracle Linux upgrade.

# Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed - OR -

# Run /var/log/cellos/remove_unkown_packages.070519194937.sh manually as root before re-running prereq check.

#

# RECOMMENDED : Let dbnodeupdate.sh remove the packages right after the mandatory backup by using the -R flag

# WARNING : Removing custom packages may impact functionality

# NOTE : When removed functionality is still required after the update, reinstall the rpm after the update

#################################################################################


Note:

By default, prerequisite check warnings such as active NFS mounts result in a failed check. If you want to allow active NFS mounts, then starting in release 12.1.2.1.1 you can use the -allow_active_network_mounts flag.

9. Fix the issues found in these two reports and run them again for no issues.

Perform a “backup only” run using the - backup flag.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -backup -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3 -allow_active_network_mounts &

Note:
The -backup action backs up the active root and /boot file system only and is sufficient for rolling back (failed) updates.

Re-running the update utility with the -backup flag (or in default updating mode) will overwrite existing backups.

Intervention steps:
1. Perform a “backup only” run using the - backup flag.
2. Run ExaChk tool
3. Run prerequisite check with the –nomodify_at_prereq flag weeks before doing the patching.
4. Check and fix the critical issues found in them and run them again for no issues.
5. Remove any blocking rpms and re-run the prerequisite check to validate if all pre-requisite are passed.
6. Run the pre-requisite checks with out the –nomodify_at_prereq flag to allow the removal of conflicting rpms using ISO.

[root@dm01 ]# nohup ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/ p22750145_121230_Linux-x86-64.zip -target_version 12.1.2.3.0.160207.3 &

7. • Perform the update. Use the –nobackup flag to skip the backup because you already made a “backup only” run.

[root@dm01 ]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip -target_version 12.1.2.2.3.160720 -allow_active_network_mounts –nobackup &

Note:
You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

For permforming the update in rolling fashion, use –rolling option. Below is the example:

[root@myclustercel01 ~]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/SAVE/p28666206_*_Linux-x86-64.zip -target_version 18.1.9.0.0.181006 -allow_active_network_mounts -rolling &


8. After the update, Reinstall any non-Exadata rpms that you removed before the Exadata update if they are needed.

9. Run Exachk. Check the issues and fix them.

10. Run imageinfo on each node. Verify that each node is running with the expected version.

11. Run crsctl stat res –t on each node. Compare its output with the prechecks output.


Rollback Steps:

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -rollback -target_version 12.1.2.3.0.160207


Note:
Having only one inactive system partition limits the rollback options to only the previous active image.

Firmware updates are not rolled back when rolling back to a previous image.

The Oracle Exadata System Software releases support later firmware releases. After rolling back, run the following commands to apply older firmware versions if needed:

/etc/init.d/lsidiag stop
/etc/init.d/lsi_mrdsnmpd stop
/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

The last command only applies to releases 11.2.3.3.0 or later.