Blogs

How to avoid the known issues with upgrading Oracle Database Appliance to 18.3

Recently, when I worked on upgrading ODA X7-2 S/M/L from version 12.2.1.4.0 to version 18.3.0.0.0. I hit two known issues after patching server components.

• Issue# 1: Onboard public network interfaces do not come up after patching or imaging.
• Issue# 2: Upgrading to 18.3 can automatically result in ADVM processes consuming Excessive CPU and IO during automatic “resilvering”. After an upgrade ADVM processes are re-disguising consuming high CPU/IO for a long time.

This blog will cover those two known issues and show how we can proactively handle them before upgrading ODA to 18.3.

To upgrade to Oracle Database Appliance Release 18.3, your deployment must be on Release 12.1.2.12 or later.

Issue# 1: Onboard public network interfaces do not come up after patching or imaging

When you apply patches or re-image Oracle Database Appliance, the onboard public network interfaces may not come up due to faulty status presented in the ILOM and remote SSH access will not be available as a result.

• Hardware Models: Oracle Database Appliance X7-2-HA, X7-2S, X7-2M
• Workaround is described in Oracle ODA 18.3 known issues document
  https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtrn/issues-with-oda-odacli.html#GUID-796ED4E6-04F6-4E09-9CB9-85DD75538C05

You can proactively handle this issue before upgrading to 18.3 by upgrading server firmware to the version which contains the fix.

Solution:

1. Download firmware package 4.1 or later from here

In my case, I downloaded package version 1.5.1.

• Patch 29782147: Oracle Server X7-2 SW 1.5.0 – FIRMWARE PACK
• Package file: zip

2) Copy package zip file to ODA node.

3) Unzip package file using root Unzip will create folder “Oracle_Server_X7-2-1.5.0.90261-FIRMWARE_PACK”

4) Change directory to “Oracle_Server_X7-2-1.5.0.90261-FIRMWARE_PACK/Firmware/Oracle-BASE-T-25Gb-lom”
[root@oak ~] # cd Oracle_Server_X7-2-1.5.0.90261-FIRMWARE_PACK/Firmware/Oracle-BASE-T-25Gb-lom

5) Clear all faults on the ilom from ILOM Web Console.

6) To update firmware, you need to install bnxtnvm tool. If you are running 12.2.1.4.0, the bnxtnvm tool is already included, and you do not need to install the rpm.

7) Update the firmware. For Bare Metal, the device names are em2 and em3.
[root@oak ~]# bnxtnvm install 20.08.01.18-57417-LOM.pkg -dev=em2
[root@oak ~]# bnxtnvm install 20.08.01.18-57417-LOM.pkg -dev=em3

8) Power Cycle ODA node.

Reference:

• Upgrading the onboard SFP28 ports on an ODA X7-2 server node (Doc ID 2373070.1)
• Oracle x7-2 ILOM Has Detected That A PCIE Link Layer Is Inactive (SPX86A-8009-3J) (Doc ID 2411407).

Issue# 2: Upgrading to 18.3 can automatically result in ADVM processes consuming excessive CPU and IO during automatic “resilvering”

Description:

• The problem will raise after updating server using command (odacli update-server -v 18.3.0.0.0).
• The problem source has been identified as è Bug 29520544 – AFTER 18.3 UPGRADE ADVM PROCESSES ARE CONSUMING HIGH CPU/IO FOR A LONG TIME.
• Upgrade to 18c can cause unnecessary disk resilvering on ODA (and any other system with > 1MB AUs).
• Any ODA upgraded to 3 which uses ACFS can encounter this problem, but the impact will vary depending on hardware types, disk capacity, size of ACFS volumes, and disk speed.

Workaround: there is no workaround. The issue will be recovered automatically, but it will take time depending on hardware types, disk capacity, size of ACFS volumes, and disk speed. You can monitor recovery progress by watching “Mirror recovery for” message as below.

# grep ‘Mirror recovery for’ /u01/app/grid/crsdata/$(hostname -s)/acfs/event.log.0
# tail -f /proc/oks/log | grep dirty

Note: Yon can’t continue with ODA storage and database home patching until the issue is resolved.

You can proactively avoid that issue by patching GI home before starting with ODA upgrade. In my case, ODA version was 12.2.1.4, so I applied path 29597701

After you apply above patch on GI home, you can start with ODA upgrade and you will NOT face that problem once server update finishes.

Reference:

• ALERT: Upgrading to 18.x from 12.x Can Automatically Result in ADVM Processes Consuming Excessive CPU and IO During Automatic Resilvering (Doc ID 2525427.1)

More from this Author