How to downgrade InfiniBand switch patches on EXADATA

Lately I was asked to apply the Exadata Quarterly Full Stack Download Patch (QFSDP) of July 2018(Patch 28183368) on the infrastructure components of a virtualized EXADATA X6-2.

The objective consists of patching the following components:

I found a lot of instructions and examples on how to upgrade the components, but none on how to downgrade in case of major issues.
I needed to test the downgrade for a specific customer.

Most of the downgrade procedures were simple enough and well documented in the readme of the patches.  However, the InfiniBand switch patch downgrade was more complex.

The particularity of this QFSDP InfiniBand patch upgrade is that it will automatically perform 2 upgrades to update our switches from version 2.2.6-2 to version 2.2.9-3.

  1. Upgrade to 2.2.7-2 version
  2. Upgrade to 2.2.9-3 version

This makes the downgrade more difficult since you must proceed with 2 downgrades to go back to your initial version (2.2.6-2)

The readme provided with the InfiniBand switch patch, points you to the Oracle Exadata Database machine Maintenance guide.
In this guide, you are instructed you to use the patchmgr utility with the downgrade parameter.

After multiple unsuccessful precheck attempts receiving error messages which were probably related to the fact that the patch had to perform 2 software upgrades, I used the following technique documented in Sun Datacenter documentation:  https://docs.oracle.com/cd/E76424_01/html/E76431/z400029a1775330.html#scrolltoc

 

To perform this test, I used the Exadata in our lab.

The InfiniBand switches version were already at version 2.2.11-2.

There is no path for direct one step downgrade from 2.2.11-2 to 2.2.6.2 and therefore we have to use the two steps approach:

  1. downgrade to version 2.2.7-2
  2. downgrade to version 2.2.6-2

Please note that you should to downgrade 1 switch at a time.

 

Patches download

1. Download the patches to the compute node inside /EXAVMIMAGES/QFSDP/ibswitch_downgrade/

Patch 26575824: Sun Datacenter InfiniBand Switch 36 Firmware Version 2.2.7

Patch 26415350: Sun Datacenter InfiniBand Switch 36 Firmware Version 2.2.6
patch_bug_26678971: This patch is related to Exadata Critical issue IB6.

(IB6) InfiniBand switch does not boot when attempting to upgrade (Doc ID 2309926.1)

 

Downgrade the switches to version 2.2.7-2

2. On a compute node which has passwordless SSH configured, unzip 2.2.7-2 patch as root:

cd /EXAVMIMAGES/QFSDP/ibswitch_downgrade
unzip p26575824_227_Generic.zip

3. Connect to the switch as ilom-admin

4. Load IB switch image 2.2.7-2

load -source sftp://root:xxxxx@xx.xx.xx.xx:/EXAVMIMAGES/QFSDP/ibswitch_downgrade/SUN_DCS_36p_2.2.7-1/SUN_DCS_36p/sundcs_36p_repository_2.2.7_2_signed.pkg

Answer Y

5. Reboot the switch

6. Check InfiniBand switch version

7. Downgrade the second switch using the same technique: (step 3 to step 6)

 

Once completed, you are ready to downgrade the switches from 2.2.7-2 to 2.2.6-2.

8. On the same compute node, unzip 2.2.6-2 patch as root:

cd /EXAVMIMAGES/QFSDP/ibswitch_downgrade/2262
unzip p26415350_226_Generic.zip

9. Connect to the first switch as ilom-admin

10. Load image 2.2.6-2

Difference between this command and the one used for 2.2.7
=> No  « : »  between ip and directory parameters

load -source sftp://root:xxxxxxx@xx.xx.xx.xx/EXAVMIMAGES/QFSDP/ibswitch_downgrade/2262/SUN_DCS_36p_2.2.6-2/SUN_DCS_36p/sundcs_36p_repository_2.2.6_2.pkg

Answer Y

11. Reboot the switch

12. Verify version

13. Ensure subnet manager is running

14. Validate configuration

15. Downgrade the second switch using the same technique: (step 9 to step 14)

 

Apply corrective patch for Exadata Critical issue IB6

16. Apply Patch for bug 26678971 – MOS 2302714.1

a) Copy patch on first switch in /tmp
scp patch_bug_26678971 root@xx.xx.xx.xx:/tmp/.

b) Change permission in /tmp
chmod 770 patch_bug_26678971

c) Execute the patch

/tmp/patch_bug_26678971


d) Remove the patch from /tmp
rm -f patch_bug_26678971

17. Apply Patch for bug 26678971 on the second switch (repeat step 16 for the second switch)

 

Now the InfiniBand switches are back to the desired version (2.2.6-2) for this downgrade test. As you noticed, using this technique is quite simple and effective.

I hope this article was helpful to you!

Share:

No Comments

No comments yet.

RSS feed for comments on this post.

Leave a comment