How to downgrade InfiniBand switch patches on Exadata
Lately I was asked to apply the Exadata Quarterly Full Stack Download Patch (QFSDP) of July 2018(Patch 28183368) on the infrastructure components of a virtualized EXADATA X6-2.
The objective consists of patching the following components:
I found a lot of instructions and examples on how to upgrade the components, but none on how to downgrade in case of major issues.
I needed to test the downgrade for a specific customer.
Most of the downgrade procedures were simple enough and well documented in the readme of the patches. However, the InfiniBand switch patch downgrade was more complex.
The particularity of this QFSDP InfiniBand patch upgrade is that it will automatically perform 2 upgrades to update our switches from version 2.2.6-2 to version 2.2.9-3.
- Upgrade to 2.2.7-2 version
- Upgrade to 2.2.9-3 version
This makes the downgrade more difficult since you must proceed with 2 downgrades to go back to your initial version (2.2.6-2)
The readme provided with the InfiniBand switch patch, points you to the Oracle Exadata Database machine Maintenance guide.
In this guide, you are instructed you to use the patchmgr utility with the downgrade parameter.
After multiple unsuccessful precheck attempts receiving error messages which were probably related to the fact that the patch had to perform 2 software upgrades, I used the following technique documented in Sun Datacenter documentation: https://docs.oracle.com/cd/E76424_01/html/E76431/z400029a1775330.html#scrolltoc
To perform this test, I used the Exadata in our lab.
The InfiniBand switches version were already at version 2.2.11-2.
There is no path for direct one step downgrade from 2.2.11-2 to 22.214.171.124 and therefore we have to use the two steps approach:
- downgrade to version 2.2.7-2
- downgrade to version 2.2.6-2
Please note that you should to downgrade 1 switch at a time.
1. Download the patches to the compute node inside /EXAVMIMAGES/QFSDP/ibswitch_downgrade/
Patch 26575824: Sun Datacenter InfiniBand Switch 36 Firmware Version 2.2.7 Patch 26415350: Sun Datacenter InfiniBand Switch 36 Firmware Version 2.2.6 patch_bug_26678971: This patch is related to Exadata Critical issue IB6. (IB6) InfiniBand switch does not boot when attempting to upgrade (Doc ID 2309926.1)
Downgrade the switches to version 2.2.7-2
2. On a compute node which has passwordless SSH configured, unzip 2.2.7-2 patch as root:
cd /EXAVMIMAGES/QFSDP/ibswitch_downgrade unzip p26575824_227_Generic.zip
3. Connect to the switch as ilom-admin
4. Load IB switch image 2.2.7-2
load -source sftp://root:email@example.com:/EXAVMIMAGES/QFSDP/ibswitch_downgrade/SUN_DCS_36p_2.2.7-1/SUN_DCS_36p/sundcs_36p_repository_2.2.7_2_signed.pkg
5. Reboot the switch
6. Check InfiniBand switch version
7. Downgrade the second switch using the same technique: (step 3 to step 6)
Once completed, you are ready to downgrade the switches from 2.2.7-2 to 2.2.6-2.
8. On the same compute node, unzip 2.2.6-2 patch as root:
cd /EXAVMIMAGES/QFSDP/ibswitch_downgrade/2262 unzip p26415350_226_Generic.zip
9. Connect to the first switch as ilom-admin
10. Load image 2.2.6-2
Difference between this command and the one used for 2.2.7 => No « : » between ip and directory parameters load -source sftp://root:firstname.lastname@example.org/EXAVMIMAGES/QFSDP/ibswitch_downgrade/2262/SUN_DCS_36p_2.2.6-2/SUN_DCS_36p/sundcs_36p_repository_2.2.6_2.pkg
11. Reboot the switch
12. Verify version
13. Ensure subnet manager is running
14. Validate configuration
15. Downgrade the second switch using the same technique: (step 9 to step 14)
Apply corrective patch for Exadata Critical issue IB6
16. Apply Patch for bug 26678971 – MOS 2302714.1
a) Copy patch on first switch in /tmp scp patch_bug_26678971 email@example.com:/tmp/. b) Change permission in /tmp chmod 770 patch_bug_26678971 c) Execute the patch /tmp/patch_bug_26678971 d) Remove the patch from /tmp rm -f patch_bug_26678971
17. Apply Patch for bug 26678971 on the second switch (repeat step 16 for the second switch)
Now the InfiniBand switches are back to the desired version (2.2.6-2) for this downgrade test. As you noticed, using this technique is quite simple and effective.
I hope this article was helpful to you!