Blogs

ODA Server Patching Failed while Patching Oracle’s Integrated Lights Out Manager

Introduction

Recently, I was working on patching ODA X6-2L from 19.6 to 19.10. Server patching failed when trying to upgrade the ilom/bios with an error.

DCS-10001:Internal error encountered: Failed update Ilom using USB.

[root@dbserv]# odacli update-server -v 19.10.0.0.0 -sko
{
   "jobId" : "04d92fb2-aaad-4b84-8e96-b0be10e04b1d",
   "status" : "Created",
   "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
   "reports" : [ ],
   "createTimestamp" : "August 30, 2021 11:05:02 AM EDT",
   "resourceList" : [ ],
   "description" : "Server Patching",
   "updatedTime" : "August 30, 2021 11:05:02 AM EDT"
[root@dbserv]# odacli describe-job -i "04d92fb2-aaad-4b84-8e96-b0be10e04b1d"

Job details
----------------------------------------------------------------
               ID: 04d92fb2-aaad-4b84-8e96-b0be10e04b1d
      Description: Server Patching
           Status: Failure
          Created: August 30, 2021 11:05:02 AM EDT
          Message: DCS-10001:Internal error encountered: Failed update Ilom using USB.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching                          August 30, 2021 11:05:09 AM EDT     August 30, 2021 11:24:04 AM EDT     Failure
Server patching                          August 30, 2021 11:05:09 AM EDT     August 30, 2021 11:24:04 AM EDT     Failure
Patch location validation                August 30, 2021 11:05:11 AM EDT     August 30, 2021 11:05:11 AM EDT     Success
dcs-controller upgrade                   August 30, 2021 11:05:11 AM EDT     August 30, 2021 11:05:11 AM EDT     Success
Creating repositories using yum          August 30, 2021 11:05:11 AM EDT     August 30, 2021 11:05:14 AM EDT     Success
Updating YumPluginVersionLock rpm        August 30, 2021 11:05:14 AM EDT     August 30, 2021 11:05:14 AM EDT     Success
Applying OS Patches                      August 30, 2021 11:05:14 AM EDT     August 30, 2021 11:18:11 AM EDT     Success
Creating repositories using yum          August 30, 2021 11:18:11 AM EDT     August 30, 2021 11:18:12 AM EDT     Success
Applying HMP Patches                     August 30, 2021 11:18:12 AM EDT     August 30, 2021 11:18:31 AM EDT     Success
Client root Set up                       August 30, 2021 11:18:31 AM EDT     August 30, 2021 11:18:35 AM EDT     Success
Client grid Set up                       August 30, 2021 11:18:35 AM EDT     August 30, 2021 11:18:51 AM EDT     Success
Patch location validation                August 30, 2021 11:18:51 AM EDT     August 30, 2021 11:18:51 AM EDT     Success
oda-hw-mgmt upgrade                      August 30, 2021 11:18:51 AM EDT     August 30, 2021 11:19:23 AM EDT     Success
OSS Patching                             August 30, 2021 11:19:23 AM EDT     August 30, 2021 11:19:24 AM EDT     Success
Applying Firmware Disk Patches           August 30, 2021 11:19:24 AM EDT     August 30, 2021 11:19:43 AM EDT     Success
Applying Firmware Controller Patches     August 30, 2021 11:19:43 AM EDT     August 30, 2021 11:19:56 AM EDT     Success
Server patching                          August 30, 2021 11:19:56 AM EDT     August 30, 2021 11:24:04 AM EDT     Failure
Checking Ilom patch Version              August 30, 2021 11:19:58 AM EDT     August 30, 2021 11:20:00 AM EDT     Success
Patch location validation                August 30, 2021 11:20:00 AM EDT     August 30, 2021 11:20:00 AM EDT     Success
Save password in Wallet                  August 30, 2021 11:20:00 AM EDT     August 30, 2021 11:20:00 AM EDT     Success
Server patching                          August 30, 2021 11:20:00 AM EDT     August 30, 2021 11:24:04 AM EDT     Failure
Apply Ilom patch                         August 30, 2021 11:20:00 AM EDT     August 30, 2021 11:24:04 AM EDT     Failure

This is a known issue addressed by Oracle support in (Doc ID 2733383.1). It applies to the below Oracle Database Appliance Hardware:

Oracle Database Appliance X7-2 S/M Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X8-2S/M Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X7-2 HA Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X8-2HA Hardware - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X6-2 S/M/L Hardware - Version All Versions to All Versions [Release All Releases]

Cause

There is a firewall in between the Oracle ILOM (Intergrated Lights Out Manager) network and the host network that is blocking port 623 (IPMI over UDP).

Recommendation

I highly recommend confirming the following two checks before you start server patching. You may add the below checks to your ODA pre-checks.
1- There is NO firewall in between the ILOM network and the host network that is blocking port 623.
You may add the below command to your ODA pre-checks to check if there is a network problem.

--- Run the command from ODA server. The command will prompt to enter ILOM root password.
ipmitool -I lanplus -H <ilomHOSTNAME> -U root chassis status

Example:

[root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis
status
Password:
Error: Unable to establish IPMI v2 / RMCP+ session
>>> Above error message indicates there is a firewall in between the ILOM network and the host network that is blocking port 623.

[root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis
status
Password:
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     :
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
>>> There is NO a firewall in between the ILOM network and the host network that is blocking port 623.

2- Make sure ILOM ipmi setting (/SP/services/ipmi/ipmi v2_0_sessions) is enabled.

[root@dbserv]# ssh dbserv-ilo.local
-> show /SP/services/ipmi

/SP/services/ipmi
Targets:

Properties:
servicestate = enabled
tls_sessions = enabled
v1_5_sessions = disabled
v2_0_sessions = enabled    <<< this should be enabled

--- If v2_0_sessions shows disabled, then you can enable it using
-> set /SP/services/ipmi v2_0_sessions=enabled
Set 'v2_0_sessions' to 'enabled'

More from this Author