Oracle 23c: Managing Flashback Database Logs Outside the Fast Recovery Area
Overview: In previous Database releases, you could store flashback database logs only in the fast recovery area Oracle 23c Database introduces the ... Read More
Learn more about why Eclipsys has been certified as a Great Place to Work in Canada, Best Workplaces in Ontario and Technology, and named Canada’s Top SME Employer for 3 years!
Learn more!Recently, I was working on patching ODA X6-2L from 19.6 to 19.10. Server patching failed when trying to upgrade the ilom/bios with an error.
DCS-10001:Internal error encountered: Failed update Ilom using USB.
[root@dbserv]# odacli update-server -v 19.10.0.0.0 -sko { "jobId" : "04d92fb2-aaad-4b84-8e96-b0be10e04b1d", "status" : "Created", "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.", "reports" : [ ], "createTimestamp" : "August 30, 2021 11:05:02 AM EDT", "resourceList" : [ ], "description" : "Server Patching", "updatedTime" : "August 30, 2021 11:05:02 AM EDT" [root@dbserv]# odacli describe-job -i "04d92fb2-aaad-4b84-8e96-b0be10e04b1d" Job details ---------------------------------------------------------------- ID: 04d92fb2-aaad-4b84-8e96-b0be10e04b1d Description: Server Patching Status: Failure Created: August 30, 2021 11:05:02 AM EDT Message: DCS-10001:Internal error encountered: Failed update Ilom using USB. Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Server patching August 30, 2021 11:05:09 AM EDT August 30, 2021 11:24:04 AM EDT Failure Server patching August 30, 2021 11:05:09 AM EDT August 30, 2021 11:24:04 AM EDT Failure Patch location validation August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:11 AM EDT Success dcs-controller upgrade August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:11 AM EDT Success Creating repositories using yum August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:14 AM EDT Success Updating YumPluginVersionLock rpm August 30, 2021 11:05:14 AM EDT August 30, 2021 11:05:14 AM EDT Success Applying OS Patches August 30, 2021 11:05:14 AM EDT August 30, 2021 11:18:11 AM EDT Success Creating repositories using yum August 30, 2021 11:18:11 AM EDT August 30, 2021 11:18:12 AM EDT Success Applying HMP Patches August 30, 2021 11:18:12 AM EDT August 30, 2021 11:18:31 AM EDT Success Client root Set up August 30, 2021 11:18:31 AM EDT August 30, 2021 11:18:35 AM EDT Success Client grid Set up August 30, 2021 11:18:35 AM EDT August 30, 2021 11:18:51 AM EDT Success Patch location validation August 30, 2021 11:18:51 AM EDT August 30, 2021 11:18:51 AM EDT Success oda-hw-mgmt upgrade August 30, 2021 11:18:51 AM EDT August 30, 2021 11:19:23 AM EDT Success OSS Patching August 30, 2021 11:19:23 AM EDT August 30, 2021 11:19:24 AM EDT Success Applying Firmware Disk Patches August 30, 2021 11:19:24 AM EDT August 30, 2021 11:19:43 AM EDT Success Applying Firmware Controller Patches August 30, 2021 11:19:43 AM EDT August 30, 2021 11:19:56 AM EDT Success Server patching August 30, 2021 11:19:56 AM EDT August 30, 2021 11:24:04 AM EDT Failure Checking Ilom patch Version August 30, 2021 11:19:58 AM EDT August 30, 2021 11:20:00 AM EDT Success Patch location validation August 30, 2021 11:20:00 AM EDT August 30, 2021 11:20:00 AM EDT Success Save password in Wallet August 30, 2021 11:20:00 AM EDT August 30, 2021 11:20:00 AM EDT Success Server patching August 30, 2021 11:20:00 AM EDT August 30, 2021 11:24:04 AM EDT Failure Apply Ilom patch August 30, 2021 11:20:00 AM EDT August 30, 2021 11:24:04 AM EDT Failure
This is a known issue addressed by Oracle support in (Doc ID 2733383.1). It applies to the below Oracle Database Appliance Hardware:
Oracle Database Appliance X7-2 S/M Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X8-2S/M Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X7-2 HA Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X8-2HA Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X6-2 S/M/L Hardware - Version All Versions to All Versions [Release All Releases]
Cause
There is a firewall in between the Oracle ILOM (Intergrated Lights Out Manager) network and the host network that is blocking port 623 (IPMI over UDP).
Recommendation
I highly recommend confirming the following two checks before you start server patching. You may add the below checks to your ODA pre-checks.
1- There is NO firewall in between the ILOM network and the host network that is blocking port 623.
You may add the below command to your ODA pre-checks to check if there is a network problem.
--- Run the command from ODA server. The command will prompt to enter ILOM root password. ipmitool -I lanplus -H <ilomHOSTNAME> -U root chassis status
Example:
[root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis status Password: Error: Unable to establish IPMI v2 / RMCP+ session >>> Above error message indicates there is a firewall in between the ILOM network and the host network that is blocking port 623. [root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis status Password: System Power : on Power Overload : false Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : always-off Last Power Event : Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false >>> There is NO a firewall in between the ILOM network and the host network that is blocking port 623.
2- Make sure ILOM ipmi setting (/SP/services/ipmi/ipmi v2_0_sessions) is enabled.
[root@dbserv]# ssh dbserv-ilo.local -> show /SP/services/ipmi /SP/services/ipmi Targets: Properties: servicestate = enabled tls_sessions = enabled v1_5_sessions = disabled v2_0_sessions = enabled <<< this should be enabled --- If v2_0_sessions shows disabled, then you can enable it using -> set /SP/services/ipmi v2_0_sessions=enabled Set 'v2_0_sessions' to 'enabled'
Overview: In previous Database releases, you could store flashback database logs only in the fast recovery area Oracle 23c Database introduces the ... Read More
Overview: Oracle 23c Database introduces a new feature to control whether a user or session is enabled for read-write operations, irrespective of the ... Read More