OVM Series Part 2: What to collect when opening a Service Request

Share on:


In my last post, I described a few commands within Oracle OVM  Manager CLI. This time, the topic is about how to identify the relevant information of your OVM (Oracle Virtual Machine) Environment as well as the logs and their location for each component.

OVM Servers and OVM Manager have their own logs, and when issues occur it’s good to have them handy before reaching out to Oracle Support. Checking the details of your configuration preemptively also helps to make sure you are communicating the right release number of the components along with their state quickly.


Check the configuration

Oracle VM Manager

Configuration can be checked through the ovm_admin command. Content is usually set by default

[root@em1 ~]# /u01/app/oracle/ovm-manager-3/bin/ovm_admin --listconfig

Oracle VM Manager Release 3.4.4 Admin tool
Oracle VM Manager Configuration
Database type               : MySQL 
Database Server hostname    : localhost
Database name               : ovs <-- Database storing the ovmm metadata 
Database Listener port      : 49500
Oracle VM Database user     : ovs
WebLogic Server admin       : weblogic
Oracle VM Manager admin     : admin

Or by checking the configuration file  where you can find the OVMM UUID and build number useful in case of reinstall/restore

[root@em1 ~]# cat /u01/app/oracle/ovm-manager-3/.config
BUILDID= <--- exact build number

Here you can have a look at the ovmm service  and the MySQL backup configuration

[root@em1 ~]# cat  /etc/sysconfig/ovmm

Oracle VM server

Configuration can be checked  through The xm commands which interact with the Xen hypervisor in the OVM host

[root@ovm-01 ~]# xm info
host                   : ovm-01
release                : 4.1.12-124.20.3.el6uek.x86_64
version                : #2 SMP Thu Oct 11 17:47:32 PDT 2018
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 3059
hw_caps                : bfebfbff:2c100800:00000000:01703f00:029ee3ff:00000000:xxx:000
virt_caps              : hvm hvm_directio
total_memory           : 147447
free_memory            : 75478
free_cpus              : 0
xen_major              : 4
xen_minor              : 4
xen_extra              : .4OVM
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=max:3792M allowsuperpage dom0_vcpus_pin dom0_max_vcpus=20 crashkernel=512M@64M
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18.0.7)
cc_compile_by          : mockbuild
cc_compile_domain      : us.oracle.com
cc_compile_date        : Thu Sep  6 08:24:27 PDT 2018
xend_config_format     : 4

-- list the vms in the ovm host
[root@ovm-server04 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
0004fb00000600000f199f55466d4b84            14  8199     1     -b---- 100696.0
0004fb00000600001a440b79c4e6e194            20  4099     2     -b----  36437.3

Directory purpose
------------ -------------------------------------------------------------------------

/etc/xen      Contains OVM Server configuration files for the OVM Server daemon and virtualized guests.


OVM agent

You can check the version by running an rpm command on the agent package or by checking its service status

[root@ovm-04 ~]# rpm -aq ovs-agent

[root@ovm-04 ~]# service ovs-agent status


Available logs

Oracle VM Server directories
 All log directories are usually located under /var/log directory, each of them logs information for a specific module

Directory         Purpose
----------------- ----------------------------------------------------------------------- /var/log          Contains the OVM Agent log file, ovs-agent.log.
                  Contains the ovmwatch.log, which logs virtual machine life cycle events.
                  Contains the ovm-consoled.log, which logs remote VNC console access, and all communication with OVM Manager. /var/log/xen      Contains OVM Server log files.
/var/log/messages Contains OVM Server messages.
 LogFile          ------      Purpose
-------------------------- ----------------------------------------------------------------
/var/log/xen/xend.log    ----  All OVM Server daemon actions. Same output as xm log command. 
/var/log/xen/xend-debug.log    More detailed logs  of OVM Server daemon. 
/var/log/xen/xen-hotplug.log   log of hotplug events if a device or network script doesn’t start up or become available. 
/var/log/xen/qemu-dm.pid.log   log for each hardware virtualized guest. Replace the pid this in the file name. 
/var/log/ovs-agent.log         log for OVM Agent. 
/var/log/osc.log               log for OVM Storage Connect Plug-ins. 
/var/log/ovm-consoled.log      log for the OVM virtual machine console.
/var/log/ovmwatch.log          log for the OVM watch daemon. 

Oracle VM Server Command-Line Tools
Besides tailing ovs-agent.log you can also use xm diagnostic commands

command   Purpose
--------- ----------------------------------------------------------
xentop    Displays real-time CPU/Mem information about OVM Server and domains
xm dmesg  Displays log information on the hypervisor.
xm log    Displays log information of the OVM Server daemon.

OVM manager

The main log file is AdminServer.log which has multiple versions according to its rotation.

  Directory File
-------------------------------------------------------------------------- -----------
/u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs/ AdminServer.log     

You can also use an OVM tool called OvmLogTool.py to generate an error summary log based on all AdminServer.log versions

[root@ovm-manager01 bin]# cd /u01/app/oracle/ovm-manager-3/ovm_tools/bin [root@ovm-manager01 bin]# python OvmLogTool.py -s -o summary
processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs/AdminServer.log00001 processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs/AdminServer.log

[root@ovm-manager01 bin]# ll summary
-rw-r--r--. 1 root root 985903 Jul 25 06:25 summary


SR-related diagnostic logs

Oracle support will probably ask you the following logs right after opening the SR.


This report will collect diagnostic and configuration information from the OVM manager host and all linked OVM servers.

i.e.: (rpm versions, syslog, network config, filesystems, disk partition details, loaded kernel modules & status of all services)

[root@em1 ~]# sosreport -v 
sosreport (version 3.4)
Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [em1]:
Please enter the case id that you are generating this report for []: 3-236xxxxx

Setting up archive ...
Setting up plugins ...
Running plugins. Please wait ...
Running 94/94: yum...
Creating compressed archive...
Your sosreport has been generated and saved in:/var/tmp/sosreport-em1.3-236xxx-YYYYMMDD.tar.xz

Note sometimes the report fails to run on one of the OVM servers. In that case, just run it separately in each OVM server. 

VMPINFO Diagnostic Tool For Oracle

VMPinfo is a script that collects diagnostic information for OVM, including the OVM Manager and all linked Oracle VM Servers. this script includes the run of sosreport, hence it’s enough to only run VMPinfo command without sosreport 

[root@em1]# cd /u01/app/oracle/ovm-manager-3/ovm_tools/support/
[root@em1 support]# ./vmpinfo3.sh --username=admin listservers
Enter OVM Manager Password:
The following server(s) are owned by this manager: ['ovm-01', 'ovm-02']
[root@ovm-manager01 support]# ./vmpinfo3.sh --username=admin

Enter OVM Manager Password:

Gathering files from all servers. This process may take some time.
Gathering OVM Model Dump files Gathering sosreport from ovm-01 Gathering sosreport from ovm-02
Data collection complete
Gathering OVM Manager Logs
Clean up metrics
Copying model files
Copying DB backup log files
Running lightweight sosreport
Archiving vmpinfo3-20210725-073817

Please send /tmp/vmpinfo3- to Oracle support


Check for Database Corruption in OVM Manager

OVM manager operations and metadata are stored in a MySQL repository database. The database is automatically backed up daily but if any corruption is detected the backups will stop and all new changes won’t be recovered in case of a crash. 

[root@em1 mysql]# cd  /u01/app/oracle/mysql/data
[root@em1 data] # cat my.cnf |grep log
[root@em1 mysql]# tail mysqld.err

Consistency check

1. Look for ONF (Object Not Found) errors in OVMM DB.

[root@em1 ~]# /usr/bin/ovm_shell.sh -u admin
    OVM Shell: Interactive Mode
    --- Run below commands in this order in OVMM shell prompt.
    >>> om = OvmClient.getOvmManager()
    >>> f = om.getFoundryContext()    
    >>> f.fixupScan()                 
    [11509]   <--- If the result is not empty then there is corruption and inconsistencies in OVMM DB.

2. Validate if Daily AutoFullBackup DB backups stopped working.

[root@em1 ~]# ls -ldt /u01/app/oracle/mysql/dbbackup/AutoFullBackup*
/u01/app/oracle/mysql/dbbackup/AutoFullBackup-20210722_222459 <--- must be <=24h old

3. Look for Object Not Found (ONF) and Cluster is null errors in OVM Manager Admin Server logs.

[root@em1 ~]# egrep -iR "cluster is null|ObjectNotFound|inconsistencies" /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs/AdminServer*

Action If corruption is confirmed, then perform an OVM Manager DB rebuild (Regeneration)  Doc ID 2038168.1



This was just a preview of what you might look at when troubleshooting the OVM environment. For my part, the common issues happening most of the time are related to some job locks or network stack hiccups. For the latter, the Support will also ask you to run commands like “netstat -ltnp” , “nc –zv” or “tcpdump” between the manager and the OVM hosts. In the next post, we will be covering the backup and recovery tools available for OVM.

Thank you for reading!

Share on:

More from this Author

OCI FortiGate HA Cluster – Reference Architecture Code Review and Fixes

OCI FortiGate HA Cluster – Reference Architecture: Code Review and Fixes

Introduction OCI Quick Start repositories on GitHub are collections of Terraform scripts and configurations provided by Oracle. These repositories ... Read More

What Autoupgrade Won’t Catch for You when Moving to 19c Part1 Ghost OLAP

What Autoupgrade Won’t Catch for You when Moving to 19c Part1: Ghost OLAP

Introduction So far, I have used Oracle AutoUpgrade, many times in 3 different OS’. Yet the more you think you’ve seen it all and reached the ... Read More

Back to Top