ZDM Installation Error in Oracle Linux 8 (Failed to initialize MySQL) and How to fix it

Share on:

zdm

Introduction

Zero Downtime Migration (ZDM) is one of the best options available to migrate your Oracle Database to the Oracle Cloud. Automation and simplicity are among their strong benefits besides the zero downtime features. Although migrating in OCI (Oracle Cloud Infrastructure) is the classic use case (see Oracle Migration & Integration Specialist Certification), I had the chance to try a different scenario lately, where the path was On-Prem Linux to Oracle Exadata at Customer.

In this blog post, I’ll describe the blocking issue I had while installing ZDM io an On-prem VM, and provide the solution.

 

1. My ZDM Environment

Discloser

Although, an SR was already opened for this issue and the Oracle Dev Team seemed to make it work in a similar environment. I could still not get ZDM to be installed in my VM (Virtual Machine), hence this blog post explains both the error causing MySQL initialization failure and the workaround.

 

Virtual Machine

OS Oracle Linux 8.4 kernel 5.4.17-2102.201.3.el8uek.x86_64
File system: /dev/mapper/vg01-lvol1 on /u01 type ext4 (rw,relatime,seclabel)
MySQL server 8.0.22
ZDM 21.3 build

[zdmuser@zdmserver ~]$ uname -a
Linux zdmserver 5.4.17-2102.201.3.el8uek.x86_64 
[zdmuser@zdmserver ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

 

Prerequisites

After downloading the ZDM installable Zip and extracting it, I have created the necessary directories for the installation

export INVENTORY_LOCATION=/u01/app/oraInventory
export ORACLE_BASE=/u01/app/oracle
export ZDM_BASE=/u01/app/oracle/zdmbase ----> ZDM config files, logs 
export ZDM_HOME=/u01/app/oracle/zdmbase/zdm21 ----> ZDM software binaries
export ZDM_INSTALL_LOC=/u01/zdm21-inst ----> ZDM installable
- Create directories
[zdmuser@zdmserver]$ mkdir -p $ORACLE_BASE $ZDM_BASE $ZDM_HOME $ZDM_INSTALL_LOC

 

Steps to Reproduce the Error

All I had to do is to run the install script with the required arguments (directories) to reproduce the behavior.

$ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
ziploc=./zdm_home.zip -zdm

 

Failure to initialize MySQL

  • The error as you can see happening at MySQL configuration stage and lacks any relevant description
Setting up MySQL...
---------------------------------------
Failed to initialize MySQL
Failed to initialize MySQL
One or more errors occurred while setting up No GI RHP.
Trying to stop MySQL in case it was started and left up.
spawn /u01/app/oracle/zdm21/mysql/server/bin/mysqladmin --defaults-file=/u01/app/oracle/crsdata/velzdm2prm/rhp/conf/my.cnf -u root -p shutdown 
WARNING: Failed to stop MySQL

Now, at this point, we can either, retry the installation forever, or start digging further.

Which Log to Check 

I had no clue really, but I decided to just hit find  under /u01 and try my luck with “mysql” as filter 😉 .

JACKPOT! here is the winner

/u01/app/oracle/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log

You got it, always look under $ORACLE_BASE/crsdata/myserver/rhp/mysql/metadata for such log 

 

The Actual Error

Clearly, the MySQL creation step failed because temporary files couldn’t be created in an ext FS as shown below.

$ more $ORACLE_BASE/zdmbase/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log
…
 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
 1 [ERROR] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data 
for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error 
number 22. 1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Out of disk space

 

What does it really mean?

In MySQL8, InnoDB files are created using a specific function (like dd) called posix_fallocate, which just writes 000s in a disk file to reserve space. See description below

zdm2

I obviously shared this in the SR, but I wanted to see what MySQL community had to say about it, so I looked it up.

 

MySQL Bug or EXT Dislike

It didn’t take long before I found the answer in stackoverflow . As explained below it seems that posix_fallocate function isn’t supported by ext file systems.

zdm3  

 

In a nutshell, we have two bugs filed related to the same issue

 

Solution: Replace ext with XFS

From the above bugs, two solutions were available for our ZDM installation problem.

  1. MySQL8 bug received a patch as a fix from a contributor. Contribution: fix_init_fail_on_ext3.patch
  2. The older bug for MySQL 5.7 had no permanent fix but using XFS got rid of the posix_fallocate bug.

I decided to go for an XFS disk on /u01, mostly because it was the least intrusive option and particularly after reading Dimitri Kravtchuk piece about MySQL perf regression on ext4 (MySQL Performance : XFS -vs- EXT4 Story) where he recommended moving to XFS for kernels higher than 4.1.

[root@zdmserver~]# mount|grep u01
/dev/mapper/vg01-lvol1 on /u01old type ext4 (rw,relatime,seclabel)
/dev/mapper/vg01-lvol2 on /u01 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
  • ZDM was successfully installed after cleanup and reinstall.

-- Uninstall
[zdmuser@zdmserver]$ $ZDM_HOME/bin/zdmservice stop deinstall
-- Re-Install
$ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \
ziploc=./zdm_home.zip -zdm

zdm4

 

Conclusion

This was an issue that had nothing to do with Oracle but allowed me to discover ZDM logs directories and choose the right file system for its MySQL DB. Hope this will help anyone who runs into the same error.

Share on:

More from this Author

OCI FortiGate HA Cluster – Reference Architecture: Code Review and Fixes

Introduction OCI Quick Start repositories on GitHub are collections of Terraform scripts and configurations provided by Oracle. These repositories ... Read More

What Autoupgrade Won’t Catch for You when Moving to 19c Part1: Ghost OLAP

Introduction So far, I have used Oracle AutoUpgrade, many times in 3 different OS’. Yet the more you think you’ve seen it all and reached the ... Read More

Back to Top