Introduction Automating infrastructure provisioning with Terraform is nothing for many, but to truly harness IaC power, seamless integration with ... Read More
ZDM Installation Error in Oracle Linux 8 (Failed to initialize MySQL) and How to fix it
Zero Downtime Migration (ZDM) is one of the best options available to migrate your Oracle Database to the Oracle Cloud. Automation and simplicity are among their strong benefits besides the zero downtime features. Although migrating in OCI (Oracle Cloud Infrastructure) is the classic use case (see Oracle Migration & Integration Specialist Certification), I had the chance to try a different scenario lately, where the path was On-Prem Linux to Oracle Exadata at Customer.
In this blog post, I’ll describe the blocking issue I had while installing ZDM io an On-prem VM, and provide the solution.
1. My ZDM Environment
Although, an SR was already opened for this issue and the Oracle Dev Team seemed to make it work in a similar environment. I could still not get ZDM to be installed in my VM (Virtual Machine), hence this blog post explains both the error causing MySQL initialization failure and the workaround.
OS Oracle Linux 8.4 kernel 5.4.17-2102.201.3.el8uek.x86_64
File system: /dev/mapper/vg01-lvol1 on /u01 type ext4 (rw,relatime,seclabel)
MySQL server 8.0.22
ZDM 21.3 build
[zdmuser@zdmserver ~]$ uname -a Linux zdmserver 5.4.17-2102.201.3.el8uek.x86_64 [zdmuser@zdmserver ~]$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 (Ootpa)
After downloading the ZDM installable Zip and extracting it, I have created the necessary directories for the installation
export INVENTORY_LOCATION=/u01/app/oraInventory export ORACLE_BASE=/u01/app/oracle export ZDM_BASE=/u01/app/oracle/zdmbase ----> ZDM config files, logs export ZDM_HOME=/u01/app/oracle/zdmbase/zdm21 ----> ZDM software binaries export ZDM_INSTALL_LOC=/u01/zdm21-inst ----> ZDM installable - Create directories [zdmuser@zdmserver]$ mkdir -p $ORACLE_BASE $ZDM_BASE $ZDM_HOME $ZDM_INSTALL_LOC
Steps to Reproduce the Error
All I had to do is to run the install script with the required arguments (directories) to reproduce the behavior.
$ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \ ziploc=./zdm_home.zip -zdm
Failure to initialize MySQL
The error as you can see happening at MySQL configuration stage and lacks any relevant description
Setting up MySQL... --------------------------------------- Failed to initialize MySQL Failed to initialize MySQL One or more errors occurred while setting up No GI RHP. Trying to stop MySQL in case it was started and left up. spawn /u01/app/oracle/zdm21/mysql/server/bin/mysqladmin --defaults-file=/u01/app/oracle/crsdata/velzdm2prm/rhp/conf/my.cnf -u root -p shutdown WARNING: Failed to stop MySQL
Now, at this point, we can either, retry the installation forever, or start digging further.
Which Log to Check
I had no clue really, but I decided to just hit find under /u01 and try my luck with “mysql” as filter 😉 .
JACKPOT! here is the winner
You got it, always look under $ORACLE_BASE/crsdata/myserver/rhp/mysql/metadata for such log
The Actual Error
Clearly, the MySQL creation step failed because temporary files couldn’t be created in an ext FS as shown below.
$ more $ORACLE_BASE/zdmbase/crsdata/zdmserver/rhp/mysql/metadata/mysql-error.log … 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 1 [ERROR] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error number 22. 1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Out of disk space
What does it really mean?
In MySQL8, InnoDB files are created using a specific function (like dd) called posix_fallocate, which just writes 000s in a disk file to reserve space. See description below
I obviously shared this in the SR, but I wanted to see what MySQL community had to say about it, so I looked it up.
MySQL Bug or EXT Dislike
It didn’t take long before I found the answer in stackoverflow . As explained below it seems that posix_fallocate function isn’t supported by ext file systems.
In a nutshell, we have two bugs filed related to the same issue
2021 for MySQL8: Bug #102384 InnoDB initialization failed on EXT3 filesystem
Solution: Replace ext with XFS
From the above bugs, two solutions were available for our ZDM installation problem.
- MySQL8 bug received a patch as a fix from a contributor. Contribution: fix_init_fail_on_ext3.patch
- The older bug for MySQL 5.7 had no permanent fix but using XFS got rid of the posix_fallocate bug.
I decided to go for an XFS disk on /u01, mostly because it was the least intrusive option and particularly after reading Dimitri Kravtchuk piece about MySQL perf regression on ext4 (MySQL Performance : XFS -vs- EXT4 Story) where he recommended moving to XFS for kernels higher than 4.1.
[root@zdmserver~]# mount|grep u01 /dev/mapper/vg01-lvol1 on /u01old type ext4 (rw,relatime,seclabel) /dev/mapper/vg01-lvol2 on /u01 type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
ZDM was successfully installed after cleanup and reinstall.
-- Uninstall [zdmuser@zdmserver]$ $ZDM_HOME/bin/zdmservice stop deinstall -- Re-Install $ ./zdminstall.sh setup oraclehome=$ZDM_HOME oraclebase=$ZDM_BASE \ ziploc=./zdm_home.zip -zdm
This was an issue that had nothing to do with Oracle but allowed me to discover ZDM logs directories and choose the right file system for its MySQL DB. Hope this will help anyone who runs into the same error.
More from this Author
Introduction Since the very beginning, everyone was introduced to Cloud Services through the console as it’s very quick. But the cloud CLI tooling ... Read More