Blogs

Exadata Cloud at Customer – Free space in ASM and adding a new Database

Some time ago I wrote a short blog about dependencies between the number of enabled CPUs and how many databases you could build. Today we got another error when we were trying to create a new database. Here is the screenshot of the error.

If you can’t read it on a small screen it says “Create Database operation failed due to an unknown error. Refer to work request ID 2580d3ff-064e-4e6f-ab06-1327fd02f40e when opening a Service Request at My Oracle Support.” and provide an error code which is “Error“

I don’t want to reiterate how helpful such a message is and how valuable such information is in the troubleshooting process. But having some experience with the ExaCC I went straight to the last created log in the “/var/opt/oracle/log//dbaasapi/db/createdb” directory. In the log the reason was stated absolutely clear:

_TESTDB.cfg matching passwdINFO: freespace in RECOC1 diskgroup is : 11473 GBINFO: freespace in DATA diskgroup is : 0 GB[WARNING] [DBAAS-11000] - The following location does not have enough space: DATA.   CAUSE: The minimum required free space is 50 GB.    ACTION: Verify the disk space usage and free an additional space of 0 GB and try again.

So it says “The following location does not have enough space: DATA. CAUSE: The minimum required free space is 50 GB.”. Let’s have a look at the Diskgroup DATA and find out how much free space I have.

SQL> SELECT name,total_mb/1024/3 total_gb, free_mb/1024/3 free_gb, usable_file_mb FROM v$asm_diskgroup;
 
NAME				 TOTAL_GB    FREE_GB USABLE_FILE_MB
------------------------------ ---------- ---------- --------------
DATAC1				    91956 2217.47266	   -2960582
RECOC1				    30648 14953.1133	   13568457
SPRC1				   306480 296081.004	  285751641
 
SQL>

It appears I have enough space available but my usable space is negative. Is it possible that the ExaCC tool checks usable space instead of free space? I checked the tools code and in the Perl module rac.pm found that indeed Oracle had conditions for the usable space in the Diskgroup. 

 0;\n" if ($get_raw_free =~ m/no/i);
print FS "col name format a30 \nselect name,free_mb from v\$asm_diskgroup where usable_file_mb > 0;\n" if ($get_raw_free !~ m/no/i);
...
redacted
...

It seems it returns the correct value for free space only if the usable_file_mb value is positive.

A short summary. When you monitor free space in the Diskgroup on your ExaCC please include the usable_file_mb to the monitoring. If you exhaust the usable free space (which is different from the mirrored free space) it will not affect any of your existing databases but prevent you from creating a new one.

More from this Author