MySQL: Updating the Configuration of a MySQL DB System on OCI
Overview: A MySQL configuration is a collection of variables that define the operation of a MySQL DB system. It is analogous to ... Read More
Adding a New Computer Node to Bare Metal Exadata
You can add a new database server to an existing Oracle Real Application Clusters (Oracle RAC) cluster running on Oracle Exadata Database Machine.
Assumptions
- The new compute node has been reimaged with the same Exadata software version running on existing compute nodes.
- A new compute node network has been configured (management, client and private).
- The new compute node hardware is X7-2.
- Exadata Image version is 20.1.9.0.0 on all compute nodes.
- Grid Infrastructure (GI) version is 19.11.0.0.210420
- Database version is 12.2.0.1.210420
- The elastic Bare Metal Exadata system consists of 4 compute nodes and 3 cell nodes.
- A single software owner (Oracle account) is in use for database and grid software.
Preparations
1- Reclaim Space on the new compute node.
This is a Bare Metal Exadata system, so it is recommended to reclaim inactive system logical volume used for DOM0 using script “/opt/oracle.SupportTools/reclaimdisks.sh“
[root@exadb05 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check Model is ORACLE SERVER X7-2 Number of LSI controllers: 1 Physical disks found: 4 (252:0 252:1 252:2 252:3) Logical drives found: 1 Linux logical drive: 0 RAID Level for the Linux logical drive: 5 Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3) Dedicated Hot Spares for the Linux logical drive: 0 Global Hot Spares: 0 Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks. Valid. Booted: Linux. Layout: Linux + DOM0. [root@exadb05 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim Model is ORACLE SERVER X7-2 Number of LSI controllers: 1 Physical disks found: 4 (252:0 252:1 252:2 252:3) Logical drives found: 1 Linux logical drive: 0 RAID Level for the Linux logical drive: 5 Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3) Dedicated Hot Spares for the Linux logical drive: 0 Global Hot Spares: 0 [INFO ] Check for Linux with inactive DOM0 system disk [INFO ] Valid Linux with inactive DOM0 system disk is detected [INFO ] Number of partitions on the system device /dev/sda: 3 [INFO ] Higher partition number on the system device /dev/sda: 3 [INFO ] Last sector on the system device /dev/sda: 3509760000 [INFO ] End sector of the last partition on the system device /dev/sda: 3509759966 [INFO ] Remove inactive system logical volume /dev/VGExaDb/LVDbSys3 [INFO ] Remove xen files from /boot [INFO ] Remove ocfs2 logical volume /dev/VGExaDb/LVDbExaVMImages
2- Copy or merge the contents of the following files using files on one of the existing compute nodes.
You may configure ssh Passwordless Authentication from the first node to the new compute node (exadb05).
2.1. Copy the contents of the /etc/security/limits.conf file from exadb01 to exadb05
2.2. Edit /etc/hosts on all nodes and add exadb05 private network two IPs (IB network IPs). Make sure that /etc/hosts file on all compute nodes contains private IPs for all compute nodes.
--- exadb05 private IPs 192.168.10.29 192.168.10.30
2.3. Copy the /etc/oracle/cell/network-config/cellinit.ora file from exadb01 to exadb05 .
2.4. Update the /etc/oracle/cell/network-config/cellinit.ora file on new compute node (exadb05) with the IPs of ib0 and ib1 interfaces.
[root@exadb05 ~]# vi /etc/oracle/cell/network-config/cellinit.ora ipaddress1=192.168.10.29/22 ipaddress2=192.168.10.30/22
2.5. Copy the /etc/oracle/cell/network-config/cellip.ora file from exadb01 to exadb05. The content of the cellip.ora file should be the same on all database servers. It contains the list of cell nodes private IPs.
[root@exadb05 ~]# cat /etc/oracle/cell/network-config/cellip.ora cell="192.168.10.15;192.168.10.16" cell="192.168.10.17;192.168.10.18" cell="192.168.10.19;192.168.10.20"
2.6. Copy the /etc/modprobe.d/exadata.conf file from exadb01 to exadb05. The contents of the configuration file should be the same on all database servers.
2.7. Copy the /etc/sysctl.conf file from exadb01 to exadb05. The contents of the file should be the same on all database servers.
Reload sysctl.conf variables on exadb05.
[root@exadb05 ~]# sysctl -p
2.8. Copy the /etc/oracle/cell/network-config/cellroute.ora file from exadb01 to exadb05.
The file defines IPs mapping between compute node’ private IPs and each cell node’ private IPs.
Modify the contents pf the file on the new node (exadb05) to use private IPs (the local InfiniBand interfaces) on exadb05
[root@exadb05 ~]# vi /etc/oracle/cell/network-config/cellroute.ora # Routes for 192.168.10.15;192.168.10.16 route="192.168.10.16;192.168.10.29" route="192.168.10.15;192.168.10.30" # Routes for 192.168.10.17;192.168.10.18 route="192.168.10.18;192.168.10.29" route="192.168.10.17;192.168.10.30" # Routes for 192.168.10.19;192.168.10.20 route="192.168.10.20;192.168.10.29" route="192.168.10.19;192.168.10.30"
3- Set up the users for the software owners on the new node (exadb05).
If you are using role-separated management, then the users are usually oracle and grid. If you use a single software owner, then the user is usually an Oracle. We are using a single owner in our example.
3.1. Obtain the current group information for the Oracle account from a exadb01.
[root@exadb01 ~]# id oracle uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba)
3.2. Add groups on the new node (exadb05) as listed on the first node.
[root@exadb05 ~]# groupadd -g 1001 oinstall [root@exadb05 ~]# groupadd -g 1002 dba
3.3. Add an Oracle account on the new node (exadb05) as listed on the first node.
[root@exadb05 ~]# useradd -u 1001 -g 1001 -G 1001,1002 -m -d /home/oracle -s \ /bin/bash oracle
3.4. Create the Oracle Base and Grid home directories on the new node (exadb05).
[root@exadb05 ~]# mkdir -p /u01/app/oracle [root@exadb05 ~]# mkdir -p /u01/app/19.0.0.0/grid [root@exadb05 ~]# chown -R oracle:oinstall /u01/app
3.5. Change the ownership on the cellip.ora and cellinit.ora files on the new node (exadb05).
[root@exadb05 ~]# chown -R oracle:oinstall /etc/oracle/cell/network-config
3.6. Set the password for oracle on the new node (exadb05).
4- Set up SSH for the oracle account
4.1. Log in to the oracle account on the new node (exadb05).
4.2. Create the DCLI group file on the new node (exadb05). The file contains the name of all compute nodes.
[root@exadb05 ~]# su - oracle [oracle@exadb05 ~]# vi /home/oracle/dbs_group exadb01 exadb02 exadb03 exadb04 exadb05
4.3. Create ssh key and copy it to all nodes. Run the following command on the new node (exadb05).
[oracle@exadb05 ~]# ssh-keygen -t rsa [oracle@exadb05 ~]# dcli -g /home/oracle/dbs_group -l oracle -k -s '-o StrictHostKeyChecking=no' oracle@exadb01's password: oracle@exadb02's password: oracle@exadb03's password: oracle@exadb04's password: oracle@exadb05's password: exadb01: ssh key added exadb02: ssh key added exadb03: ssh key added exadb04: ssh key added exadb05: ssh key added
4.4. Verify oracle SSH equivalency.
[oracle@exadb05 ~]$ dcli -g dbs_group -l oracle date exadb01: Fri Aug 27 10:02:43 EDT 2021 exadb02: Fri Aug 27 10:02:43 EDT 2021 exadb03: Fri Aug 27 10:02:43 EDT 2021 exadb04: Fri Aug 27 10:02:43 EDT 2021 exadb05: Fri Aug 27 10:02:43 EDT 2021
Clone Oracle Grid Infrastructure to the New Compute Node
Connect to the first compute node (exadb01) as oracle and run all steps below.
1- Verify the hardware and operating system installation using the cluster verification utility (cluvfy).
The phrase Post-check for hardware and operating system setup was successful should appear at the end of the report. If the cluster verification utility fails to validate the storage on the replacement server, you can ignore those messages.
--- Verify command cluvfy stage -post hwos -n <new_server>,<existing_node> -verbose [oracle@exadb01 ~]$. oraenv [ORACLE_SID]: +ASM1 [oracle@exadb01 ~]$ cd $ORACLE_HOME/bin [oracle@exadb01 ~]$ ./cluvfy stage -post hwos -n exadb05,exadb01 -verbose . . . Post-check for hardware and operating system setup was successful. CVU operation performed: stage -post hwos Date: Aug 27, 2021 10:06:59 AM CVU home: /u01/app/19.0.0.0/grid/ User: oracle
2- Verify peer compatibility.
The phrase Verification of peer compatibility was successful should appear at the end of the report.
If the only failed components are related to the physical memory, swap space and disk space, then it is safe to continue.
--- Verify command cluvfy comp peer -refnode <existing_node> -n <new_server> -orainv oinstall -osdba dba [oracle@exadb01 ~]$. oraenv [ORACLE_SID]: +ASM1 [oracle@exadb01 ~]$ cd $ORACLE_HOME/bin [oracle@exadb01 ~]$ ./cluvfy comp peer -refnode exadb01 -n exadb05 -orainv oinstall -osdba dba . . . Verifying Peer Compatibility ...PASSED Verification of peer compatibility was successful. CVU operation performed: peer compatibility Date: Aug 27, 2021 10:13:11 AM CVU home: /u01/app/19.0.0.0/grid/ User: oracle
3- Verify GI home is ready to be cloned to the new node.
--- Verify command cluvfy stage -pre crsinst -n <new_server> -verbose [oracle@exadb01 ~]$. oraenv [ORACLE_SID]: +ASM1 [oracle@exadb01 ~]$ cd $ORACLE_HOME/bin [oracle@exadb01 ~]$ ./cluvfy stage -pre crsinst -n exadb05 -verbose . . . Failures were encountered during execution of CVU verification request "stage -pre crsinst". Verifying Group Existence: asmadmin ...FAILED exadb05: PRVG-10461 : Group "asmadmin" selected for privileges "OSASM" does not exist on node "exadb05". Verifying Group Existence: asmdba ...FAILED exadb05: PRVG-10461 : Group "asmdba" selected for privileges "OSDBA" does not exist on node "exadb05". Verifying Group Membership: asmadmin ...FAILED exadb05: PRVG-10460 : User "oracle" does not belong to group "asmadmin" selected for privileges "OSASM" on node "exadb05". Verifying Group Membership: asmdba ...FAILED exadb05: PRVG-10460 : User "oracle" does not belong to group "asmdba" selected for privileges "OSDBA" on node "exadb05". Verifying RPM Package Manager database ...INFORMATION PRVG-11250 : The check "RPM Package Manager database" was not performed because it needs 'root' user privileges. CVU operation performed: stage -pre crsinst Date: Aug 27, 2021 10:15:07 AM CVU home: /u01/app/19.0.0.0/grid/ User: oracle >>> FAILED check - Verifying Group Existence (OSASM, OSDBA, asmadmin, asmdba) is ignorable
4- Verify node addition pre-requisites.
The phrase Pre-check for node addition was successful should appear at the end of the report.
--- Verify command cluvfy stage -pre nodeadd -n <new_server> -verbose [oracle@exadb01 ~]$. oraenv [ORACLE_SID]: +ASM1 [oracle@exadb01 ~]$ cd $ORACLE_HOME/bin [oracle@exadb01 ~]$ ./cluvfy stage -pre nodeadd -n exadb05 -verbose . . . Pre-check for node addition was successful. CVU operation performed: stage -pre nodeadd Date: Aug 27, 2021 11:00:11 AM CVU home: /u01/app/19.0.0.0/grid/ User: oracle
5- Add a new compute node to the cluster.
--- Add command
addnode.sh -silent "CLUSTER_NEW_NODES={<new_server>}" \
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={<new_server>-vip}" \
"CLUSTER_NEW_NODE_ROLES={hub}"
[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/addnode
[oracle@exadb01 ~]$ ./addnode.sh -silent -ignoreSysPrereqs -ignorePrereq "CLUSTER_NEW_NODES={exadb05}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={exadb05-vip}" "CLUSTER_NEW_NODE_ROLES={hub}"
Copy Files to Remote Nodes in progress.
.................................................. 6% Done.
.................................................. 11% Done.
....................
Copy Files to Remote Nodes successful.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 21% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2021-08-27_11-14-47AM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 49% Done.
Saving cluster inventory in progress.
.................................................. 83% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/19.0.0.0/grid was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2021-08-27_11-14-47AM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 90% Done.
Update Inventory in progress.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2021-08-27_11-14-47AM.log
Update Inventory successful.
.................................................. 97% Done.
s a root user, execute the following script(s):
1. /u01/app/oraInventory/orainstRoot.sh
2. /u01/app/19.0.0.0/grid/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[exadb05]
Execute /u01/app/19.0.0.0/grid/root.sh on the following nodes:
[exadb05]
The scripts can be executed in parallel on all the nodes.
Successfully Setup Software.
.................................................. 100% Done.
6- Connect to exadb05 as root and execute below two root scripts.
[root@exadb05 ~]$ /u01/app/oraInventory/orainstRoot.sh [root@exadb05 ~]$ /u01/app/19.0.0.0/grid/root.sh
7- Check the cluster, once the /u01/app/19.0.0.0/grid/root.sh script successfully completes.
[oracle@exadb05 ~]$ . oraenv [ORACLE_SID]: +ASM1 [oracle@exadb05 ~]$ $ORACLE_HOME/bin/crsctl check cluster -all ************************************************************** exadb01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadb02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadb03: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadb04: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadb05: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [oracle@exadb05 ~]$ crsctl status res -t >>> Confirm all cluster resources are ONLINE on new compute node as they are on existing compute nodes.
8- Update the GI Inventory.
Add the new node (exadb05) in the GI inventory on all the other nodes.
--- update command
runInstaller -updatenodelist -ignoreSysPrereqs ORACLE_HOME=<GI_HOME> "CLUSTER_NODES={<list_of_nodes_names>}" LOCAL_NODE=<node_name> CRS=TRUE
[oracle@exadb01 ~]$. oraenv
[ORACLE_SID]: +ASM1
[oracle@exadb01 ~]$ cd $ORACLE_HOME/oui/bin
./runInstaller -updatenodelist -ignoreSysPrereqs ORACLE_HOME=/u01/app/19.0.0.0/grid "CLUSTER_NODES={exadb01,exadb02,exadb03,exadb04,exadb05}" LOCAL_NODE=exadb01 CRS=TRUE
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 24318 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
>>> Note that this has to be executed on every node of your cluster; remember that you need to adapt the LOCAL_NODE parameter on each node.
Clone Oracle Database Homes to the New Compute Node
Connect to the first compute node (exadb01) as oracle and run all steps below.
Oracle database home: /u01/app/oracle/product/12.2.0.1/dbhome_1
1- Add the Oracle Database ORACLE_HOME to the new node (exadb05).
[oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/addnode
[oracle@exadb01 ~]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={exadb05}"
.
.
.
WARNING: The following configuration scripts need to be executed as the "root"
user in each cluster node.
/u01/app/oracle/product/12.2.0.1/dbhome_1/root.sh #On nodes exadb05
To execute the configuration scripts:
Open a terminal window.
Log in as root.
Run the scripts on each cluster node.
The Cluster Node Addition of /u01/app/oracle/product/12.2.0.1/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
2- Connect to exadb05 as root and execute the below root script.
[root@exadb05 ~]$ /u01/app/oracle/product/12.2.0.1/dbhome_1/root.sh Check the /u01/app/orcale/product/12.2.0.1/dbhome_1/install/root_exadb05_<date>.log file for the output of the script.
3- Run the Oracle Database Configuration Assistant (DBCA) in interactive mode to add the database instance to the new compute node (exadb05).
[oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/bin [oracle@exadb01 ~]$ ./dbca - On the Database Operation screen, select Instance Management. Click Next. - On the Instance Operation screen, select Add an instance. Click Next. - On the Database List screen, select the cluster database to which you want to add an instance. - The List Instance screen displays the current instances. Click Next to add a new instance. - The Add Instance screen displays the default name and the newly added node to the cluster. Accept the defaults and click Next. - On the Summary screen, verify the plan and click Finish. - On the Progress screen, watch for 100% completion. - On the Finish screen, acknowledge the confirmation that the new instance was successfully added.
4- Verify that the instance has been added.
[oracle@exadb01 ~]$ . oraenv [ORACLE_SID]: PRODDB1 [oracle@exadb01 ~]$ srvctl status database -d PRODDB [oracle@exadb01 ~]$ srvctl config database -d PRODDB >>> Confirm instance PRODDB5 is running on exadb05
5- Verify the administrative privileges on the new node (exadb05).
[oracle@exadb01 ~]$ . oraenv [ORACLE_SID]: PRODDB1 [oracle@exadb01 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/bin [oracle@exadb01 ~]$ ./cluvfy comp admprv -o db_config -d /u01/app/oracle/product/12.2.0.1/dbhome_1 -n exadb05
6- Ensure the instance parameters are set for the new node (exadb05).
- Init parameters local_listener cluster_interconnects [oracle@exadb05 ~]$ . oraenv [ORACLE_SID]: PRODDB5 [oracle@exadb05 ~]$ sqlplus / as sysdba SQL> alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=<NEW_NODE_VIP>)(PORT=1521))' scope=spfile sid='PRODDB5'; SQL> alter system set cluster_interconnects = '<NEW_NODE_PRIV-IP1>,<NEW_NODE_PRIV-IP2>' scope=spfile sid='PRODDB5'; --- Restart PRODDB5 instance to get changes applied. [oracle@exadb05 ~]$ srvctl stop instance -d PRODDB -i PRODDB5 [oracle@exadb05 ~]$ srvctl start instance -d PRODDB -i PRODDB5 --- Confirm parameters new values [oracle@exadb05 ~]$ sqlplus / as sysdba SQL> show parameter cluster_interconnects SQL> show parameter local_listener
More from this Author
Oracle 23c: INTERVAL Data Type Aggregations
Overview: Oracle 23c introduces the use of SUM and AVG functions with INTERVAL datatype This enhancement makes it easier to calculate totals and ... Read More
