Oracle RAC (Real Application Clusters) and InfiniBox storage snapshots provide an excellent data protection option for large, mission critical databases. InfiniBox snapshots are efficient, immutable, and promotable. 

Oracle RAC databases can be image-copied using snapshots in InfiniBox snapshot groups, and according to a customizable schedule for the best data protection coverage for each environment. Restoring Oracle RAC databases from InfiniBox snapshot groups is fast, easy, and space efficient. Since version 12.2, Oracle RAC fully supports recovery using storage snapshots.

Oracle RAC overview

Oracle RAC is a high availability solution for Oracle databases that uses a scale-out architecture to support applications. Using multiple server nodes, which can be physical or virtualized, the Oracle RAC nodes field user connections, user work, as well as system work that can be spread out across many Oracle RAC nodes in the cluster. All server nodes see one virtual copy of the data. It is possible to have multiple copies of the data spread across multiple InfiniBox storage frames to provide even higher availability than a single InfiniBox frame can provide. This architecture is discussed in the InfiniBox Replication section of this document.

Oracle RAC on InfiniBox - architecture

Oracle Grid Infrastructure

Oracle RAC introduces a cluster management layer that Oracle calls Grid Infrastructure. This software provides the cluster management support for Oracle Databases. It also provides for the Oracle Storage infrastructure software that stores the Oracle Database Data. The storage infrastructure is a file system layer that allows for single node access, or multiple node concurrent access through the use of Oracle Automatic Storage Management, or ASM. 

For more details about installing, configuring, and managing Oracle Grid Infrastructure, refer to the Grid Infrastructure documentation in the Oracle Document Center.

Oracle ASM

Oracle Automatic Storage Management (ASM) is the layer that InfiniBox storage interacts with. Mapping several InfiniBox Fibre Channel volumes to a cluster of hosts allows for all hosts in the cluster to access the data simultaneously. Oracle ASM manages the concurrent access and protection of the data for the Oracle Database.

Oracle ASM disk groups are created to store datafiles, control files, redo logs, and concurrency devices such as the Oracle Cluster Registry and voting disk. These logical devices manage the health of the Oracle cluster.

All ASM objects are stored in folders within the Oracle ASM disk groups.

Oracle ACFS

Oracle Advanced Clustered File System is a more comprehensive file system that is mounted on a host that uses ASM as a base storage platform. It creates and manages clustered file systems that are POSIX compliant. Both internal Oracle database objects as well as any other Linux/Unix/AIX/Windows objects can be stored in ACFS, and accessed across all nodes in the Oracle RAC environment. ACFS is fully supported on InfiniBox storage. 

Oracle data layout

For database data, the best approach is to create many volumes of equal size to store Oracle datafiles. Depending on the anticipated total size of the Oracle RAC database, the size of each volume dictates how many volumes to create. For example, for a 100TB database, you can create 20 x 5TB volumes. As the database grows over time, adding a 5TB volume would be the easiest way to increase the space within the ASM disk group. As datafiles are initially created, Oracle ASM stripes the data across all physical devices within the disk group. 

Oracle ASM support for volume resize

If the existing size of the volumes does not meet current requirements, Oracle ASM and InfiniBox support online volume size increase. Volume size reduction is not supported. 

In the InfiniBox GUI, for each volume to be increased in size, right-click the volume and select Modify Volume. In the Size field, enter the new desired size that is larger than the previous size.

oracle@ibox3901> vol.resize vol=oratemp size=9TB -y
Volume "oratemp" resized
oracle@ibox3901> vol.query vol=oratemp
Name of an existing volume (multiple values, separated by commas)
oratemp   oratemp-3 oratemp-5 
oracle@ibox3901> vol.query vol=oratemp 
NAME     THIN      SIZE        USED      ALLOCATED    SNAPSHOTS WP  POOL  COMPRESSED DATA REDUCTION MAPPED REPLICATION ROLE 
oratemp   yes      9.00 TB     1.76 TB   126.99 GB    4.27 GB   no  gl... yes        13.52 : 1      yes    -        

Resize can also be performed from the InfiniBox API. To access the API documentation from the InfiniBox GUI, click the menu icon at the upper right corner of the window, and select API Documentation.

The API reference provides the commands needed to programmatically perform the resize.

Within Oracle Grid Infrastructure, use the ASMCMD:

# Resize disks in a disk group, if all disks are the same size. This tells Oracle ASM to scan all disks and resize accordingly, automatically
# First in Grid Infrastructure ASMCMD: 
ALTER DISKGROUP <diskgroup_name> RESIZE ALL;

# If there are disks of different sizes in the disk group, such as redo logs:

ALTER DISKGROUP <diskgroup_name> RESIZE DISK <asm_disk_name> SIZE <new_size_in_MB/GB/TB>;

Oracle redo logs in a separate disk group

Oracle redo logs provide the best performance when they are created in a separate disk group away from datafiles. Oracle ASM uses a 512 byte block size for redo log storage. Oracle datafile storage in ASM uses a 4k block size to store data. This maps to the size of a single redo entry, which is 512 bytes. On a database that is update intensive, redo entry writes are more efficient if the block size matches the redo entry size. In our labs, up to 30% improved write performance has been observed. 

Oracle archive logs can be placed inside ASM disk groups, or the archive_log_dest can be set to a visible file system. During a recovery operation, Oracle knows where archive logs are kept.

Oracle RAC and InfiniBox storage snapshots

Infinidat InfiniBox storage snapshots provide the best snapshot system in the storage industry. InfiniBox snapshots are fast, efficient, and immutable. The use of InfiniBox consistency groups complies with Oracle's storage snapshot requirements for full recovery support. These Oracle requirements are:

• The database is crash consistent when the snapshot is created.
• The storage snapshot system preserves write order fidelity for each datafile at the time the snapshot is created.
• The storage snapshot system stores the date and time each snapshot is created.

InfiniBox consistency groups

InfiniBox consistency groups (CGs) are used to group all Oracle data files into a single entity. Snapshots and replication are managed using those entities. A snapshot of a consistency group is called a 'snapshot group' or a 'snap group'. When a snapshot group is created, all elements in the CG have the exact same timestamp. This ensures that all elements are consistent with one another, no matter how many elements are in the CG.

Using the GUI to create a consistency group

1. In the InfiniBox management console GUI, click the Datasets icon on the left toolbar.
2. Open the Consistency Groups tab, and click Create Group.
   
   
   
3. In the Create a Consistency Group window, set the consistency group parameters.
   
   
   • Group Name - Enter a name for the consistency group.
   • Pool - Select a pool from the list of available pools. 
   • Snapshot policy - (Starting from v7.3.10) Select either the name of an existing snapshot policy, or None.
4. Click Create.

Using the CLI to create a consistency group

oracle@ibox3901> cg.create name=ora222-2 pool=glennf snapshot_policy=Oracle -y
Consistency group "ora222-2" created

Using the GUI to create a snapshot group

Create a snapshot group from the consistency group with the relevant options.

Snapshot names within the snap group are comprised of:

• the volume name
• a prefix
• a suffix

You can lock the snapshot for a specified duration.

1. In the InfiniBox management console GUI, click the Datasets icon on the left toolbar.
2. Open the Consistency Groups tab.
3. Right-click a consistency group, and select Create Snap Group from the pop-up menu.
4. In the Create Snap Group window, enter a name for the snap group, and enter a prefix and/or suffix.
5. (Optional) Click Advanced, and specify the duration to keep the snapshot locked.
   
   

   Locking a snapshot prevents any user from deleting it and releasing its capacity until the time specified.

   
   
6. Click Create Snap Group.

Oracle Database image snapshots

There are two options for collecting the Oracle Database image with snapshots. 

• Set Oracle to backup mode and then create a snapshot. Oracle remains operational, and continues fielding writes. Much more redo is generated, which can be used to assist in recovery. Oracle must open every data file to modify the header with backup mode details. When supporting very large databases (over 1PB), going into backup mode takes a long time. One Infinidat customer with a database larger than 1PB claims it takes hours to go into backup mode. 
• It is generally faster to create a crash-consistent image of the database with snapshots.  Oracle now supports full recovery of crash-consistent storage snapshots.

InfiniBox storage snapshot strategy

InfiniBox snapshots can be created as frequently as desired. The system supports tens of thousands of snapshots, and there is no performance penalty suffered. Some customers take snapshots every 15 minutes, to match their normal recovery strategy. With the use of custom snapshot policies on InfiniBox, the snapshots can be created automatically, and with appropriate options. For example, with locking and overwriting to rotate snapshot groups' content every 14 days. The following is a screen shot of the Snapshot Scheduler built into InfiniBox. There is also a more comprehensive software package from support called SnapRotator that automates snapshots and recovery of bare metal and VMs.

Snapshot storage consumption is related to the data change rate of the database. Generally, an OLTP database trends similar to the data change rate. The efficiency of InfiniBox data compression determines the actual allocated space used within the snapshot. The purple column circled shows the data reduction rate. In this case, as seen from the consistency group member listing, this is a 13:1 compression ratio, as this is a text heavy database. The green circled column shows  the resulting snapshot consumption of the consistency group over time. 

InfiniBox 'Promote This Snap Group'

InfiniBox storage snapshots are fully promotable to volumes. This is a reliable, quick way to set up non-production environments with snapshots of even the largest databases.

• There is no data movement during the process, so it is fast and it is extremely space efficient.
• The only space that is used is the data that has been modified in the original production snapshot since it was created. Over time, it is possible that the newly promoted snapshot group might come close to equaling the size of the original volumes that they came from, but usually that is not the case. Most applications on the database won't modify ALL the data, even over the life of the database.
• The promoted volume names and the CG name can now be changed to appropriate names for use elsewhere.
• They can be treated just like regular volumes.
• They provide the same high performance as the original volumes.
• The logical block addresses simply point back to the original volumes LBA's, and they do not take up more space in RAM and SSD cache. 

To promote a snapshot group:

1. Right-click the snapshot group, and select  Promote This Snap Group.
   
   
   
2. The snapshot group shows up in the pool listing as a separate consistency group with the name of the original snapshot group. The volume names match the original snapshot group name.
   

oracle@ibox3901> cg.snap.promote sg=ora222Prod2025.10.08_02-56_UTC -y
Snapshot group "ora222Prod2025.10.08_02-56_UTC" promoted

Oracle RAC recovery using InfiniBox snapshot groups

To recover the RAC cluster using InfiniBox snapshots:

• Ensure that all databases on all nodes aree down. All nodes in the cluster can be up.
• Ensure that ASM disk groups that are part of the snapshot group are unmounted.
• From the GUI, right-click the snapshot group to restore from, and select Restore from This Snap Group.

• In the CLI, run:

racle@infinibox3901> cg.restore source=ora222-second cg=ora222 -y

ASM mount diskgroups. ASM performa a pseudo fsck to correct the device headers with the proper timestamp. ASM disk groups are then be brought online.

There are two recovery options.

• One is to use RMAN.

rman target / <<EOF >rman-restoredb.txt

{
SET UNTIL TIME "TO_DATE('2025-10-04 04:00:00', 'YYYY-MM-DD HH24:MI:SS')";
RESTORE ARCHIVELOG ALL;
RECOVER DATABASE;
sql 'ALTER DATABASE OPEN RESETLOGS';
}`
EOF

This new addition to the RMAN recover command 'snapshot time' tells RMAN to attempt to recover the database assuming that a storage snapshot was created at 22:30:00. The database attempts to play forward the database using archive logs to the "until" time of 23:00:00.

• The second option is within sqlplus. A recoverable, crash-consistent image of the database is brought up with the mount command. The ;recover database until cancel; allows the dba to manually apply archive logs until satisfied with the recovery.

sqlplus / as sysdba

startup database mount

recover database until cancel
/arcdest/arc1
/arcdest/arc2
...

media recovery complete
alter database open resetlogs;

Oracle RAC backup and recovery using Oracle RMAN to InfiniBox NFS

Oracle RMAN backup of Oracle RAC

For Oracle RMAN backup across Oracle RAC, the default approach is that when RMAN is executed, all channels are spread out across all nodes in the cluster. The behavior is controlled by the listener configuration. Another approach is to run RMAN on a small subset of nodes, away from user activity, to limit the CPU/memory resources on those nodes. This behavior is also controlled by the listener configuration. 

For both approaches, the best performance for RMAN is to use multiple channels. Each Oracle channel reads and writes a limited data throughput. Even on high-speed flash systems, if more channels are used, each channel reads a subset of the Oracle database data and allows for parallel read and write operations.  

In the example below, a total of 8 channels are configured. Each channel backs up to two NFS mount points. In this case, the two mount points are to the same NFS share on InfiniBox using two different IP addresses and ports. This maximizes read and write throughput, and reduces Oracle RMAN total backup execution. 

rman target / <<EOF

run
{
configure controlfile autobackup on;
configure controlfile autobackup format for device type disk to '/u01/app/oracle/software/ora222/ora222-%F';
ALLOCATE CHANNEL CH1 DEVICE TYPE DISK FORMAT '/dbfs/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH2 DEVICE TYPE DISK FORMAT '/dbfs2/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH3 DEVICE TYPE DISK FORMAT '/dbfs/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH4 DEVICE TYPE DISK FORMAT '/dbfs2/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH5 DEVICE TYPE DISK FORMAT '/dbfs/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH6 DEVICE TYPE DISK FORMAT '/dbfs2/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH7 DEVICE TYPE DISK FORMAT '/dbfs/222/ora222/ora222-%U';
ALLOCATE CHANNEL CH8 DEVICE TYPE DISK FORMAT '/dbfs2/222/ora222/ora222-%U';

backup filesperset=1 section size 10G database;

CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET;

release channel CH1 ;
release channel CH2 ;
release channel CH3 ;
release channel CH4 ;
release channel CH5 ;
release channel CH6 ;
release channel CH7 ;
release channel CH8 ;
}
EOF

Oracle RMAN backup target -InfiniBox NFS/SMB filesystems

• Create an NFS / SMB share on InfiniBox and create an export/share (called dbfs).
• Mount the share on a server (Linux). In this example, this server has 2 x 25Gb ports for backup. Identify the ports, and create host routes using the ports to the appropriate IP addresses in the NAS network space on the target InfiniBox.

# create the mount points for the NFS shares
mkdir -p /dbfs/ora222 /dbfs2

chown -R oracle:dba /dbfs/ora222

# identify host ports
ifconfig -a
ens5f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
    inet 172.31.39.133 netmask 255.255.224.0 broadcast 172.31.63.255


ens5f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
    inet 172.31.59.165 netmask 255.255.224.0 broadcast 172.31.63.255

# create host routes 
route add -host 172.31.62.251 dev ens5f0 
route add -host 172.31.62.252 dev ens5f1 

# Check routes
[root@io-wt-222 rep-nfs]# netstat -rn
Kernel IP routing table
Destination   Gateway     Genmask     Flags  MSS Window irtt Iface
0.0.0.0     172.31.95.254  0.0.0.0     UG    0 0     0 eno3
172.31.32.0   0.0.0.0     255.255.224.0  U     0 0     0 ens5f1
172.31.32.0   0.0.0.0     255.255.224.0  U     0 0     0 ens5f0
172.31.62.251  0.0.0.0     255.255.255.255 UH    0 0     0 ens5f1
172.31.62.252  0.0.0.0     255.255.255.255 UH    0 0     0 ens5f0
172.31.64.0   0.0.0.0     255.255.224.0  U     0 0     0 eno3

# mount the share 
mount -t nfs -o async,nconnect=8,rw,tcp,vers=3 172.31.62.251:/dbfs /dbfs
mount -t nfs -o async,nconnect=8,rw,tcp,vers=3 172.31.62.252:/dbfs /dbfs2

This RMAN backup script then uses the two interfaces to fully use both 25Gb ethernet ports to back up the database.

# Query showing backup results
INPUT_TYPE  STATUS  START_TIME   END_TIME      HRS SUM_BYTES_BACKED_IN_GB SUM_BYTES_PER_SEC_MB
------------- --------- -------------- -------------- ------- ---------------------- --------------------
DB FULL    COMPLETED 10/01/25 04:51 10/01/25 05:11   .33       5703.23242      4862.70608

# This backup of 5.7TB Oracle database took 21 minutes at 4,862 MB/s across 2 x 25Gb ports

Oracle RMAN recovery of Oracle RAC

• Ensure that all databases on all nodes are down.
• Ensure that Oracle ASM disk groups are offline.
• Run an 'RMAN restore until time'. 
• Recover the database.
• Open the database.

rman target / <<EOF >rman-restoredb.txt
 
{
SET UNTIL TIME "TO_DATE('2025-10-04 04:00:00', 'YYYY-MM-DD HH24:MI:SS')";
RESTORE ARCHIVELOG ALL;
RECOVER DATABASE;
sql 'ALTER DATABASE OPEN RESETLOGS';
}`

EOF

Oracle RAC and InfiniBox Storage Replication 

Use InfiniBox Active-Active (A-A) replication to protect Oracle RAC even further:

• Allow volumes on one InfiniBox to A-A replicate to a second InfiniBox.
• With the use of Linux/Unix ALUA (Asymmetric Logical Unit Access) and the host state (optimized/non-optimized), the volumes can seen by a single host from both InfiniBox systems.

In this example, host1 with a set of volumes on Site A InfiniBox is Active-Active replicating to Site B InfiniBox. Site B volumes are mapped back to host1. host1 and host2 are configured as non-optimized. The volumes that are replicated indicate that host1 should treat the Site A volumes as primary, and Site B volumes as secondary.

On host1, the multipath configuration for a volume would look like this:

# multipath -ll
mpathb (36742b0f000004e710000000000000b09) dm-0 NFINIDAT,InfiniBox    
size=931G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:0:1 sdb 8:16  active ready running
| |- 1:0:2:1 sdf 8:80  active ready running
| |- 8:0:1:1 sdk 8:160 active ready running
| |- 8:0:5:1 sdl 8:176 active ready running
| |- 1:0:4:1 sdi 8:128 active ready running
| `- 8:0:0:1 sdn 8:208 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
 |- 1:0:1:1 sde 8:64  active ready running
 |- 1:0:3:1 sdj 8:144 active ready running
 |- 1:0:5:1 sdo 8:224 active ready running
 |- 8:0:2:1 sdd 8:48  active ready running
 |- 8:0:3:1 sdy 65:128 active ready running
 `- 8:0:4:1 sdaa 65:160 active ready running

The multipath in a non-optimized configuration would show a primary/secondary configuration showing the top paths as primary, and the bottom paths as secondary. Note the line in front of the first 6 paths. This denotes that the top 6 are the primary paths. If a failure of all primary paths are detected, the secondary paths will immediately take over (ALUA). The application does not see any errors. The operating system shows some path timeout errors. 

If both host1 and host2 are configured as optimized, then when host1 writes data, it will write to all paths. The multipath command does not show a primary / secondary tiered path listing as above example shows. Instead, it shows an inline list of 12 paths available for writing with the same priority. 

# multipath -ll
mpathb (36742b0f000004e710000000000000b09) dm-0 NFINIDAT,InfiniBox    
size=931G features='0' hwhandler='0' wp=rw
-+- policy='round-robin 0' prio=50 status=active
|- 1:0:0:1 sdb 8:16  active ready running
|- 1:0:2:1 sdf 8:80  active ready running
|- 8:0:1:1 sdk 8:160 active ready running
|- 8:0:5:1 sdl 8:176 active ready running
|- 1:0:4:1 sdi 8:128 active ready running
|- 8:0:0:1 sdn 8:208 active ready running
`-+- policy='round-robin 0' prio=50 status=enabled
|- 1:0:1:1 sde 8:64  active ready running
|- 1:0:3:1 sdj 8:144 active ready running
|- 1:0:5:1 sdo 8:224 active ready running
|- 8:0:2:1 sdd 8:48  active ready running
|- 8:0:3:1 sdy 65:128 active ready running
|- 8:0:4:1 sdaa 65:160 active ready running

Generally, most configurations are setup as non-optimized, and in a failover configuration.

Summary

Oracle RAC and InfiniBox storage snapshots provide an excellent data protection option for large, mission critical databases. InfiniBox snapshots are efficient, immutable, and promotable. 

Oracle RAC databases can be image-copied using snapshots in InfiniBox snapshot groups, and according to a customizable schedule for the best data protection coverage for each environment. Restoring Oracle RAC databases from InfiniBox snapshot groups is fast, easy, and space efficient. Since version 12.2, Oracle RAC fully supports recovery using storage snapshots.