• Introduction
• InfiniBox 3-way replication example
• Querying consistency group information
• Verifying consistent volume mapping
• Adding a member to the replicated consistency group
  • Cleaning up the staging area snapshots
    • Identifying the snapshots to be deleted from the source InfiniBox
    • Deleting the snapshots on the source InfiniBox
    • Identifying the snapshots to be deleted from the target InfiniBox
    • Deleting the snapshots on the target InfiniBox
• Resizing a volume in a replicated consistency group
• Removing a volume from a replicated consistency group
  • Deleting a volume (optional)

Introduction

All InfiniBox management consoles support the following Async replica operations for concurrent replicas:

• Sync now
• Suspend replica
• Resume replica
• Update RPO and interval

The processes for adding, removing, and resizing consistency group members in a concurrent replica require several steps. These processes are explained in this document.

InfiniBox 3-way replication example

In the examples used in this document, a consistency group containing a single volume is replicated to and from ibox-a and ibox-b in Active-Active mode.

The same consistency group is replicated from ibox-a to ibox-c in Async mode.

On ibox-a and ibox-b:

• CG_1 is the name of the consistency group.
• vol_1 is the name of its member.

On ibox-c:

• CG_1-target is the name of the consistency group.
• vol_1-target is the name of its member.

Querying consistency group information

The following are query results for these InfiniBoxes.

# Query ibox-a
cg.member_query cg=CG_1 --columns=name,cg
NAME           CG
vol_1          CG_1

# Query ibox-b
cg.member_query cg=CG_1 --columns=name,cg
NAME           CG
vol_1          CG_1

# Query ibox-c
cg.member_query cg=CG_1-target --columns=name,cg
NAME           CG
vol_1-target   CG_1-target

The query results show that:

• On ibox-a, consistency group CG_1 contains the volume vol_1.
• On ibox-b, consistency group CG_1 contains the volume vol_1.
• On ibox-c, consistency group CG_1-target contains the volume vol_1-target.

Verifying consistent volume mapping

Before adding or removing a replicated consistency group member, query each of the existing volumes in the consistency group on both InfiniBoxes where they are replicated in Active-Active mode. Verify that each volume is mapped the same way on both InfiniBoxes.

• When a new member is added to the same replicated consistency group, set its mapping to the same values.
• When a member is removed from the replicated consistency group, use these values in the vol.unmap command.
  
  

# Verify that the volume is mapped the same way on both ibox-a and ibox-b
# Query mapping on ibox-a 
vol.map_query name=vol_1
NAME     MAPPING TYPE   MAPPED TO   LUN/NSID
vol_1    CLUSTER        CLU-MI-a    11
vol_1    CLUSTER        CLU-MI-b    11

# Query mapping on ibox-b  
vol.map_query name=vol_1
NAME     MAPPING TYPE   MAPPED TO   LUN/NSID
vol_1    CLUSTER        CLU-MI-a    11
vol_1    CLUSTER        CLU-MI-b    11

Adding a member to the replicated consistency group

In this example, volume vol_2 is created in consistency group CG_1 on ibox-a.

• The consistency group will continue to be replicated in Active-Active mode with ibox-b. The volume name and the consistency group name are the same on ibox-a and ibox-b.
• The consistency group will continue to be replicated in Async mode to ibox-c,  where the volume name will be vol_2-target in consistency group CG_1-target.

To add a member to the replicated consistency group:

1. It is recommended to record the mapping of existing volumes in the consistency group. See "Querying volume mapping" in this document.

2. Create the new volume on the source InfiniBox.

   # On ibox-a, create vol_2 
   vol.create name=vol_2 size=5.00GiB pool=demo
   Volume "vol_2" created

3. Replicate the new member in Async mode to the target InfiniBox that the consistency group is replicated to.

   # On ibox-a, create an Async replica of vol_2 to ibox-c
   replica.create source=vol_2 replication_type=ASYNC system=ibox-c remote_pool=demo dataset_type=VOLUME new_target_name=vol_2 -target rpo=300 interval=60
   Replica for volume "vol_2" created

   Wait for the replication to finish. If the volume is empty, the replication finishes quickly.

4. Query both InfiniBoxes to verify that there is now a consistency group Async replica and a new single volume Async replica.

   # Query ibox-a
   replica.query replication_type=ASYNC
   ASYNC REPLICAS
   
   LOCAL DATASET     TYPE      ROLE    REMOTE SYSTEM   REMOTE DATASET   STATE     JOB STATE   PROGRESS  RPO STATE  RESTORE POINT
   CG_1              CG        SOURCE  ibox-c          CG_1-target      ACTIVE    DONE        100%      OK         2025-03-24 16:48:53
   vol_2             VOLUME    SOURCE  ibox-c          vol_2-target     ACTIVE    DONE        100%      OK         2025-03-24 16:48:45
    
   # Query ibox-c
   replica.query replication_type=ASYNC
   ASYNC REPLICAS
   
   LOCAL DATASET     TYPE      ROLE    REMOTE SYSTEM   REMOTE DATASET   STATE     JOB STATE   PROGRESS  RPO STATE  RESTORE POINT
   CG_1-target       CG        TARGET  ibox-a          CG_1             TARGET    -           N/A       OK         2025-03-24 16:54:53
   vol_2-target      VOLUME    TARGET  ibox-a          vol_2            TARGET    -           N/A       OK         2025-03-24 16:55:45

5. On the source InfiniBox, delete the new volume Async replica and also the consistency group Async replica. Retain their staging areas for recreating the Async replicas later.

   To assist in cleaning up snapshots after they are no longer needed, it is recommended to note the time these commands are run.

   # On ibox-a, delete the volume replica in which ibox-c is the target; keep the staging area
   replica.delete local_dataset=vol_2 dataset_type=VOLUME remote_system=ibox-c retain_staging_area=yes -y
   Replica for volume "vol_2" deleted, and its staging area retained
   
   # On ibox-a, delete the consistency group replica in which ibox-c is the target; keep the staging area
   replica.delete local_dataset=CG_1 dataset_type=CG remote_system=ibox-c retain_staging_area=yes -y
   Replica for consistency group "CG_1" deleted, and its staging area retained

   A snapshot or snapshot group is generated on the source and on the target InfiniBoxes.

   A snapshot group contains a snapshot of each member of a consistency group.

   
   

6. On the source InfiniBox, suspend the Active-Active replication of the consistency group.

   # On ibox-a, suspend the replica
   replica.suspend local_dataset=CG dataset_type=CG_1 remote_system=ibox-b -y
   Replica for consistency group "CG_1" suspended

   The paths on ibox-b are now offline.

7. On the source InfiniBox, add the new volume member vol_2 to the consistency group.

   # On ibox-a, add the new volume
   cg.add_member cg=CG_1 member=vol_2 new_target_name=vol_2 -y
   Volume "vol_2" added to consistency group "CG_1"

8. On the source InfiniBox, resume the suspended replication of the Active-Active replicated consistency group.

   # On ibox-a, resume the replication
   replica.resume local_dataset=CG_1 dataset_type=CG remote_system=ibox-b -y
   Replica for consistency group "CG_1" resumed

   The paths on ibox-b are now back online.

9. On the target InfiniBox, add the target volume to the target consistency group.

   # On ibox-c, add the target volume to the target CG
   cg.add_member cg=CG_1-target member=vol_2-target -y
   Volume "vol_2-target" added to consistency group "CG_1-target"

10. On the source InfiniBox, recreate the Async replica of the consistency group from the staging area snapshot.

    # On ibox-a, resume the Async replica from the staging area
    replica.advanced.create_cg source=CG_1 replication_type=ASYNC system=ibox-c target=CG_1-target rpo=600 interval=60 -y
    Replica for consistency group "CG_1" created using automatic configuration

11. It is recommended to run queries on both the source InfiniBox and the target InfiniBox to verify that the Async replica has been recreated and that the new volume is a member of the consistency group.

    # Query ibox-a for replicated datasets
    replica.query replication_type=ASYNC
    ASYNC REPLICAS
    
    LOCAL DATASET     TYPE   ROLE    REMOTE SYSTEM    REMOTE DATASET    STATE    JOB STATE   PROGRESS  RPO STATE  RESTORE POINT
    CG_1              CG     SOURCE  ibox-c           CG_1-target       ACTIVE   DONE        100%      OK         2025-03-24 17:10:47
    
    # Query ibox-c for replicated datasets 
    replica.query replication_type=ASYNC
    ASYNC REPLICAS
    
    LOCAL DATASET     TYPE   ROLE    REMOTE SYSTEM    REMOTE DATASET    STATE    JOB STATE   PROGRESS  RPO STATE  RESTORE POINT
    CG_1-target       CG     TARGET  ibox-a           CG_1              TARGET   -           N/A       OK         2025-03-24 17:08:47
    
    # Query ibox-a for consistency group members
    cg.member_query cg=CG_1
    NAME                  CG             SIZE         USED     ALLOCATED  MAPPED
    vol_1                 CG_1       21.47 GB     36.37 MB      16.84 MB  yes
    vol_2                 CG_1        5.37 GB            0             0  no
    
    # Query ibox-c for consistency group members
    cg.member_query cg=CG_1-target
    NAME                  CG              SIZE         USED     ALLOCATED  MAPPED
    vol_1-target          CG_1-target  1.47 GB     16.84 MB      16.84 MB  yes
    vol_2-target          CG_1-target  5.37 GB            0             0  no

12. On all InfiniBoxes, map volumes vol_2 and vol_2-target to their respective hosts/clusters.

    # On ibox-a, map the new volume (vol_2) to each of its clusters
    vol.map vol=vol_2 cluster=CLU-MI-a lun=12 -y
    vol_2: Volume "vol_2" mapped to LUN 12 in cluster "CLU-MI-a"
    vol.map vol=vol_2 cluster=CLU-MI-b lun=12 -y
    vol_2: Volume "vol_2" mapped to LUN 12 in cluster "CLU-MI-b"
    
    # On ibox-b, map the new volume (vol_2) to each of its clusters
    vol.map vol=vol_2 cluster=CLU-MI-a lun=12 -y
    vol_2: Volume "vol_2" mapped to LUN 12 in cluster "CLU-MI-a"
    vol.map vol=vol_2 cluster=CLU-MI-b lun=12 -y
    vol_2: Volume "vol_2" mapped to LUN 12 in cluster "CLU-MI-b"
    
    # On ibox-c, map the new volume (vol_2-target) to each of its clusters
    vol.map vol=vol_2-target cluster=CLU-BO-a lun=12 -y
    vol_2-target: Volume "vol_2-target" mapped to LUN 12 in cluster "CLU-BO-a"

13. Rescan all the respective hosts/clusters.

Cleaning up the staging area snapshots

Snapshots groups contain individual snapshots of each volume that is a member of the consistency group.

During the process of adding a member to the replicated consistency group, one or more snapshots and snapshot groups were generated on the source InfiniBox and on the target InfiniBox. These snapshots are used for recreating the replica. It is recommended to delete these snapshots after replication is enabled.

The snapshots and snapshot groups can only be deleted manually.

Identifying the snapshots to be deleted from the source InfiniBox

To identify the snapshots and snapshot groups to be deleted:

1. Query the source InfiniBox for snapshots and snapshot groups.

   • Snapshot groups that were generated by a replica.delete command with the retain staging option include "_replica-" in their snapshot group name.

   • Volume snapshots that were generated by a replica.delete command include "auto-reclaim" in their snapshot name.
     

     # Run these queries on ibox-a
      
     # Query for snapshot groups that contain "_replica-" in their name
     cg.snap.query --columns=name --grep=_replica
     NAME
     ibox-c_68_1742822939594_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67
     
     # Query for snapshots that contain "auto-reclaim" in their name
     vol.snap.query --columns=name --grep=auto-reclaim
     NAME
     auto-reclaim-rc-87.0001-88-1742822939594

      

2. These snapshot names include the Unix epoch as a timestamp. In the snapshot names in this example, the epoch was recorded as
   1742822939594

   Use an epoch converter to view the timestamp in human-readable format.
   
   
   

3. Use the human-readable date to identify which snapshots and snapshot groups were created as staging areas at the time the replicas were deleted. 

Deleting the snapshots on the source InfiniBox

To remove a snapshot group and all its members:

1. On the source InfiniBox, delete each snapshot and snapshot group that was created as a staging area for adding the member.

   For example:

   # On ibox-a, delete snapshot groups that are no longer needed
   cg.snap.delete sg=ibox-c_68_1742822939594_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67 delete_members=yes -y
   Snapshot group "ibox-c_68_1742822939594_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67" deleted

2. On the source InfiniBox, ensure that there are no other snapshot groups that contain "_replica-" in their name.

   # On ibox-a, query for snapshot groups that were created as staging areas
   cg.snap.query --columns=name --grep=_replica

3. On the source InfiniBox, ensure that there are no other snapshots that contain "auto-reclaim" in their name.

   # On ibox-a, query for snapshots that were created as staging areas
   vol.snap.query --columns=name --grep=auto-reclaim

Identifying the snapshots to be deleted from the target InfiniBox

To identify the snapshots and snapshot groups to be deleted:

1. Query the target InfiniBox for snapshots and snapshot groups.

   • Snapshot groups that were generated by a replica.delete command with the retain staging option include "_replica-" in their snapshot group name.

   • Volume snapshots that were generated by a replica.delete command include "auto-reclaim" in their snapshot name.

     # Run these queries on ibox-c
      
     # Query for snapshot groups that contain "_replica-" in their name
     cg.snap.query --columns=name --grep=_replica
     NAME
     ibox-a_87_1742822937658_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67
     
     # Query for snapshots that contain "auto-reclaim" in their name
     vol.snap.query --columns=name --grep=auto-reclaim
     NAME
     auto-reclaim-rc-68.0001-69-1742822937213

2. The snapshot names include the Unix epoch as a timestamp.

   Use an epoch converter to view the timestamp in human-readable format.
   

3. Use the human-readable date to identify which snapshots and snapshot groups were created as staging areas at the time the replicas were deleted. 

Deleting the snapshots on the target InfiniBox

To remove a snapshot group and all its members:

1. On the target InfiniBox, delete each snapshot and snapshot group that was created for adding the member.

   For example:

   # On ibox-c, delete snapshot groups that are not longer needed
   cg.snap.delete sg=ibox-c_68_1742822939594_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67 delete_members=yes -y
   Snapshot group "ibox-c_68_1742822939594_replica-51d6919b-c7b5-4b8c-b5cb-55df2e67" deleted

2. On the target InfiniBox, ensure that there are no other snapshot groups that contain "_replica-" in their name.

   # On ibox-c, query for snapshot groups that were created as staging areas
   cg.snap.query --columns=name --grep=_replica

3. On the target InfiniBox, ensure that there are no other snapshots that contain "auto-reclaim" in their name.

   # On ibox-c, query for snapshots that were created as staging areas
   vol.snap.query --columns=name --grep=auto-reclaim

Resizing a volume in a replicated consistency group

You can enlarge a volume in an Active-Active or Async replicated consistency group.

In this example, the volume to be enlarged is vol_2 that was previously added to consistency group CG_1. This volume is replicated to the target InfiniBox as vol_2-target in consistency group CG_1-target.

To enlarge a volume in a replicated consistency group:

1. On the source InfiniBox, query the volume details.
   

   # Query ibox-a for members of the specific consistency group
   cg.member_query cg=CG_1
   NAME            CG             SIZE        USED     ALLOCATED   MAPPED
   vol_1           CG_1       21.47 GB    36.37 MB      16.84 MB   yes
   vol_2           CG_1        5.37 GB           0             0   yes

2. On the source InfiniBox, delete the Async replica of the consistency group in which the volume is a member. Retain the staging area for recreating the Async replica later.

   To assist in cleaning up snapshot groups after they are no longer needed, it is recommended to note the time this command is run.

   # On ibox-a, delete the consistency group replica in which ibox-c is the target; keep the staging area
   replica.delete local_dataset=CG_1 dataset_type=CG remote_system=ibox-c retain_staging_area=yes -y
   Replica for consistency group "CG_1" deleted, and its staging area retained

3. On the source InfiniBox, suspend the Active-Active replication of the consistency group.

   #On ibox-a, suspend the Active-Active replica
   replica.suspend local_dataset=CG_1 dataset_type=CG remote_system=ibox-b -y
   Replica for consistency group "CG_1" suspended

   The paths on ibox-b are now offline.

4. On both the source and the target InfiniBox, resize the volume.

   # On ibox-a, resize vol_2
   vol.resize vol=vol_2 size=20GiB -y
   Volume "vol_2" resized
   
   # On ibox-c, resize vol_2-target to the same size as vol_2
   vol.resize vol=vol_2-target size=20GiB -y
   Volume "vol_2-target" resized

5. On the source InfiniBox, resume the suspended replication of the Active-Active replicated consistency group.

   # On ibox-a, resume the replication
   replica.resume local_dataset=CG_1 dataset_type=CG remote_system=ibox-b -y
   Replica for consistency group "CG_1" resumed

   The paths on ibox-b are now back online.

6. On the source InfiniBox, recreate the Async replica of the consistency group from the staging area snapshot.

   # On ibox-a, resume the Async replica from the staging area
   replica.advanced.create_cg source=CG_1 replication_type=ASYNC system=ibox-c target=CG_1-target rpo=600 interval=60 -y
   Replica for consistency group "CG_1" created using automatic configuration

7. It is recommended to run queries on all InfiniBoxes to verify that the new size is correct.

   # Query ibox-a for the updated volume size
   cg.member_query cg=CG_1
   NAME           CG               SIZE          USED      ALLOCATED   MAPPED
   vol_1          CG_1         21.47 GB      36.37 MB       16.84 MB   yes
   vol_2          CG_1         21.47 GB             0              0   yes
   
   # Query ibox-b for the updated volume size
   cg.member_query cg=CG_1
   NAME           CG               SIZE          USED      ALLOCATED   MAPPED
   vol_1          CG_1         21.47 GB      16.84 MB       16.84 MB   yes
   vol_2          CG_1         21.47 GB             0              0   yes
   
   # Query ibox-c for the updated volume size
   cg.member_query cg=CG_1-target
   NAME           CG               SIZE         USED     ALLOCATED   MAPPED
   vol_1-target   CG_1-target  21.47 GB     16.84 MB      16.84 MB   yes
   vol_2-target   CG_1-target  21.47 GB            0             0   yes

8. During the process of resizing a replicated consistency group member, one or more snapshots or snapshot groups was generated. It is recommended to delete these snapshots after replica is recreated.
   For more information, see "Cleaning up the staging area snapshots" in this document.

Removing a volume from a replicated consistency group

The purpose of a consistency group is to enable the creation of consistent snapshots for all included member volumes. When a member is removed from a consistency group, it is no longer consistent with the other members of the group. If needed, the member can be moved to a pool other than the pool of the consistency group.

You can remove a volume from an Active-Active or Async replicated consistency group. Removing a volume's membership from a consistency group does not delete the volume.

In this example, the volume to be removed is vol_2 that was previously added to consistency group CG_1. This volume is replicated:

• in Active-Active mode between ibox-a and ibox-b
• in Async mode from ibox-a to ibox-c, as vol_2-target in consistency group CG_1-target

To remove a volume from a replicated consistency group:

1. On the source InfiniBox, delete the Async replica of the consistency group in which the volume is a member. Retain the staging area for recreating the Async replica later.

   To assist in cleaning up snapshot groups after they are no longer needed, it is recommended to note the time this command is run.

   # On ibox-a, delete the consistency group replica in which ibox-c is the target; keep the staging area
   replica.delete local_dataset=CG_1 dataset_type=CG remote_system=ibox-c retain_staging_area=yes -y
   Replica for consistency group "CG_1" deleted, and its staging area retained

2. On the source InfiniBox, suspend the Active-Active replication of the consistency group.

   #On ibox-a, suspend the Active-Active replica
   replica.suspend local_dataset=CG_1 dataset_type=CG remote_system=ibox-b -y
   Replica for consistency group "CG_1" suspended

   The paths on ibox-b are now offline.

3. On the source InfiniBox, remove the volume from the consistency group. Retain the staging area for recreating the Async replica later.

   To assist in cleaning up snapshot groups after they are no longer needed, it is recommended to note the time this command is run.

   #On ibox-a, remove the member from the consistency group
   cg.remove_member cg=CG_1 member=vol_2 retain_staging_area=yes -y
   Volume "vol_2" removed from consistency group "CG_1", and its staging area retained

4. On the source InfiniBox, resume the suspended replication of the Active-Active replicated consistency group.

   # On ibox-a, resume the replication
   replica.resume local_dataset=CG_1 dataset_type=CG remote_system=ibox-b -y
   Replica for consistency group "CG_1" resumed

   The paths on ibox-b are now back online.

5. On the target InfiniBox, remove the volume from the consistency group.

   # On ibox-c, remove the vol_2-target volume from CG_1-target
   infinishell ibox-c.lab.com -c "cg.remove_member cg=CG_1-target member=vol_2-target -y"
   Volume "vol_2-target" removed from consistency group "CG_1-target"

6. On the source InfiniBox, recreate the Async replica of the consistency group from the staging area snapshot.

   # On ibox-a, resume the Async replica from the staging area
   replica.advanced.create_cg source=CG_1 replication_type=ASYNC system=ibox-c target=CG_1-target rpo=600 interval=60 -y
   Replica for consistency group "CG_1" created using automatic configuration

   The volume exists on ibox-a and ibox-b but is now no longer replicated.

7. During the process of removing the replicated consistency group member, one or more snapshots or snapshot groups was generated. It is recommended to delete these snapshots after replica is recreated.

   For more information, see "Cleaning up the staging area snapshots" in this document.

Deleting a volume (optional)

You cannot delete a volume if it is a member of a consistency group. First you must remove the volume from its consistency groups.

To optionally delete a volume that was already removed from a replicated consistency group:

1. On all InfiniBoxes, query the mapping of the volume to be removed from the consistency group.
   

   # Query mapping on ibox-a 
   vol.map_query name=vol_2
   NAME          MAPPING TYPE  MAPPED TO  LUN/NSID
   vol_2         CLUSTER       CLU-MI-a   11
   vol_2         CLUSTER       CLU-MI-b   11
    
   # Query mapping on ibox-b  
   vol.map_query name=vol_2
   NAME          MAPPING TYPE  MAPPED TO  LUN/NSID
   vol_2         CLUSTER       CLU-MI-a   11
   vol_2         CLUSTER       CLU-MI-b   11
   
   # Query mapping on ibox-c  
   vol.map_query name=vol_2-target
   NAME          MAPPING TYPE  MAPPED TO  LUN/NSID
   vol_2-target  CLUSTER       CLU-BO-a   12

2. On all InfiniBoxes, use the information shown in the mapping query to unmap the volume to be deleted.

   # On ibox-a, unmap the volume from each of its mappings (In this example, it is mapped to 2 clusters)
   vol.unmap vol=vol_2 cluster=CLU-MI-a -y
   vol_2: Volume "vol_2" unmapped from LUN 12 in cluster "CLU-MI-a"
   
   vol.unmap vol=vol_2 cluster=CLU-MI-b -y
   vol_2: Volume "vol_2" unmapped from LUN 12 in cluster "CLU-MI-b"
   
   # On ibox-b, unmap the volume from each of its mappings (In this example, it is mapped to 2 clusters) 
   vol.unmap vol=vol_2 cluster=CLU-MI-a -y
   vol_2: Volume "vol_2" unmapped from LUN 12 in cluster "CLU-MI-a"
   
   vol.unmap vol=vol_2 cluster=CLU-MI-b -y
   vol_2: Volume "vol_2" unmapped from LUN 12 in cluster "CLU-MI-b"
   
   # On ibox-c, unmap the vol_2-target volume from each of its mappings (In this example, it is mapped to 1 cluster) 
   vol.unmap vol=vol_2-target cluster=CLU-BO-a -y
   vol_2-target: Volume "vol_2-target" unmapped from LUN 12 in cluster "CLU-BO-a"

   
   

3. Delete the volume on all InfiniBox systems.

   # On ibox-a, delete the volume
   vol.delete vol=vol_2 -y
   Volume "vol_2" deleted
   
   # On ibox-b, delete the volume
   vol.delete vol=vol_2 -y
   Volume "vol_2" deleted
   
   # On ibox-c, delete the volume
   vol.delete vol=vol_2-target -y
   Volume "vol_2-target" deleted