Overview 

AADM is an orchestration tool created to make migrations between Infinidat systems simple, safe, and transparent to end-users.  It was specifically designed to address real-world scenarios that would typically complicate data migration projects, allowing InfiniBox to InfiniBox migrations to be completed quickly and with high confidence.  The most common use-case for this is when a new InfiniBox is installed, and data from an older InfiniBox needs to be migrated over to it. 

Features

• Supports Windows, Linux, vSphere, AIX, and Solaris operating systems
• Supports iSCSI and FC protocols
• Supports fabric protocol and topology changes within a migration
• Supports safe migration of clusters with dead nodes (logins verified before cut-over)
• Supports Legacy platforms such as Windows 2008R2 and RHEL / CentOS 5.7+ lacking ALUA support
• Supports migration of Consistency Groups
• Supports migration of volumes with async replicas (preserving DR RPO)
• Supports the private credential stores for individual users

Orchestration Node Requirements

• The orchestration node must be CentOS or RHEL version 7 or 8, or a current Ubuntu LTS platform with at least 4GB of memory and 2 CPUs
• The orchestration node may be physical or virtual
• Root credentials will be required to install the software on the orchestration node, but the execution of AADM does not require admin privileges
• The orchestration platform will need firewall rules as follows
  • Port 443 access to InfiniBox storage systems (for HTTPS/REST)
  • Port 22 access to UNIX and vSphere platforms (for SSH)
  • Port 5985 access to (physical) Windows servers (for WinRM)
  • If Commvault IntelliSnap is being used with vSphere, AADM will need port 443 access to the vCenter server

Environment Requirements

Before initiating a data-mobility migration, the environment should meet the following requirements:

• The Infinidat systems should have a minimum firmware level of 5.0.3
• Replication should be configured between the source and target Infinidat systems, with each node having lagged 10Gb connections
• A witness should be configured for the replication link between the source and target Infinidat systems
• The hosts should be zoned to see the target Infinidat system and should have path redundancy
• The host pathing policies should be configured according to Infinidat best-practices
• The host HBA driver and firmware levels should meet Infinidat requirements
• Credentials for the host and storage platforms should be available/accessible
• Before migrating vSphere datastores, make sure the virtual machines using these datastore are not protected via VMware SRM 
  If they are, remove the datastores from the SRM protection groups, and recreate the protection groups once migration is finished

Supported Devices and Operating Systems

• Windows 2008R2 and above
• RHEL / Centos 5.7 and above
• SUSE 12 and newer
• AIX 5.1 and above
• Solaris 11
• vSphere 6.5 and above
• Consistency groups are supported
• iSCSI is supported, as are protocol conversions during migration

AADM Setup

AADM Installation

AADM is installed as an RPM or DEB package depending on the OS platform.  

For CentOS/RHEL systems:

$ rpm -i <dmstool.rpm>

For Ubuntu systems:

$ dpkg -i <dmstool.deb>

AADM Upgrade

In order to upgrade AADM the package must be removed and re-installed.  

For CentOS/RHEL systems:

$ rpm -e dmstool
$ rpm -ivh <dmstool.rpm>

For Ubuntu systems:

$ apt-get remove dmstool
$ dpkg -i <dmstool.deb>

AADM --help

AADM has a contextual help feature via the --help option:

$ aadm --help
usage: aadm [-h] {lun_list,replicate,cutover,complete,migrate,store_credentials,debugger} ...

Active-Active Data Migration orchestration utility

positional arguments:
  {lun_list,replicate,cutover,complete,migrate,store_credentials,debugger}
    lun_list            List Infinidat LUNs
    replicate           Instantiate Active/Active LUN replicas
    cutover             Transfer access of volumes from source system to target system
    complete            Completes the A/A migration by dropping the replica and (optionally) destroying the LUN from the source system
    migrate             Automatic migration of volumes using Active/Active replicas
    store_credentials   Store encrypted credentials for orchestration

If issued after a subcommand, the options specific to that subcommand context are displayed. For example, the following lists the options specific to the migrate subcommand:

$ aadm migrate --help
usage: aadm migrate [-h] [--host HOST] [--cluster CLUSTER] --source SOURCE --target TARGET [--cg CGS] [--volume VOLUMES] [--pool POOL] [--rename RENAME] [--create_pools]
                    [--background] [--remove_source_volumes] [--vcenter_refresh]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host name
  --cluster CLUSTER     Cluster name
  --source SOURCE       Source Infinidat system (or env:AADM_SOURCE_SYSTEM)
  --target TARGET       Target Infinidat system (or env:AADM_TARGET_SYSTEM)
  --cg CGS              Consistency Group(s) to migrate
  --volume VOLUMES      Volume(s) to migrate
  --pool POOL           Target Infinidat storage pool (or env:AADM_TARGET_POOL)
  --rename RENAME       Volume rename regex, e.g. <search>/<replace>
  --create_pools        Enable automatic target pool creation
  --background          Do not wait for the replica to synchronize
  --remove_source_volumes
                        Remove volumes from source system after migration
  --vcenter_refresh     Issue vcenter storage rescan commands (this is necessary for Commvault snaps)

Credentials Management

During execution, AADM will store encrypted credentials for Infinidat systems and hosts under $HOME/.infi.credentials_store to expedite execution.  In most environments, the storage admin doesn't have administrative credentials for hosts, so this allows us to cache credentials by handing off screen sharing control to the various administrators at the beginning of the migration.  The cached credentials are not visible to the operator and can be cleared after the migration for security purposes.  A best-practice approach is to configure administrative accounts specifically for the migration and to change the passwords on or disable those accounts after the migration. 

Default host credentials an be store using the aadm store_credentils --host default --user <admin_account> syntax. 

Cluster credentials can be set using the aadm store_credentials --cluster <cluster_name> --source <ibox_name> --user <admin_user> syntax.

The operator will be prompted if there are no stored credentials for a migration host, or if the stored credentials are invalid as follows:

$ aadm lun_list --host ps-aadm-01
   ... issuing async discovery for member ps-aadm-01 ...

Connecting to ps-aadm-01
Username: Administrator
Password:
Remember username and password [y/N]? y
   ... completing discovery for ps-aadm-01 ...

Preparing for Migrations

AADM connects to the hosts and storage to perform discovery, configuration, and validation functions.  The SSH protocol is used to access all hosts with the exception of Windows, which uses the WinRM protocol.  Communication with Infinidat is performed with the REST protocol over HTTPS...  In order for this discovery to succeed, the AADM orchestration node will need access to the appropriate hosts and ports (22 for SSH and 5986 for WinRM), and 443 for HTTPS).  During discovery, AADM collects interface, fabric, device, and path information to create a relational model that is used to understand system connectivity to the source and target Infinidat systems.  This relational model creates a simple means to verify path counts and states during the cut-over phase.  

The first step in preparing for a migration is to run the hostscan utility (included with the AADM package) to verify proper connectivity to the target Infinidat system.  The object is to verify that the target storage has been zoned to the platform and that the HBA driver and firmware versions are compatible.   In the following output, we can see that the host ps-centos7-01 is zoned to ibox2817 and has 6 paths across 2 HBAs:

$ hostscan --host ps-centos7-01
Discovering host ps-centos7-01 ...

################################
HOST PATH REPORT: ps-centos7-01 (centos 7)
FC-HBA host2 DRVR:10.01.00 FW:8.07.00 (21:00:00:24:ff:91:ce:22) -> switch 10:00:50:eb:1a:2b:3c:b3
    ibox2233  hct 2:0:5 wwpn 57:42:b0:f0:00:08:b9:14 (sdi, sdj, sdk, sdl)
    ibox2233  hct 2:0:0 wwpn 57:42:b0:f0:00:08:b9:24 (sda, sdb, sdc, sdd)
    ibox2233  hct 2:0:1 wwpn 57:42:b0:f0:00:08:b9:34 (sde, sdf, sdg, sdh)
    ibox2817  hct 2:0:2 wwpn 57:42:b0:f0:00:0b:01:14
    ibox2817  hct 2:0:3 wwpn 57:42:b0:f0:00:0b:01:24
    ibox2817  hct 2:0:4 wwpn 57:42:b0:f0:00:0b:01:34
FC-HBA host3 DRVR:10.01.00 FW:8.07.00 (21:00:00:24:ff:91:ce:23) -> switch 10:00:50:eb:1a:6f:d5:0c
    ibox2233  hct 3:0:5 wwpn 57:42:b0:f0:00:08:b9:13 (sdu, sdv, sdw, sdx)
    ibox2233  hct 3:0:0 wwpn 57:42:b0:f0:00:08:b9:23 (sdm, sdn, sdo, sdp)
    ibox2233  hct 3:0:2 wwpn 57:42:b0:f0:00:08:b9:33 (sdq, sdr, sds, sdt)
    ibox2817  hct 3:0:1 wwpn 57:42:b0:f0:00:0b:01:13
    ibox2817  hct 3:0:3 wwpn 57:42:b0:f0:00:0b:01:23
    ibox2817  hct 3:0:4 wwpn 57:42:b0:f0:00:0b:01:33
################################

In this case, the host has the appropriate driver and firmware levels and is zoned with redundancy to the target storage platform.

The next preparation step is to determine what volumes you would like to migrate and how much data is contained in these volumes.  This will determine the duration of the migration session.  

$ aadm lun_list --host ps-centos7-01
   ... issuing async discovery for member ps-centos7-01 ...
   ... completing discovery for ps-centos7-01 ...

FC: ibox2233  30 GiB   6742b0f0000008b9000000000576afb3 OPT_VNX__3
FC: ibox2233  30 GiB   6742b0f0000008b9000000000576afa7 OPT_VNX__2
FC: ibox2233  30 GiB   6742b0f0000008b9000000000576afad OPT_VNX__1
FC: ibox2233  15 GiB   6742b0f0000008b9000000000576afa1 PS_DATABASE_04

In this case, the amount of data being migrated is very small and will take just over a minute.  If these volumes added up to several terabytes, it would likely take an hour.  Understanding this is key to migration planning.  

Executing Migrations 

The process of performing a migration consists of the following basic operations, each of which corresponds to an AADM subcommand:

1. replicate: Creating Active/Active volume replicas on the target system and synchronizing the data
2. cutover: Transitioning host pathing from the source system to the target system
3. complete: Terminating the Active/Active replica

For example:

aadm replicate --host io-wt-14 --source ibox2233 --target ibox2817
aadm cutover --host io-wt-14 --source ibox2233 --target ibox2817
aadm complete --host io-wt-14 --source ibox2233 --target ibox2817

The three functions can also be executed through the macro subcommand migrate:

aadm migrate --host io-wt-14 --source ibox2233 --target ibox2817

AADM makes decisions based on an AI model constructed from discovery data.  In order to ensure this model is correct, only one AADM instance may be running at any given time.  When executing bulk migrations across several systems, the replicate subcommand can be used with the --background option so the command returns immediately after the A/A replica is created.  This allows an operator to create replicas for several systems and have them all synchronizing concurrently between the Infinidat systems.   The operator can subsequently use the migrate command to perform the cutover and complete functions as a macro rather than issuing those subcommands individually.  Using the cutover and complete functions allows a finer granularity of control over the timing of operations. 

The only pieces of information required to execute a migration are the hostname, the source Infinidat, and the target Infinidat. 

[kfred@ps-dmsdemo ~]$ aadm migrate --host ps-centos7-01 --source ibox2233 --target ibox2817
Discovering source system ibox2233 ...
Discovering target system ibox2817 ...
... issuing async discovery for member ps-centos7-01 ...
... completing discovery for ps-centos7-01 ...
Setting up replicas ...

OPT_VNX__1 is SYNCHRONIZED
OPT_VNX__2 is SYNCHRONIZED
PS_DATABASE_04 is SYNCHRONIZED
OPT_VNX__3 is SYNCHRONIZED

    OPT_VNX__3 100%|██████████████████████████████████████████████| 25/25 GB [01:19 < 00:00] 0.317 GB/s
    OPT_VNX__2 100%|██████████████████████████████████████████████| 25/25 GB [01:03 < 00:00] 0.398 GB/s
    OPT_VNX__1 100%|██████████████████████████████████████████████| 25/25 GB [00:51 < 00:00] 0.494 GB/s
PS_DATABASE_04 100%|██████████████████████████████████████████████| 14/14 GB [01:05 < 00:00] 0.216 GB/s
         Total 100%|██████████████████████████████████████████████| 89/89 GB [01:19 ] 1.128 GB/s

Mapping replicas of OPT_VNX__3, OPT_VNX__2, OPT_VNX__1, PS_DATABASE_04 through ibox2817

Verifying paths through ibox2817 are functional ...
Quiescing paths from ibox2233 in preparation for mapping removal ...
Un-mapping ibox2233 paths for OPT_VNX__3, OPT_VNX__2, OPT_VNX__1, PS_DATABASE_04

Discovering system ibox2233 ...
Dropping ibox2233 replicas of OPT_VNX__3, OPT_VNX__2, OPT_VNX__1, PS_DATABASE_04 ...

Migrating a cluster requires the --cluster argument.  AADM will discover the cluster nodes from the Infinidat source system cluster definition, and will create the host and cluster definitions on the target Infinidat if necessary.

(py3env) [kfred@ps-dmsdemo aadm]$ aadm migrate --cluster ps_esxi_cluster --source ibox2233 --target ibox2817 --remove_source_volumes --volume PS_AADM2_01 --rename AADM2/ESXAADM
Discovering source system ibox2233 ...
Discovering target system ibox2817 ...
Discovering cluster ps_esxi_cluster ...
   ... issuing async discovery for member io-wt-98 ...
   ... issuing async discovery for member io-wt-99 ...
   ... completing discovery for io-wt-98 ...
   ... completing discovery for io-wt-99 ...
Setting up replicas ...
   ... renaming replica of PS_AADM2_01 to PS_ESXAADM_01

PS_AADM2_01 is SYNCHRONIZED

PS_ESXAADM_01 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|  307/307 GB [06:31   <   00:00] 0.786 GB/s
        Total 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|  307/307 GB [06:31            ] 0.786 GB/s


Mapping replicas of PS_AADM2_01 through ibox2817

Verifying paths through ibox2817 are functional ...
Quiescing paths from ibox2233 in preparation for mapping removal ...
Un-mapping ibox2233 paths for PS_AADM2_01

Discovering system ibox2233 ...
Dropping ibox2233 replicas of PS_ESXAADM_01 ...
Removed ibox2233 volumes [PS_AADM2_01]

Volume Selection Logic

During migration, the scope of execution is limited to the hosts and clusters specified on the command line.  Only volumes mapped to the specified hosts and clusters will be operated upon.  Thus if a cluster member has a privately mapped volume, it must be migrated separately from the cluster volumes.  The scope of the migration can be tested with the AADM lun_list subcommand when the --array option is provided.  All of a given host's luns may be viewed using the AADM lun_list subcommand without further qualifiers.  For example:

$ aadm lun_list --host ps-aadm-01
   ... issuing async discovery for member ps-aadm-01 ...
   ... completing discovery for ps-aadm-01 ...

FC: <IBOX_2817>  30 GiB   6742b0f0000008b90000000002f65aa3 <6742b0f0000008b90000000002f65aa3>
FC: <IBOX_2817>  10 GiB   6742b0f0000008b9000000000b0d3949 <6742b0f0000008b9000000000b0d3949>
FC: <IBOX_2817>  30 GiB   6742b0f0000008b900000000035bbd63 <6742b0f0000008b900000000035bbd63>
FC: <IBOX_2817>  20 GiB   6742b0f000000b010000000000d55d62 <6742b0f000000b010000000000d55d62>
FC: <IBOX_2817>  20 GiB   6742b0f000000b010000000000d55d64 <6742b0f000000b010000000000d55d64>
FC: <IBOX_2817>  30 GiB   6742b0f0000008b90000000002f65aa5 <6742b0f0000008b90000000002f65aa5>
FC: <IBOX_2817>  10 GiB   6742b0f000000b010000000000e8c464 <6742b0f000000b010000000000e8c464>
FC: <IBOX_2817>  20 GiB   6742b0f000000b010000000000d55d60 <6742b0f000000b010000000000d55d60>
FC: <IBOX_2817>  30 GiB   6742b0f0000008b900000000035bbd65 <6742b0f0000008b900000000035bbd65>
FC: <IBOX_2817>  30 GiB   6742b0f0000008b900000000035bbd61 <6742b0f0000008b900000000035bbd61>
FC: <IBOX_2817>  20 GiB   6742b0f000000b010000000000d55d66 <6742b0f000000b010000000000d55d66>
FC: <IBOX_2817>  30 GiB   6742b0f0000008b90000000002f65aa1 <6742b0f0000008b90000000002f65aa1>

When the source system is specified the lun selection logic is invoked and only volumes matching both the host and system arguments will be listed:

$ aadm lun_list --host ps-aadm-01 --source ibox2817
Discovering source system ibox2817 ...
   ... issuing async discovery for member ps-aadm-01 ...
   ... completing discovery for ps-aadm-01 ...
skipping clustered vol AADM_HODGEPODGE_01
skipping clustered vol PS-AADM-VOL2
skipping clustered vol PS-AADM-VOL3
skipping clustered vol PS-AADM-VOL1
skipping clustered vol PS-AADM-VOL10
skipping clustered vol PS-AADM-VOL7
skipping clustered vol PS-AADM-VOL4
skipping clustered vol PS-AADM-VOL5
skipping clustered vol PS-AADM-VOL8
skipping clustered vol PS-AADM-VOL9
skipping clustered vol PS-AADM-VOL6

FC: ibox2817  10 GiB   6742b0f000000b010000000000e8c464 MY__LOCAL_VOl1

In this example, the only volume matching --host ps-aadm-01 and --source ibox2817 is MY__LOCAL_VOl1.  In order to migrate the remaining volumes, the operator must target the cluster as follows:

$ aadm lun_list --cluster ps-win2008-FO --source ibox2817
Discovering source system ibox2817 ...
Discovering cluster ps-win2008-fo ...
   ... issuing async discovery for member ps-aadm-01 ...
   ... issuing async discovery for member ps-aadm-02 ...
   ... completing discovery for ps-aadm-01 ...
   ... completing discovery for ps-aadm-02 ...
Removing volume AADM_HODGEPODGE_01 from considration as it has additional maps to hosts ['ps-win2019-01'] and clusters ['PS_ESXi_Cluster']

FC: ibox2817  30 GiB   6742b0f0000008b90000000002f65aa3 PS-AADM-VOL3
FC: ibox2817  30 GiB   6742b0f0000008b900000000035bbd63 PS-AADM-VOL5
FC: ibox2817  20 GiB   6742b0f000000b010000000000d55d62 PS-AADM-VOL8
FC: ibox2817  20 GiB   6742b0f000000b010000000000d55d64 PS-AADM-VOL9
FC: ibox2817  30 GiB   6742b0f0000008b90000000002f65aa5 PS-AADM-VOL1
FC: ibox2817  20 GiB   6742b0f000000b010000000000d55d60 PS-AADM-VOL7
FC: ibox2817  30 GiB   6742b0f0000008b900000000035bbd65 PS-AADM-VOL6
FC: ibox2817  30 GiB   6742b0f0000008b900000000035bbd61 PS-AADM-VOL4
FC: ibox2817  20 GiB   6742b0f000000b010000000000d55d66 PS-AADM-VOL10
FC: ibox2817  30 GiB   6742b0f0000008b90000000002f65aa1 PS-AADM-VOL2

When the --cluster and --source are specified, the private volume "MY_LOCAL_VOL1" as well as the more broadly mapped volume "AADM_HODGEPODGE_01" are excluded from consideration.

Statelessness

If an AADM session is interrupted, it can simply be restarted.  It has been designed to accommodate code failure, dropped VPN sessions, etc. The critical functions are provided by the Infinidat system firmware, and these data services will continue functioning even if AADM were to crash during the orchestration.  AADM is designed to determine if the things it needs for a migration already exist, and will move on to the next step if they do.  It is deterministic in its lookups of hosts, basing such a search on IQNs and WWPNs, and not on names, so a user can pre-configure a host definition and AADM will reliably find and use it.  If a host or cluster definition doesn't exist, it will create them as needed.  

Migration Exceptions

Cluster node or host connectivity issues

If a cluster node is lacking a path to one of the cluster volumes from the source system, AADM will issue an operator warning but will continue.  As long as the target storage path count matches or exceeds the source storage path count it will proceed with the migration, including the case where a cluster node has no paths to the source storage volume.  

Cluster member inoperable

It is reasonable that in a large cluster that some number of nodes may be out of commission awaiting a tech service window for repair.  In such cases, AADM will facilitate migration provided certain criteria are met to protect data availability.  During cluster discovery, if a member is unresponsive, this will be noted on the terminal and the operator will be prompted to acknowledge the condition before continuing.  At cut-over, AADM will require that the problematic host does not have initiator logins to the source system.  If the host has crashed and is still powered on, it can simply be powered off to resolve this condition.  

 Volumes created on old Infinidat firmware

Volumes created prior to firmware level 2.2 (this should only be possible if the InfiniBox serial is less than 2000) will not behave properly as an Active/Active replica.  They will tend to oscillate in and out of synchronization, and will most likely not be in the proper state for cutover.  Support has indicated that volumes created prior to firmware level 2.2 are not supported as active/active replicas, but the InfiniBox firmware will not protect you from doing this and it isn't tested.