Introduction

Overview

The InfiniBox CSI Driver is a plugin that enables InfiniBox storage management in Kubernetes environments. 

Use Helm chart or OpenShift Operator mechanisms to deploy the InfiniBox CSI Driver.

Deploying the InfiniBox CSI Driver requires:

• One or more secrets (one per InfiniBox)
• A controller instance (one per cluster)
• One or more node instances (one per worker node)

Use the InfiniBox CSI Driver to:

• Manage multiple InfiniBox storage arrays
• Provision and remove Persistent Volumes (PVs)
• Take snapshots and restore from snapshots
• Create clones of PVs
• Create raw block storage 
• Extend (resize) PVs
• Import external datasets as PVs

The InfiniBox CSI Driver supports the following access protocols:

• iSCSI

• Fibre Channel (FC)
• NVMe/TCP

• NFS

• NFS-TreeQ - for very large clusters with hundreds of thousands of PVs per InfiniBox system

The InfiniBox CSI Driver supports the following access modes:

• RWX (ReadWriteMany) - iSCSI, FC, NFS, NVMe/TCP
• ROX (ReadOnlyMany) - iSCSI, FC, NFS, NFS-TreeQ, NVMe/TCP
• RWO (ReadWriteOnce) - iSCSI, FC, NFS, NFS-TreeQ, NVMe/TCP

Software requirements and support

InfiniBox prerequisites

• A dedicated pool for every Kubernetes storage class (recommended)

• A pool admin (recommended) or an admin account 

• A network space configured for iSCSI, NFS, NVMe/TCP, or NFS-TreeQ access

Kubernetes cluster prerequisites

All worker nodes in the cluster must be able to access InfiniBox via the protocols you intend to attach.

• We recommend following the Best Practices Guides for setting up hosts for your OS.
• For Ethernet attachment, ensure that your router/firewall configurations allow the traffic.
• For FC attachment, ensure that zoning is configured correctly.

The InfiniBox CSI Driver, like similar CSI Drivers, manages low-level connectivity such as mounts and multipathing settings. This means that your Kubernetes cluster must allow "privileged pods", including both the API server and the Kubelet. Normally, privileged pods are enabled by default in many environments, including kubeadm, Rancher K3s, GCE, and GKE.

For iSCSI access, nodes must have:

• Multipath driver
• iscsid
• Filesystem software (XFS / EXT3 / EXT4)

For NVMe/TCP access, follow the instructions in Infinidat's Best Practice Guides for Setting Up Hosts with NVMe/TCP for your OS.

For FC access, nodes must have:

• Multipath driver
• FC HBA driver
• Filesystem software (XFS / EXT3 / EXT4)
• Proper FC zoning

For NFS or NFS-TreeQ, nodes must have:

• NFS client software

Installing InfiniBox CSI Driver using a Helm chart

To install InfiniBox CSI Driver using a Helm chart:

1. Download the InfiniBox CSI Driver from GitHub. To download a version, run:

   git clone --single-branch --branch v<version_number> https://github.com/Infinidat/infinibox-csi-driver.git

2. Go to the infinibox-csi-driver/deploy/Helm/infinibox-csi-driver folder.
3. For each InfiniBox to be defined, update the following InfiniBox credentials in the Infinibox_Cred list in the values configuration file values.yaml:

   • hostname: InfiniBox management interface IP address or host name.

   • username / password: InfiniBox credentials.

   • inbound_user / inbound_secret / outbound_user / outbound_secret: optional credentials for iSCSI CHAP authentication.  Note that different secret values are required for inbound and outbound or the InfiniBox will raise an error.

   • SecretName: name to be used later in the StorageClass to define a specific InfiniBox for persistent volumes.

4. It is recommended to use a dedicated namespace, such as infinidat-csi, for InfiniBox CSI Driver deployment. To create the namespace, run:

   kubectl create namespace infinidat-csi

5. Install the driver using Helm:

   helm install csi-infinibox -n=infinidat-csi ./

   In the Helm chart values, you can specify that you want the CSI Driver to use the short host names of Kubernetes nodes when registering InfiniBox hosts.  This is useful when the fully qualified host name of a Kubernetes node exceeds the 65-character limit on the InfiniBox for host names.  The Helm chart value is removeDomainName, by default this is set to false.

Optional - InfiniBox CSI Driver Helm Settings - Advanced Configuration

By default, these settings are not exposed. To change them, first "uncomment" them.

1. #multipathCleanupDelay: "1"
   • The time delay, in seconds, between a multipath flush and the removal of associated SCSI devices.
   • The default is 1  second.
2. #multipathWait: "250"
   • The time, in milliseconds, to wait for multipath devices to show up.
   • The default is  250  milliseconds.
3. #fcSearchDiskDelay: "10"
   • The time, in seconds, to wait for "devicemapper" to show up as part of publishing a new volume on a node.
   • The default is 10 seconds.
4. #resize2fsDelay: "5"
   • The time, in seconds, to wait for the "resize2fs" resize commands to run when a volume is expanded.
   • The default is 5 seconds.
5. IboxHostNamingConvention: 
   • Either leave empty to revert to the default, or set the 'nodename' as a parameter.
   • The default is 'hostname -f'.
6. logFormat: 
   • Default = "json"
   • Optional = "text"

Export rule cleanup after unmount

Starting from version 2.23.0, the InfiniBox CSI Driver can automatically remove IP addresses from NFS export rule permissions after unmounting a filesystem.

By default, this feature is disabled. To enable it:

• Uncomment the cleanupNFSPerms flag, and set it to true.
  cleanupNFSPerms: "true"

If no IP addresses remain in the export rule permissions, the driver removes the export entirely. 

Automatic protocol selection based on node protocol labeling

The InfiniBox CSI Driver can automatically select a block storage protocol based on the node’s protocol label. 

To enable this feature:

• the appropriate node labels must be applied
  • For NVMe/TCP:  infinidat.com/node-protocol=nvme
  • For FC:  infinidat.com/node-protocol=fc
  • For iSCSI:  infinidat.com/node-protocol=iscsi
• the correct 'ProtocolSecret' must be specified
• the correct 'SecretName' must be specified
• the relevant network_space details must be added

# This is an example that supports FC and NVMe labeled nodes.(FC has no network_space so it's supported with an additional setting.

ProtocolSecret: "infinibox-protocol-secret"
Infinibox_Protocol_Secrets:
  - SecretName: "infinibox-protocol-secret"
    nvme:
      network_space: "default_nvme_space"


# This is an example that supports FC and iSCSI labeled nodes. FC has no network_space so it's supported with an additional setting.

ProtocolSecret: "infinibox-protocol-secret"
Infinibox_Protocol_Secrets:
  - SecretName: "infinibox-protocol-secret"
    iscsi:
      network_space: "default_iscsi_space"

# This is an example that supports FC, NVMe, and iSCSI labeled nodes. FC has no network_space so it's supported with an additional setting.

ProtocolSecret: "infinibox-protocol-secret"
Infinibox_Protocol_Secrets:
  - SecretName: "infinibox-protocol-secret"
    iscsi:
      network_space: "default_iscsi_space"
	nvme:
      network_space: "default_nvme_space"

Protocol secret configuration

The protocol secret configuration can include the following fields:

 protocolSecret: "infinibox-protocol-secret"
  Infinibox_Protocol_Secrets: 
  - SecretName: "infinibox-protocol-secret"
    preferred_auto_order: "fc,nvme,iscsi"
    storage_protocol: "fc"
    nfs:
      network_space: "default_nas_space"
      nfs_export_permissions: ""
    iscsi:
      network_space: "default_iscsi_space"
      useCHAP: "none"
    nvme:
      network_space: "default_nvme_space"

Installing InfiniBox CSI Driver using Operator (OpenShift only)

InfiniBox CSI Driver supports an Operator that can be deployed using Red Hat OpenShift OperatorHub or another standard Operator deployment method.

In the examples on this page, we use a dedicated namespace named infinidat-csi.

To install InfiniBox CSI Driver using Operator (OpenShift only):

1. To create a namespace into which to install the 'InfiniBox CSI Driver - Operator':
   • Go to Administration > Namespaces.
   • In the Create Namespace window, assign the namespace a meaningful name, such as infinidat-csi, and click Create.
     
     
     
2. In the OpenShift console, browse to the Operators > OperatorHub view, and search for InfiniBox.
   
3. Select InfiniBox CSI Driver - Operator, and click Install.
   
4. In the Install Operator window:
   • In the Update channel section, select stable.
   • In the Installation mode section, select A specific namespace on the cluster.
   • In the Installed Namespace section, do one of the following:
     • Request the Operator's wizard to create a namespace named 'infinidat-csi'.
     • Select the name of the dedicated namespace that was previously configured using OpenShift's namespace creation. 
   • In the Update approval section, select Automatic if you want OpenShift to auto-update your InfiniBox CSI Driver Operator whenever an update becomes available (recommended), or select Manual if you want to manage the updates yourself.
     The 'Update approval' setting only affects Operator upgrades. 
     

     

     The InfiniBox CSI Driver (' infiniboxcsidriver.yaml ') itself has a different setting inside its YAML file: autoUpdate.

     By default, it is set to 'false'. To enable automatic InfiniBox CSI Driver upgrades, set it to 'true'.

     infiniboxcsidriver.yaml

       autoUpdate: false

     
     

   • Click the Install button to install the Operator.
     

5. Apply the following Security Context Constraints to the created service accounts in the dedicated namespace (must be done using the CLI):
   

   oc adm policy add-scc-to-user privileged -z infinidat-csi-operator-controller-manager
   oc adm policy add-scc-to-user privileged -z infinidat-csi-operator-infinidat-csi-driver
   oc adm policy add-scc-to-user privileged -z infinidat-csi-operator-infinidat-csi-node
   oc adm policy add-scc-to-user privileged -z infinidat-csi-operator-infinidat-csi-controller
   oc adm policy add-scc-to-user anyuid -z infinidat-csi-operator-controller-manager
   oc adm policy add-scc-to-user anyuid -z infinidat-csi-operator-infinidat-csi-driver
   oc adm policy add-scc-to-user anyuid -z infinidat-csi-operator-infinidat-csi-node
   oc adm policy add-scc-to-user anyuid -z infinidat-csi-operator-infinidat-csi-controller
   oc adm policy add-scc-to-user hostnetwork -z infinidat-csi-operator-controller-manager
   oc adm policy add-scc-to-user hostnetwork -z infinidat-csi-operator-infinidat-csi-driver
   oc adm policy add-scc-to-user hostnetwork -z infinidat-csi-operator-infinidat-csi-node
   oc adm policy add-scc-to-user hostnetwork -z infinidat-csi-operator-infinidat-csi-controller

6. Browse to Operators > Installed Operators > Operator details > InfiniBox CSI Driver - Operator, and select the InfiniboxCsiDriver tab.
   
7. Click the Create InfiniboxCsiDriver button.
   
   
8. Creating the InfiniboxCsiDriver can be done in one of two ways:

   • Yaml:
     Update the InfiniBox credentials in the YAML file as needed, and click Create.
     
     
     

     • Optional - InfiniBox CSI Driver Settings - Advanced Configuration

       • Starting from version 2.21.0, the InfiniBox CSI Driver offers the option of editing driver wait times for 'multipathd' operations in environments with heavy workloads.

         To set them, add the following settings:

         • multipathCleanupDelay: "1"
           • The time delay, in seconds, between a multipath flush and the removal of associated SCSI devices.
           • The default is 1 second.
         • multipathWait: "250"
           • The time, in milliseconds, to wait for multipath devices to show up.
           • The default is 250 milliseconds.
         • fcSearchDiskDelay: "10"
           • The time, in seconds, to wait for "devicemapper" to show up as part of publishing a new volume on a node.
           • The default is 10 seconds.
         • resize2fsDelay: "5"
           • The time, in seconds, to wait for the "resize2fs" resize commands to run when a volume is expanded.
           • The default is 5 seconds.
         • IboxHostNamingConvention: 
           • Either leave empty to revert to the default, or set the 'nodename' as a parameter.
           • The default is 'hostname -f'.
       • Starting from version 2.26.0, a log format can be selected.
         • logFormat: 
           • Default = "json"
           • Optional = "text"

       • The InfiniBox CSI Driver can automatically remove IP addresses from NFS export rule permissions after unmounting a filesystem.

         By default, this feature is disabled. To enable it:

         • Uncomment the cleanupNFSPerms flag, and set it to true.
           cleanupNFSPerms: "true"

         If no IP addresses remain in the export rule permissions, the driver removes the export entirely. 

         
         

       • Starting from version 2.23.0, the InfiniBox CSI Driver can automatically select a storage protocol based on the node’s available connectivity.

         Important

         For automatic protocol selection to work correctly, all nodes must have identical protocol configurations.

         To enable this feature:

         • Set the protocolSecret value to a valid Kubernetes Secret name. For example: protocolSecret "infinibox-protocol-secret".
           • The protocolSecret string must exactly match the SecretName defined in the Infinibox_Protocol_Secrets list. See the example below.
           • If protocolSecret is left blank (""), automatic protocol selection is disabled.
         • Set the storage_protocol to "auto" in the protocol secret to allow the driver to select the preferred storage protocol based on availability and the preferred_auto_order list.
         To view a description of the protocol secret configuration fields and an example, see the Protocol secret configuration section above.

         
         

       • Starting from version 2.22.0, InfiniBox CSI Driver allows users to specify tolerations and resource limits:
         

         nodeservertolerations:
         
           - key: ""
         
             effect: ""
         
             tolerationSeconds: 0
         
             operator: "Exists"
         
             value: ""
         
           nodeserverresources:
         
             requests:
         
               cpu: "0.1"
         
               memory: "150Mi"
         
             limits:
         
               cpu: "0.5"
         
               memory: "450Mi"
         
           controllerservertolerations:
         
           - key: ""
         
             effect: ""
         
             tolerationSeconds: 0
         
             operator: "Exists"
         
             value: ""
         
           controllerserverresources:
         
             requests:
         
               cpu: "0.1"
         
               memory: "150Mi"
         
             limits:
         
               cpu: "0.5"
         
               memory: "450Mi"

   • Starting from InfiniBox CSI Driver version 2.10.0, OpenShift admins can modify the following CustomResource configuration.

     • Specify nodeSelector to determine which nodes the driver is installed on: 

       ## Default values (generic label):
       
       nodeSelector:
           kubernetes.io/os: linux
       
       
       ## With an Infinidat label:
       
       nodeSelector:
           infinidat: true
           kubernetes.io/os: linux

     • Strip the domain name from the node when registered on InfiniBox:

       ## Default value - FQDN:   
       
       removeDomainName: false
       
       ## Host name only - true:   
       
       removeDomainName: true

   • Form View: 
     
     

     The 'Infinibox_Cred' is deprecated.

   • Fill out the 'Ibox_Creds' with your InfiniBox settings, and click 'Create'.
     
     
9. Browse to the Workloads > Pods view, and confirm that both the Operator and the InfiniBox CSI Driver are running. These are displayed as:
   • "controller-manager-*"
   • "infiniboxcsidriver-sample-driver-0"
     

Upgrades - Helm chart

Upgrading to a new InfiniBox CSI Driver requires uninstalling the older version, updating dependencies, and applying changes to CRDs and RBAC configurations.

To upgrade:

1. Uninstall the old version and install the new version using a fresh copy of the Helm charts.
2. Prepare the Kubernetes nodes by upgrading to at least version 8.0.1 of the external snapshotter.
3. Modify the Helm chart to match any local changes (templates, values.yaml).
4. Uninstall the old external snapshotter and install the new version, including the necessary CRDs (VolumeGroupSnapshot).
5. Install the new CSI Driver using the updated Helm chart.

Upgrades - OpenShift Operator

The information in this section applies to OpenShift only.

InfiniBox CSI Driver follows standard Kubernetes upgrade methods. A new version deployment does not affect existing persistent volumes (PVs) or persistent volume claims (PVCs), unless otherwise indicated in the relevant release notes.

The InfiniBox CSI Driver Operator 'Update Approval' setting is defined during installation in the 'Install Operator' window, and it can be changed later. If it is set to Automatic, then the 'InfiniBox CSI Driver Operator' is always kept up-to-date. If it is set to Manual, all upgrades must be done manually.

To upgrade the Operator manually:

1. Uninstall the existing Operator.
   See the 'Uninstalling InfiniBox CSI Driver using OpenShift Operator' section.
2. Install the Operator again.
   See the 'Installing InfiniBox CSI Driver using Operator (OpenShift only)' section.
   
   

  autoUpdate: false

Uninstalling

Uninstalling InfiniBox CSI Driver using a Helm chart

To uninstall the driver from the infinidat-csi namespace, run:

helm uninstall csi-infinibox -n=infinidat-csi

If you installed into a different namespace, replace infinidat-csi in the command.

Uninstalling InfiniBox CSI Driver using OpenShift Operator

1. Browse to Operators > Installed Operators > Operator details > InfiniBox CSI Driver - Operator, and select the InfiniboxCsiDriver tab.
   
2. In the action menu on the right of the driver row, click Delete InfiniboxCsiDriver.
   
3. Browse back to Operators > Installed Operators.
   
4. In the action menu on the right of the InfiniBox CSI Driver - Operator row, click Uninstall Operator.
5. In the confirmation window, click Uninstall.
   
6. Go to User Management > RoleBindings.
   
7. For each of the following ClusterRoleBindings that exists in 'InfiniBox CSI Driver - Operator', open the action menu on the right of its row, and click Delete ClusterRoleBinding:
   • infinidat-csi-operator-infinidat-csi-attacher
   • infinidat-csi-operator-infinidat-csi-controller
   • infinidat-csi-operator-infinidat-csi-driver
   • infinidat-csi-operator-infinidat-csi-node
   • infinidat-csi-operator-infinidat-csi-provisioner
   • infinidat-csi-operator-infinidat-csi-resizer
   • infinidat-csi-operator-infinidat-csi-snapshotter
     
     

Using the InfiniBox CSI Driver

Sample yaml files for different protocols are available at https://github.com/Infinidat/infinibox-csi-driver/tree/master/deploy/examples.

Defining a StorageClass

A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, backup policies, or other arbitrary policies determined by the cluster administrators. This concept of different classes for different purposes is sometimes called “profiles” in other storage systems. See Kubernetes documentation for more details.

A StorageClass with an InfiniBox CSI Driver maps to a specific pool on an InfiniBox. No two StorageClasses are allowed within the same InfiniBox pool.

StorageClass example for Fibre Channel protocol

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: infi-fc-storageclass-demo
provisioner: infinibox-csi-driver
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters: 
  # this section points to the secret file with the credentials (defining InfiniBox IP, user name and password)
  csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
  csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
  csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

  # define file system for the provisioned volume. Supported options are xfs, ext3, ext4
  fstype: xfs

  # define InfiniBox-specific parameters 
  storage_protocol: "fc"
  pool_name: "k8s_csi"
  provision_type: "THIN"
  ssd_enabled: "true"

StorageClass example for iSCSI protocol

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: infi-iscsi-storageclass-demo
provisioner: infinibox-csi-driver
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters: 
  # this section points to the secret file with the credentials (defining InfiniBox IP, user name and password)
  csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
  csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
  csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

  # define whether CHAP should be used to protect access to volumes. Supported options: none, chap, mutual_chap
  useCHAP: "mutual_chap" 

  # define file system and permissions for the provisioned volume
  fstype: xfs  				# options are xfs, ext3, ext4
  uid: 1000    				# optional - set uid if volume mountpoint should be chown'ed
  gid: 1000    				# optional - set gid if volume mountpoint should be chown'ed
  unix_permissions: 777 	# optional - set permissions if volume mountpoint should be chmod'ed

  # define InfiniBox-specific parameters 
  storage_protocol: "iscsi"
  network_space: "iscsi1","iscsi2"
  pool_name: "k8s_csi"
  provision_type: "THIN"
  ssd_enabled: "true"

StorageClass example for NFS protocol

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibox-nfs-storageclass-demo
provisioner: infinibox-csi-driver
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions: # optional: defaults shown below, be sure to include vers=3 if you override
  - vers=3
  - tcp
  - rsize=262144
  - wsize=262144
parameters:
    # reference secret with InfiniBox credentials
    csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
    csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
    csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
    csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
    csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
 	csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  	csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

    # InfiniBox configuration
    storage_protocol: nfs
    network_space: NAS # InfiniBox network space name
    pool_name: nfs-example-pool # InfiniBox pool name
    provision_type: THIN
    ssd_enabled: "true"

    # optional parameters
    # snapdir_visible: "true"   # optional: specify whether .snapshot directory is visible
    # NOTE:  uid, guid, unix_permissions parameters require no_root_squash to be true!
    # uid: "1000"               # optional: override default UID for filesystem mount 
    # gid: "1000"               # optional: override default GID for filesystem mount
    # unix_permissions: "777"   # optional: override default permissions for filesystem mount
    # privileged_ports_only: no # optional: force use of  privileged ports only
    # nfs_export_permissions : "[{'access':'RW','client':'192.168.147.190-192.168.147.199','no_root_squash':true}]" # ip address range example
    # nfs_export_permissions : "[{'access':'RW','client':'*','no_root_squash':true}]" # wildcard example

StorageClass example for NFSv4 protocol

NFSv4 is supported by InfiniBox version 8.5.0 and above.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibox-nfs-storageclass-demo
provisioner: infinibox-csi-driver
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions: # optional: defaults shown below, be sure to include vers=3 if you override
  - vers=3
  - tcp
  - rsize=262144
  - wsize=262144
parameters:
    # reference secret with InfiniBox credentials
    csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
    csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
    csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
    csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
    csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
 	csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  	csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

    # InfiniBox configuration
    storage_protocol: nfs
    network_space: NAS # InfiniBox network space name
    pool_name: nfs-example-pool # InfiniBox pool name
    provision_type: THIN
    ssd_enabled: "true"
    vers=4.1

    # optional parameters
    # snapdir_visible: "true"   # optional: specify whether .snapshot directory is visible
    # NOTE:  uid, guid, unix_permissions parameters require no_root_squash to be true!
    # uid: "1000"               # optional: override default UID for filesystem mount 
    # gid: "1000"               # optional: override default GID for filesystem mount
    # unix_permissions: "777"   # optional: override default permissions for filesystem mount
    # privileged_ports_only: no # optional: force use of  privileged ports only
    # nfs_export_permissions : "[{'access':'RW','client':'192.168.147.190-192.168.147.199','no_root_squash':true}]" # ip address range example
    # nfs_export_permissions : "[{'access':'RW','client':'*','no_root_squash':true}]" # wildcard example

StorageClass example for NVMe/TCP protocol

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: infi-nvme-storageclass-demo
provisioner: infinibox-csi-driver
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters: 
  # this section points to the secret file with the credentials (defining InfiniBox IP, user name and password)
  csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
  csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
  csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
  csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
  csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
  csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

  # define file system for the provisioned volume. Supported options are xfs, ext3, ext4
  fstype: xfs

  # define InfiniBox-specific parameters 
  storage_protocol: "nvme"
  pool_name: "k8s_csi"
  provision_type: "THIN"
  ssd_enabled: "true"

StorageClass example for NFS-TreeQ

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibox-treeq-storageclass-demo-retain
provisioner: infinibox-csi-driver
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - hard
  - rsize=1048576
  - wsize=1048576
parameters: 
    pool_name: testpool
    network_space: NAS
    provision_type: THIN
    storage_protocol: nfs_treeq
    fs_prefix: csit_
    ssd_enabled: "true"
    csi.storage.k8s.io/provisioner-secret-name: infinibox-creds
    csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi
    csi.storage.k8s.io/controller-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-stage-secret-name: infinibox-creds
    csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-publish-secret-name: infinibox-creds
    csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/controller-expand-secret-name: infinibox-creds
    csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
  	csi.storage.k8s.io/node-expand-secret-name: infinibox-creds
  	csi.storage.k8s.io/node-expand-secret-namespace: infinidat-csi

    # optional parameters
    # the following parameters can be used to override InfiniBox limits
    # max_filesystems: "999"
    # max_treeqs_per_filesystem: "20"
    # max_filesystem_size: 30gib
    # uncomment if you do not want to have the default export (e.g. node ip address) created
    # nfs_export_permissions : "[{'access':'RW','client':'192.168.147.190-192.168.147.199','no_root_squash':true}]" # ip address range example
    # nfs_export_permissions : "[{'access':'RW','client':'*','no_root_squash':true}]" # wildcard example

Creating a sample StorageClass

$ kubectl create -f storageclass.yaml
storageclass.storage.k8s.io/ibox-nfs-storageclass-demo created

$ kubectl get storageclass
NAME                         PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE  ALLOWVOLUMEEXPANSION  AGE
ibox-nfs-storageclass-demo   infinibox-csi-driver   Delete          Immediate          true                  9s

Defining fsGroup

Starting from version 2.10.0, the InfiniBox CSI Driver honors fsGroup in the pod specification, on a per-pod basis. When a pod specification has fsGroup included in the securityContext, the fsGroup will be applied to the volume when it is mounted. 

When fsGroup is specified by the pod, the following parameters of the InfiniBox CSI Driver StorageClass are ignored: uid, gid, unix_permissions. If these are specified in the StorageClass, they will not be used for any pod that uses a securityContext with fsGroup. For any other pod NOT including the fsGroup in the securityContext, they will be applied if present in the same StorageClass.

Working with fsGroup requires the following:

• Kubernetes v1.26 or higher.

• The storage class must specify the fsType. If the desired fsType is set to the default (ext4), it still must be explicitly stated:

  csi.storage.k8s.io/fstype: ext4

• The accessMode of the PVC must be set to ReadWriteOnce.
   

  spec:
      accessModes:
      - ReadWriteOnce

  fsGroupChangePolicy is not supported by Kubernetes for CSI Drivers when fsGroup is specified. "Always" is used by the InfiniBox CSI Driver regardless of the pods setting. This may increase the time for the volume to mount if there are a large number of files and directories in the volume. 

  fsGroup Pod example

  kind: Pod
  apiVersion: v1
  metadata:
    name: nfs-test-fsgroup
    namespace: infinidat-csi
  spec:
    securityContext:
      runAsUser: 3000
      runAsGroup: 3000
      fsGroup: 3000
    containers:
      - name: csitest
        image: infinidat/csitestimage:latest
        volumeMounts:
        - mountPath: "/tmp/csitesting"
          name: ibox-csi-volume
        command: [ "csitestimage" ]
    volumes:
      - name: ibox-csi-volume
        persistentVolumeClaim:
          claimName: nfs-pvc-fsg

Defining a Persistent Volume Claim (PVC)

A persistent volume claim (PVC) is a request for the platform to create a persistent volume (PV). Each PVC contains the spec (specification) and status of the claim. 

PVC example for pool annotations

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
  namespace: infinidat-csi
  annotations: # pool1 and networkspace1 are an example
    infinidat.com/pool_name: pool1
    infinidat.com/network_space: networkspace1
    infinidat.com/volume_metadata: "Annotation_Text_Goes_Here"

PVC example for system annotations

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc-anno-secret
  namespace: infinidat-csi
  annotations:
    infinidat.com/ibox_secret: infinibox-creds
    infinidat.com/volume_metadata: "Annotation_Text_Goes_Here" 

In the example above, the secret named infinibox-creds is used to determine on which InfiniBox to create the PVC. To use the InfiniBox secret annotation, configure the StorageClass as follows:

parameters:    

	# reference secret with InfiniBox credentials
    csi.storage.k8s.io/controller-expand-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
    csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
    csi.storage.k8s.io/controller-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
    csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
    csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
    csi.storage.k8s.io/node-stage-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
    csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi

For more examples, refer to the InfiniBox CSI Driver Github page.

PVC example for Fibre Channel or iSCSI

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
  namespace: infinidat-csi
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Block # Supported options: Block, Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: infi-storageclass-demo

PVC example for NFS or NFS-TreeQ

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: infi-pvc-demo
  namespace: infinidat-csi
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: infi-storageclass-demo

Creating a sample PVC

$ kubectl create -f pvc.yaml
persistentvolumeclaim/ibox-pvc-demo created
$ kubectl get pvc -n infinidat-csi
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ibox-pvc-demo Bound csi-d38f4662c8 1Gi RWX infi-nfs-storageclass-demo 4s

$ kubectl get pv csi-d38f4662c8
NAME            CAPACITY  ACCESS MODES  RECLAIM POLICY   STATUS  CLAIM               STORAGECLASS                REASON  AGE
csi-d38f4662c8  1Gi       RWX           Delete           Bound   infi/ibox-pvc-demo  infi-nfs-storageclass-demo          26s

Defining a volume snapshot class

apiVersion: snapshot.storage.k8s.io/v1 
kind: VolumeSnapshotClass 
metadata: 
 name: ibox-snapshotclass-demo-locking 
 namespace: infinidat-csi 
 annotations: 
   snapshot.storage.kubernetes.io/is-default-class: "true" 
driver: infinibox-csi-driver 
deletionPolicy: Delete 
parameters: 
 lock_expires_at: "1 Hours" 
 csi.storage.k8s.io/snapshotter-secret-name: infinibox-creds 
 csi.storage.k8s.io/snapshotter-secret-namespace: infinidat-csi

Starting from version 2.8.0, InfiniBox CSI Driver supports immutable snapshots. This feature locks snapshots for a pre-determined period of time during which no changes can be made. In the VolumeSnapshotClass, under parameters (see example above), you can set a lock_expires_at: parameter to a specific length of time. Use one of the following formats:

• "__ Hours"
  • Example:
    lock_expires_at: "18 Hours"
• "__ Days"
  • Example:
    lock_expires_at: "7 Days"
• "__ Weeks"
  • Example:
    lock_expires_at: "4 Weeks"
• "__ Years"
  • Example:
    lock_expires_at: "1 Years"

If you do not specify any value, or if you omit the lock_expires_at: parameter, the default is 0 (unlocked).

Defining a snapshot

A snapshot of a PVC is a read-only InfiniBox snapshot of the relevant PV.

$ cat snapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: infi-pvc-snapshot-demo
  namespace: infinidat-csi
spec:
  snapshotClassName: infi-snapshotclass-demo
  source:
    name: infi-pvc-demo
    kind: PersistentVolumeClaim

$ kubectl create -f snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/ibox-pvc-snapshot-demo created
$ kubectl get volumesnapshot -n infinidat-csi
NAME                    AGE
ibox-pvc-snapshot-demo  10s
$ kubectl get volumesnapshotcontent
NAME                                              AGE
snapcontent-40cf3378-4dde-42c9-87f0-8c6a9771e40e  18s

Defining a PV restore from a snapshot

A PVC can be defined as a restore from a previously created snapshot. For InfiniBox, the underlying PV is created as a ReadWrite snapshot of a ReadOnly snapshot.

$ cat restoresnapshot.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: infi-snapshot-pvc-restore-demo-2
  namespace: infinidat-csi
spec:
  storageClassName: infi-nfs-storageclass-demo
  dataSource:
    name: infi-pvc-snapshot-demo
    kind: VolumeSnapshot
    apiGroup: "snapshot.storage.k8s.io"
  accessModes: 
  - ReadWriteOnce
  resources:
     requests:
       storage: 2Gi

$ kubectl create -f restoresnapshot.yaml
persistentvolumeclaim/ibox-snapshot-pvc-restore-demo-2 created
$ kubectl get pvc infi-snapshot-pvc-restore-demo-2 -n infi
NAME                               STATUS   VOLUME          CAPACITY  ACCESS MODES  STORAGECLASS                AGE
ibox-snapshot-pvc-restore-demo-2   Bound    csi-40e9d7d588  1Gi       RWO           infi-nfs-storageclass-demo  13s

Defining a clone

A new PVC can be created as a ReadWrite snapshot of another PVC, creating an instant clone.

$ cat clonepvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ibox-pvc-clone-demo
  namespace: infinidat-csi
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ibox-nfs-storageclass-demo
  resources:
    requests:
      storage: 2Gi
  dataSource:
    kind: PersistentVolumeClaim
    name: infi-pvc-demo

$ kubectl create -f clonepvc.yaml
persistentvolumeclaim/ibox-pvc-clone-demo created

$ kubectl get pvc ibox-pvc-clone-demo -n infinidat-csi
NAME                 STATUS  VOLUME          CAPACITY  ACCESS MODES   STORAGECLASS                AGE
ibox-pvc-clone-demo  Bound   csi-eb7aa34161  1Gi       RWO            infi-nfs-storageclass-demo  9s

Expanding a PV

An existing PV can be expanded using the kubectl edit command. Kubernetes interprets a change to the spec "storage" field as a request for more space, and it triggers automatic volume resizing.

$ kubectl edit pvc infi-pvc-demo -n infinidat-csi
....

spec:
accessModes:
- ReadWriteMany
resources:
  requests:
    storage: 1Gi <<<<< modify this field
storageClassName: infi-nfs-storageclass-demo
....

Dataset deletion considerations

The CSI spec assumes that datasets are not related, allowing snapshots to have lifecycles independent of their originating datasets. This is not aligned to the InfiniBox snapshot implementation. To minimize the impact, the InfiniBox CSI Driver currently attempts to automatically delete descendant datasets when deletion of the parent dataset is requested. For example, if you try to delete a PV with existing snapshots, the snapshots will also be deleted. However, this will fail for any datasets that are currently attached to hosts. As a best practice, before deleting a dataset, confirm that you want to also delete all its descendants.

Importing an existing volume

To import an existing PV and make it manageable by the InfiniBox CSI Driver, manually create a PV yaml file that describes the parameters of the existing volume.  

Importing an existing iSCSI PV

$ cat importpv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: infinibox-csi-driver
  name: gtvolpv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  csi:
    controllerExpandSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    controllerPublishSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    driver: infinibox-csi-driver
    nodePublishSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    nodeStageSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    volumeAttributes:
      Name: "gtvolpv"
      fstype: "ext4"
      network_space: "iscsi1"
      portals: "172.20.37.54,172.20.37.55,172.20.37.57"
      storage_protocol: "iscsi"
      useCHAP: "none"
	  iqn: iqn.2009-11.com.infinidat:storage:infinibox-sn-36000-2436
    volumeHandle: 9676520$$iscsi
  persistentVolumeReclaimPolicy: Delete
  storageClassName: infi-iscsi-storageclass-demo
  volumeMode: Filesystem

$ cat importpvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: infi-import-pvc-demo
  namespace: infinidat-csi
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: infi-iscsi-storageclass-demo
  volumeName: gtvolpv

Importing an existing NFS PV

$ cat importpv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: infinibox-csi-driver
  name: gtfspv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 2Gi
  csi:
    controllerExpandSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    controllerPublishSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    driver: infinibox-csi-driver
    nodePublishSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    nodeStageSecretRef:
      name: infinibox-creds
      namespace: infinidat-csi
    volumeAttributes:
      ipAddress: 172.20.37.53
      volPathd: /gtfs_pv
      storage_protocol: nfs
      exportID: "10098" #InfiniBox export ID
    volumeHandle: 7955656$$nfs #InfiniBox file system ID
  persistentVolumeReclaimPolicy: Delete
  storageClassName: infi-nfs-storageclass-demo
  volumeMode: Filesystem



$ cat importpvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: infi-import-pvc-demo
  namespace: infinidat-csi
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  storageClassName: infi-nfs-storageclass-demo
  volumeName: gtfspv

Managing additional InfiniBox storage arrays

A separate secret must be defined within the Kubernetes cluster for every managed InfiniBox array. It must include an InfiniBox host name, administrator credentials, and optional CHAP authentication credentials (for iSCSI), all encoded in Base64.

To encode an entry:

$ echo -n infi0001.company.com | base64
aWJveDAwMDEuY29tcGFueS5jb20=

Sample secret file:

apiVersion: v1
kind: Secret
metadata:
  name: infi0001-credentials
  namespace: infinidat-csi
type: Opaque
data:
  hostname: aWJveDAwMDEuY29tcGFueS5jb20=
  node.session.auth.password: MC4wMDB1czA3Ym9mdGpv
  node.session.auth.password_in: MC4wMDI2OHJ6dm1wMHI3
  node.session.auth.username: aXFuLjIwMjAtMDYuY29tLmNzaS1kcml2ZXItaXNjc2kuaW5maW5pZGF0OmNvbW1vbmlu
  node.session.auth.username_in: aXFuLjIwMjAtMDYuY29tLmNzaS1kcml2ZXItaXNjc2kuaW5maW5pZGF0OmNvbW1vbm91dA==
  password: MTIzNDU2
  username: azhzYWRtaW4=

Starting from version 2.9.0, users can specify multiple InfiniBox credentials when they install the InfiniBox CSI Driver. The CSI Driver creates a secret in the Helm chart or the OpenShift CustomResource for each InfiniBox that is defined.

Users can opt to manage the secret creation themselves and not have the CSI Driver create any InfiniBox secrets. To select this option, set the Helm chart or OpenShift CustomResource value of skipCredentialsCreation to True.

Troubleshooting

Use standard Kubernetes troubleshooting actions to debug InfiniBox CSI Driver issues. By default, debugging is enabled in the values configuration file, values.yaml, and can be disabled:

• Enabled → logLevel: "debug"
• Disabled → logLevel: "info"

To generate a log file for the InfiniBox CSI Driver, run:

kubectl logs <pod name> <container name>

# For OpenShift run:
oc logs <pod name> <container name>

Pods of interest related to the InfiniBox CSI Driver include:

Controller (infiniboxcsidriver-sample-driver-0), which includes 5 containers:

• driver (the main container to focus on for troubleshooting)
• resizer
• snapshotter
• provisioner
• attacher

Node - select an instance running on the relevant worker. Each instance includes 2 containers:

• driver (the main container to focus on for troubleshooting)
• registrar

Operator - for operator-managed deployments only

Relevant error messages can be found using the kubectl describe command.

$ kubectl get nodes
NAME         STATUS  ROLES   AGE  VERSION
gtouret-k51  Ready   master  21d  v1.18.0
gtouret-k52  Ready   <none>  21d  v1.18.0
gtouret-k53  Ready   <none>  21d  v1.18.0

$ kubectl get pods -n infinidat-csi -o wide
NAME                      READY  STATUS   RESTARTS  AGE   IP             NODE 
csi-infinibox-driver-0    5/5    Running  0         2d4h  10.244.2.55    gtouret-k53 
csi-infinibox-node-85g4t  2/2    Running  0         2d4h  172.20.87.214  gtouret-k51 
csi-infinibox-node-jw9hx  2/2    Running  0         2d4h  172.20.87.99   gtouret-k52 
csi-infinibox-node-qspr4  2/2    Running  0         2d4h  172.20.78.63   gtouret-k53

$ kubectl logs csi-infinibox-driver-0 driver -n infinidat-csi | tail -2

time="2020-04-25T05:07:03Z" level=info msg="Called createVolumeFrmPVCSource"
time="2020-04-25T05:07:03Z" level=info msg="Request made for method: GET and apiuri /api/rest/filesystems/10748988"

Sending support data to Infinidat

Starting from version 2.18.0, InfiniBox CSI Driver automatically sends support data to Infinidat to improve supportability. You can disable this feature.

Using a Helm Chart to stop sharing support data with Infinidat

In your values.yaml configuration file, remove the comment mark in the following command:

#createEvents: "false".

Using OpenShift to stop sharing support data with Infinidat

In your CustomResource file, set the CreateEvents command to false:

"createEvents: false"

InfiniBox CSI Driver metrics exporter

The InfiniBox CSI Driver metrics exporter collects InfiniBox-related statistics that can be analyzed and presented by the Prometheus monitoring system. 

The following InfiniBox statistics can be collected: 

You can run the InfiniBox CSI Driver metrics exporter on OpenShift or on plain Kubernetes.

Enabling HTTPS/TLS Security

Starting from version 2.23.0, the InfiniBox CSI Driver supports the InfiniBox custom metrics server listening over HTTPS/TLS using a user-supplied certificate and key. The InfiniBox custom metric server is used for monitoring the InfiniBox CSI Driver via Prometheus. 

To enable this feature:

1. Use the deploy-tls.yaml file to create the Deployment. This sets the TLS_LISTEN environment variable to "true".
2. Generate a self-signed certificate and key:
   

   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=example.com"

   
   
3. Create a Kubernetes TLS Secret:
   

   kubectl create secret tls my-tls-secret --cert=tls.crt --key=tls.key

OpenShift

For monitoring and analytics purposes, Prometheus is installed in OpenShift environments by default.

To install infinidat-csi-metrics on OpenShift:

• Create the following resources on your OpenShift instance:
  • oc create configmap infinidat-csi-metrics-config --from-file=./config.yaml
  • oc create -f cluster-monitoring-config.yaml
  • oc create -f deploy.yaml
  • oc create -f servicemonitor.yaml

To view infinidat-csi-metrics:

• After infinidat-csi-metrics is installed, query it for metrics that begin "ibox_"

To uninstall infinidat-csi-metrics:

• Run the following commands:
  • oc delete configmap infinidat-csi-metrics-config
  • oc delete -f cluster-monitoring-config.yaml
  • oc delete -f deploy.yaml
  • oc delete -f servicemonitor.yaml

Kubernetes

Use a Helm chart to install the InfiniBox CSI Driver metrics exporter.

To install infinidat-csi-metrics on Kubernetes:

1. The Prometheus community provides a Helm chart. Run the following command to add it as a repo:
   
   • Helm repo add prometheus-community https://prometheus-community.github.io/Helm-charts
2. Run the following command to create a namespace for Prometheus to be installed into:
   
   • kubectl create ns prom
3. Run the following commands to deploy the Prometheus stack. Note that the values.yaml file configures Prometheus to scrape the metrics collected by the InfiniBox CSI Driver metrics exporter.
   • cd deploy/metrics
   • Helm install -f values.yaml prometheus prometheus-community/kube-prometheus-stack -n prom

To view the infinidat-csi-metrics deployment:

• Run the following command:
  
  • kubectl get all -n prom

To access the Prometheus dashboard:

• Run the following command:
  
  • kubectl port-forward -n prom prometheus-prom-kube-prometheus-stack-prometheus-0 9090
• Browse to the Grafana-web application:
  • http://localhost:9090

To access the Grafana dashboard:

• Run the following command:
  • kubectl port-forward -n prom prom-grafana-6c578f9954-rjdmk 3000
• Browse to the Grafana-web application.
  • http://localhost:3000
• Enter the grafana-web application authentication credentials.
  The default user/password credentials are admin/prom-operator.

To uninstall Prometheus:

• Run the following commands:
  • Helm uninstall prometheus -n prom
  • kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
  • kubectl delete crd alertmanagers.monitoring.coreos.com
  • kubectl delete crd podmonitors.monitoring.coreos.com
  • kubectl delete crd probes.monitoring.coreos.com
  • kubectl delete crd prometheuses.monitoring.coreos.com
  • kubectl delete crd prometheusrules.monitoring.coreos.com
  • kubectl delete crd servicemonitors.monitoring.coreos.com
  • kubectl delete crd thanosrulers.monitoring.coreos.com

To configure Prometheus to scrape infinidat-csi-metrics:

• Add the following to the Prometheus configuration:
  • yaml:
    • job_name: 'infinidat-csi'
    • static_configs:
    • targets: ['infinidat-csi-metrics.infinidat-csi.svc.cluster.local:11007]

Troubleshooting the InfiniBox CSI Driver metrics exporter

Run the curl command on the metrics service to verify that metrics are being produced and collected.

To debug infinidat-csi-metrics:

1. Run the following command:
   • kubectl port-forward service/infinidat-csi-metrics 11007:11007
2. On another terminal, run:
   • curl localhost:11007/metrics
3. View the results. If there are errors, contact Infinidat support.
4. To view the configured targets, run the following command:
   • curl localhost:11007/targets
     
5. Verify that there is an entry for infinidat-csi. If there are no entries:
   • Repeat the installation instructions.
   • If there are still no entries, contact Infinidat support.

Consistency group and snapshot group support

Starting from version 2.15.0, InfiniBox CSI Driver implements ‘Volume Group Snapshots’.

This feature allows users to make snapshots (SGs) of groups of volumes with a single resource request.

InfiniBox CSI Driver - Limitation

The VolumeGroupSnapshot implementation can be tested as a ‘Tech Preview’.  Currently, only dynamic provisioning of VolumeGroupSnapshots is supported.

Kubernetes user flow

A. Creating the VolumeGroupSnapshotClass Resource

Kubernetes users first create a ‘VolumeGroupSnapshotClass’ resource. This resource defines InfiniBox security credentials to use when creating the snapshots.  InfiniBox CSI Driver implementation also defines the name of the consistency group (CG) that is created or used for creating snapshot groups (SGs).  A CG is an InfiniBox construct under which volume snapshots are organized.

Sample VolumeGroupSnapshotClass Resource:

apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
kind: VolumeGroupSnapshotClass
metadata:
  name: infinibox-groupsnapclass
deletionPolicy: Delete
driver: infinibox-csi-driver
parameters:
  csi.storage.k8s.io/group-snapshotter-secret-name: infinibox-creds
  csi.storage.k8s.io/group-snapshotter-secret-namespace: infinidat-csi
  infinidat.com/cgname: mycg

In the ‘infinidat.com/cgname’ parameter, specify the name of the consistency group to be created in InfiniBox.

B. Applying labels to PVCs

Next, add labels to PVCs that will construct the SG.  This is how Kubernetes VolumeGroups identifies the volumes to include in the volume snapshot group.

In the following example, two PVCs are labeled with a unique value that will be used to identify them as PVCs to be included in the VolumeGroupSnapshot. Two snapshots will be created as a result.

kubectl label pvc iscsi-pvc iscsi-pvc-anno app.kubernetes.io/name=mygroup

C. Creating VolumeGroupSnapshot Resource(s)

Next, create VolumeGroupSnapshot resources so that a snapshot group will be created from all the volumes (CG) with the relevant label. 

Sample VolumeGroupSnapshot resource:

apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
kind: VolumeGroupSnapshot
metadata:
  name: mygroup-groupsnapshot
spec:
  source:
    selector:
      matchLabels:
        # The PVCs will need to have this label for it to be
        # included in the VolumeGroupSnapshot
        app.kubernetes.io/name: mygroup
  volumeGroupSnapshotClassName: infinibox-groupsnapclass

You can view the status of the VolumeGroupSnapshot.

$ kubectl get volumegroupsnapshot
NAME                     READYTOUSE   VOLUMEGROUPSNAPSHOTCLASS           VOLUMEGROUPSNAPSHOTCONTENT                              CREATIONTIME   AGE
mygroup-groupsnapshot    true         infinibox-groupsnapclass           groupsnapcontent-42f9b0a1-a9b5-441b-9297-fe1483afcc4e   24h            25h
mygroup-groupsnapshot2   true         infinibox-groupsnapclass           groupsnapcontent-f3a11bd7-707f-46cd-9e1c-87bc35ab5525   21h            21h

If the snapshots are available for use, the Ready To Use status will be ‘true’.

Kubernetes installation prerequisites

For the VolumeGroupSnapshot feature to work in Kubernetes, enable the feature in the external-snapshotter CSI sidecar:

 - name: snapshot-controller
          image: registry.k8s.io/sig-storage/snapshot-controller:v8.0.1
          args:
            - "--v=5"
            - "--leader-election=true"
            - "--enable-volume-group-snapshots=true"

Install the required CRDs when you install the snapshot controller:

- snapshot.storage.k8s.io_volumesnapshotclasses.yaml
  - snapshot.storage.k8s.io_volumesnapshotcontents.yaml
  - snapshot.storage.k8s.io_volumesnapshots.yaml
  - groupsnapshot.storage.k8s.io_volumegroupsnapshotclasses.yaml
  - groupsnapshot.storage.k8s.io_volumegroupsnapshotcontents.yaml
  - groupsnapshot.storage.k8s.io_volumegroupsnapshots.yaml

To enable the VolumeGroupSnapshot feature, configure the InfiniBox CSI Driver in the Helm chart’s controllerserver.yaml template file:

  - name: snapshotter
          image: {{ required "csi snapshotter sidecar container image." .Values.images.snapshottersidecar }}
          imagePullPolicy: {{ .Values.images.snapshottersidecar_pull_policy | default "Always" | quote }}
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
            - "--extra-create-metadata=true"
            - "--enable-volume-group-snapshots=true"
            - "--snapshot-name-prefix={{ required "Must provide a value to prefix to driver created snapshot names" .Values.volumeNamePrefix }}"
            - "--snapshot-name-uuid-length=10"

The VolumeGroupSnapshot feature is only available through the Technical Preview feature, which must be enabled on the cluster. 

InfiniBox 'snapshot promote' feature support

Introduction

Starting from version 2.25.0, InfiniBox CSI Driver supports the 'snapshot promote' feature that promotes an InfiniBox snapshot to a master volume.

Promoting a snapshot

InfiniBox 'snapshot promote' support is provided by the 'Iboxpromote' Custom Resource and the associated 'Iboxpromote-controller'. The 'Iboxpromote-controller' operates as a sidecar within the existing InfiniBox CSI Driver.

Example of a CR structure in YAML format:

apiVersion: csidriver.infinidat.com/v1
kind: Iboxpromote
metadata:
  annotations:
    infinidat.com/secret_name: infinibox-creds
    infinidat.com/secret_namespace: infinidat-csi
  name: iboxpromote-snapshot-test
spec:
  description: "test snapshot promote"  # user defined description
  
  # SNAPSHOT can be the InfiniBox Snapshot name ("SNAPSHOT"), or the Kubernetes SNAPSHOT ("KUBE_SNAPSHOT") name:
  entity_type: "SNAPSHOT"               # valid values: SNAPSHOT, KUBE_SNAPSHOT

  entity_name: "example-snapshot"    # the name of a snapshot to be promoted
  base_action: "NEW"                    # valid values: NEW

InfiniBox replication support

Introduction

InfiniBox CSI Driver supports different types of replication:

• Active-Active
• Synchronous
• Asynchronous

Replication support is provided by the 'iboxreplica' Custom Resource and the associated 'iboxreplica-controller'. The 'iboxreplica-controller' operates as a sidecar within the existing InfiniBox CSI Driver.

• The iboxreplica Custom Resource allows Kubernetes users to create and manage replicas of volumes, filesystems, and consistency groups.
• The iboxreplica-controller manages replication events and updates the status of the replicas in response to changes.

Creating a replica

Ensure that an 'iboxreplica' Custom Resource (CR) is defined before creating a replica.

Active-Active replication

Example of a CR structure in YAML format:

apiVersion: csidriver.infinidat.com/v1
kind: Iboxreplica
metadata:
  annotations:
    infinidat.com/secret_name: infinibox-creds
    infinidat.com/secret_namespace: infinidat-csi
  name: iboxreplica-aa-volume-sample
spec:
  is_preferred: true                        # set only for ACTIVE_ACTIVE, true = local system, false = the remote system
  description: "aa-volume-iboxreplica-test" # user defined description
  entity_type: "VOLUME"                     # valid values can be VOLUME or CG
  local_entity_name: "test-volume" # this is the volume name to be replicated
  replication_type: "ACTIVE_ACTIVE"         # valid values can be ACTIVE_ACTIVE or ASYNC
  link_remote_system_name: "remote_ibox_name" #the replication link name
  remote_pool_id: 12345        # the pool ID on the remote ibox

  # New for InfiniBox CSI Driver version 2.26.0
  
  remote_ibox_credential_name: "ibox_name"
  remote_ibox_credential_namespace: "infinidat-csi"
  remote_create_pvc: true
  remote_pvc_name_suffix: "-box-ht23"
  remote_pvc_namespace: infinidat-csi
  remote_pool_name: csitesting2

Synchronous replication

Example of a CR structure in YAML format:

apiVersion: csidriver.infinidat.com/v1
kind: Iboxreplica
metadata:
  annotations:
    infinidat.com/secret_name: infinibox-creds
    infinidat.com/secret_namespace: infinidat-csi
  name: iboxreplica-sync-volume
spec:
  description: "sync-volume-iboxreplica-test"  # user defined description
  entity_type: "VOLUME"                         # valid values are VOLUME, CONSISTENCY_GROUP, FILESYSTEM
  local_entity_name: $_E2E_IBOXREPLICA_VOLUME   # the volume name of the volume to be replicated
  remote_entity_name: $_E2E_IBOXREPLICA_VOLUME   # the remote (target) volume name of the volume to be replicated
  replication_type: "SYNC"                     # valid values can be SYNC, ASYNC or ACTIVE_ACTIVE
  base_action: "NEW"                            # value value is NEW
  link_remote_system_name: $_E2E_IBOXREPLICA_LINK_REMOTE_SYSTEM_NAME # the replication link name on the InfiniBox
  remote_pool_id: $_E2E_IBOXREPLICA_REMOTE_POOL_ID # the pool ID on the target InfiniBox

Asynchronous replication

Example of a CR structure in YAML format:

apiVersion: csidriver.infinidat.com/v1
kind: iboxreplica
metadata:
  annotations:
infinidat.com/secret_name: infinibox-creds
infinidat.com/secret_namespace: infinidat-csi
  name: iboxreplica-cg-sample
spec:
  sync_interval: 240000
  description: "mycgreplicatest"
  entity_type: "CONSISTENCY_GROUP"
  local_entity_name: "test-cg"
  replication_type: "ASYNC"
  base_action: "NEW"
  link_remote_system_name: "remote_ibox_name"
  rpo_value: 300000
  remote_pool_id: 12345

Required fields

Automatic replication within StorageClass

Configuring volume replication

To configure volume replication, the Kubernetes user specifies the following parameters in their StorageClass:

replication_type: "SYNC" # "SYNC" / "ASYNC" / "ACTIVE_ACTIVE"  
link_remote_system_name: "myremoteiboxlink" 
remote_pool_id: "123" 
local_ibox_credential_name: "infinibox-creds" 
local_ibox_credential_namespace: "infinidat-csi"
remote_ibox_credential_name:  "remote-ibox-secret"
remote_ibox_credential_namespace: "infinidat-csi"
remote_kubeconfig_secret_name: "ALTERNATE_KUBE_CLUSTER"
remote_kubeconfig_secret_namespace: "infinidat-csi"
remote_create_pvc: "true"

In the above example, the required parameters for this feature are as follows:

• 'replication_type' is the replication type to use when creating the replica on the InfiniBox. Possible values are "SYNC", "ASYNC", and "ACTIVE_ACTIVE".
• 'link_remote_system_name' is the link name on the InfiniBox. It represents a pre-defined InfiniBox replication link to another InfiniBox.
• 'remote_pool_id' is the internal InfiniBox ID of the pool on the target InfiniBox.
• 'local_ibox_credential_name' is the name of the Kube Secret that holds the local InfiniBox credentials.
• 'local_ibox_credential_namespace' is the namespace of the Kube Secret.

• 'remote_kubeconfig_secret_name' is the name of the Kube Secret that holds the alternate Kube Cluster kubeconfig.

• 'remote_kubeconfig_secret_namespace' is the namespace of the alternate Kube Cluster Secret. 

• 'remote_ibox_credential_name' is the name of the Kube Secret that holds the remote InfiniBox credentials.

• 'remote_ibox_credentials_namespace' is the namespace of the Kube Secret for the remote InfiniBox.

• 'remote_create_pvc' is the flag for activating this automatic 'pvc' creation.

Configuring CG Replication

To configure CG replication, specify a ConsistencyGroup within a StorageClass. In this case, it means that any new volume that is created will be added to an existing ConsistencyGroup on InfiniBox. 

In this case, specify the following parameters within the StorageClass:  

replication_type: "ACTIVE_ACTIVE" # "SYNC" / "ASYNC" / "ACTIVE_ACTIVE" 
link_remote_system_name: "myremoteiboxlink" 
remote_pool_id: "123" 
cg_name: "testcg"   # the CG name
local_ibox_credential_name: "infinibox-creds" 
local_ibox_credential_namespace: "infinidat-csi"

In the above example, the required parameters for this feature are as follows:

1. 'replication_type' is the replication type to use when creating the replica on the InfiniBox. Possible values are "SYNC", "ASYNC", and "ACTIVE_ACTIVE".
2. 'link_remote_system_name' is the link name on the InfiniBox. It represents a pre-defined InfiniBox replication link to another InfiniBox.
3. 'remote_pool_id' is the internal InfiniBox ID of the pool on the target InfiniBox.
4. 'cg_name' is the name of an existing ConsistencyGroup on InfiniBox.
5. 'local_ibox_credential_name' is the name of the Kube Secret that holds the local InfiniBox credentials.
6. 'local_ibox_credential_namespace' is the namespace of the Kube Secret.

Configuring PV/PVC Creation for Replicas

For the InfiniBox CSI Driver to create PV/PVC for newly created volumes, specify the following parameters in the StorageClass:

remote_ibox_credential_name: remote-ibox-credentials
remote_ibox_credential_namespace: infinidat-csi
remote_create_pvc: "true"
remote_pvc_name_suffix: "-remote-ibox"
remote_pvc_namespace: infinidat-csi
remote_network_space: default_iscsi_space
remote_pool_name: myremoteiboxpool

In the above example, the required parameters for this feature are as follows:

1. 'remote_ibox_credential_name' is the Kube Secret that holds the credential for the target InfiniBox.
2. 'remote_ibox_credential_namespace' is the Namespace of the Kube Secret for the target InfiniBox.
3. 'remote_create_pvc'  is a boolean value. To enable this feature, set it to 'true'.
4. 'remote_pvc_name_suffix' is a user-defined suffix that will be appended to newly created PVCs for the remote volume.
5. 'remote_pvc_namespace' is the Kube Namespace to create the PVC within on the Kube cluster.
6. 'remote_network_space' is the target InfiniBox 'Network Space' that will be encoded in the remote volume PV.
7. 'remote_pool_name' is the target InfiniBox pool name that will be encoded in the target volume PV.

Viewing the replica status

To check the status of an iboxreplica, run the following command:

kubectl describe iboxreplica/iboxreplica-cg-sample

Name:         iboxreplica-cg-sample
Namespace:    
Labels:   	<none>
Annotations:  infinidat.com/secret_name: infinibox-creds
          	infinidat.com/secret_namespace: infinidat-csi
API Version:  csidriver.infinidat.com/v1
Kind:         iboxreplica
Metadata:
  Creation Timestamp:  2024-09-24T15:30:39Z
  Finalizers:
	infinidat.com/iboxreplica
  Generation:    	1
  Resource Version:  635050
  UID:           	b6799ea8-538a-42e8-bce1-e2c73e29767d
Spec:
  base_action:          	NEW
  Description:              mycgreplicatest
  entity_type:          	CONSISTENCY_GROUP
  link_remote_system_name:  box-ht12
  local_entity_name:    	ocp414-408ee5819f
  remote_pool_id:       	65993
  replication_type:     	ASYNC
  rpo_value:            	300000
  sync_interval:        	240000
Status:
  Id: 	1730
  State:  ACTIVE
Events:   <none>

To view the replication controller log, run the following command:

kubectl logs infinidat-csi-driver-driver-0 -c iboxreplica

Deleting a Replica

To delete a replica, run the following command:

Kubectl delete iboxreplica/iboxreplica-cg-sample

This command also deletes the associated replication entity on InfiniBox.

InfiniBox Consistency Group support

Introduction

InfiniBox CSI Driver supports the grouping of volumes within one group. This entity is called a Consistency Group (CG).

A CG, and all of the volumes that reside in it, can be replicated in the following ways:

• Active-Active
• Synchronous
• Asynchronous

Replication support and CG replication support are highlighted in the section above. 

Consistency Group Resource

Example of a CR structure in YAML format:

apiVersion: csidriver.infinidat.com/v1
kind: Iboxcg
metadata:
  annotations:
    infinidat.com/secret_name: infinibox-creds
    infinidat.com/secret_namespace: infinidat-csi
  name: iboxcg-add-test
spec:
  description: "test add member to cg"  # user defined description
  local_cg_name: "CG_Name"
  local_volume_name: "Volume_Name"
  base_action: "ADD"                    # valid values: ADD, REMOVE