Follow

Limiting the filesystem capacity and number of inodes

InfiniBox allows imposing capacity limits on the filesystem tree level. The limit can be either on the capacity of the tree or on the number of inodes, or on both. The TreeQ feature provides the storage admin with means for controlling the usage of system resources (represented by trees).

InfiniBox multiple levels of capacity and operational provisioining:

  • Pools - the InfiniBox system is divided into pools - storage units that can impose capacity limits on the filesystems (and volumes) that are in them
  • Filesystems - the filesystem stores many files and deploys protection schemes (snapshots and replica) on the filesystem unit
  • TreeQ - a filesystem subtree (or a folder) that can be assigned with a quota on the number of files or the capacity. A single filesystem can contain thousands of TreeQs

For ease of operation, INFINIDAT recommends to create a small number of filesystems, keeping in mind that the number of files within a single filesystem is practically unlimited and a high file count does not incur a performance impact on InfiniBox.

Examples of quotas

The quota can be imposed on either the capacity or number of inodes, or both.

  • 2GB capacity
  • 1,000 inodes
  • 1GB capacity and 1,200 inodes (in such a case, the first hard quota to be reached, blocks the subtree IO. See below for more details)

Snapshots tree quota

  • Writable snapshots inherit their TreeQ definitions from the parent filesystem. These definitions can be overriden
  • Restoring a filesystem from a snapshot restores the TreeQ and their definitions to the time the snapshot was created

Target filesystems

  • A filesystem that is a target of a replica inherits the TreeQ definitions from the source filesystem. These definitions cannot be modified on the target, as long as it is part of the replica.
  • When a quota is reached an event is generated on the source side only 

As symbolic links are small enough to be stored within the filesystem metadata structure, they do not require an actual file. A symbolic link quota is calculated as follows:

  • 1 inode
  • 0 capacity

While hard links share the inode structure (changing the attributes of one link affect other links), they consume some small metadata structure in the filesystem namespace. Therefore, InfiniBox counts them as an inode. A hard link quota is calculated as follows:

  • 1 inode
  • 0 capacity

Limits

  • The TreeQ can have a capacity of 500MB - 10,000 PB
  • The TreeQ can have up to 100*108 inodes
  • The number of TreeQ per filesystem is 4,095 
  • Size limits:
    • Hard capacity >= 1GB (1*109not GiB)
    • Hard capacity - Soft capacity >= 100MB (100*106, not MiB)
    • Soft inode > 1
    • Soft capacity >= 500 MB (500*106not MiB)

Types of quotas

Available quota types

The quota that is associated with a filesystem tree comes in two types:

  • Soft quota - A threshold for alerting the storage administrator of high quota usage, without blocking writes. This allows the administrator time to respond before writes will be rejected 
    • When the soft quota is exceeded: 
      • An event is generated within a 50MB window above the quota
      • The tree keeps accepting IO
  • Hard quota - a threshold that blocks IO when reached.
    • When the hard quota is reached: 
      • An event is generated when the a user tries to reach above the assigned quota (inodes, Capacity)
        • Events are only generated once per hour per TreeQ hard quota
        • The event may be triggered within a 1MB window below the quota
      • For capacity quota: the tree ceases accepting writes. Read and delete operations are accepted 
      • For inodes quota: the TreeQ ceases accepting creates (while writes to existing inodes are not ceased)

Implementation of both quota types

  • When both soft quota and hard quota are implemented all of the above behavior is valid, with the exception of a case when both quota types are defined. Both soft and hard quota can be reached within an atomic user operation (for example, when a file is moved into a TreeQ). In such a case, the event for crossing the soft quota is not generated.
    • This behavior is applicable for either capacity or file count.
  • As a convention, as the soft quota incurs softer measures than the hard quota (generating an event vs. blocking I/O), the soft quota is always smaller than the hard quota.

Inapplicability of TreeQ for some filesystems

Note that the TreeQ feature is not applicable for:

  • Write-protected filesystems
  • Locked filesystems

To enable TreeQ for such filesystems, they need to get out of these states (i.e. write-enabled, or move them to a non-locked pool).

Quota states

Each of the TreeQ's quotas (Capacity / inodes) is has one of the following states:

  • NO_QUOTA - when no quota was set by the admin (the system will only count usage but won't send events / block the user from writing)
  • BELOW_QUOTA - the soft limit was not exceeded yet, users can write and no event will be triggered
  • SOFT_EXCEEDED - the TreeQ has exceeded its soft quota but has not reached the hard limit (user can still write data)

Transition between states

  • The soft quota threshold is always crossed before the hard quota threshold
    • The soft and hard quota can be crossed together, as discussed above
  • The filesystem state returns to BELOW_QUOTA when the capacity (or number of inodes) decrease

Hysteresis - decreasing the number of generated events

As the capacity of the filesystem subtree may frequently change and repeatedly keep crossing (or reaching) a threshold, InfiniBox implements a mechanism for decreasing the number of generated events.

Crossing the soft quota threshold

The following image provides examples for crossing the soft quota:

  1. Both soft quota and hysteresis thresholds were crossed, so an event is generated
  2. The capacity is above the soft quota, so when new capacity was added, the soft quota threshold was not crossed. No event is generated
  3. The capacity moves from below the soft quota to above the soft quota, but the hysteresis threshold was not crossed. No event is generated
  4. Both soft quota and hysteresis limit were crossed so an event is generated

Crossing the hard quota threshold

The event generation in a case where the hard quota is reached depends on the time that has passed since the last event. The default time interval is 60 minutes. 

CaseTime that has passed since the last eventEvent is generated?Writes / file creation blocked?
11st timeyesyes
2< configurable timenoyes
3> configurable timeyesyes


Writing to a nearly full TreeQ

When creating a file and writing into it while the TreeQ is nearly full, the size of the file may exceed the TreeQ size. In such a case, the file will be created but the write operation will fail.

Note that as the TreeQ is smaller than the filesystem, the file can be moved to another TreeQ and remains writable.

Moving hard links between TreeQs

Hard links crossing TreeQs are not allowed. When moving a hard link to another TreeQ, the Linux "mv" command will fail back to copy and remove operations instead.

Moving files across TreeQ boundaries

Moving files from one TreeQ to another undergoes as follows:

  1. The directory structure is recreated as it cannot be copied
  2. The files are moved to the new directory
  3. The old directory - now empty from files - is deleted

Was this article helpful?
0 out of 0 found this helpful

0 out of 0 found this helpful

Comments