General Configuration

Ngenea HSM requires the following components to correctly function:

  • a modified GPFS file system policy;
  • a master Ngenea HSM configuration file with at least one storage target; and
  • a configuration file for each storage target.

The following sections describe how to apply the above pre-requesites.

GPFS/Spectrum Scale Configuration

Transparent Recall file system policy

In order to enable transparent recall on a file system, insert the transparent recall rules provided by ArcaStream / Pixit Media at the beginning of the current file system policy.

Note that if no current policy is in place, you will also need to include a default placement policy rule at the end of the policy, ensuring that new files are written to the correct GPFS storage pool. This is not required if you only have a system GPFS storage pool. For example:

/* By default place all data on the sata1 pool */
RULE 'default' SET POOL 'sas1'

Ensure that any future alterations to the GPFS file system policy do not replace, modify or precede any of the Ngenea HSM policy.

Cluster Configuration

In order to provide the maximum number of concurrent transparent recall and migration threads, set the following GPFS cluster configuration parameters on any node which is expected to issue transparent recall requests.

policySystemEvalLimit 64
dmapiWorkerThreads 64

Also, ensure that workerThreads is set to greater than 64:

workerThreads 128

Master Configuration File

The master configuration file defines the available storage targets, and directs migration and recall requests to the appropriate target.

The default location for Ngenea HSM configuration files is within /opt/arcapix/etc/, and the default location for the master configuration file is /opt/arcapix/etc/ngenea.conf.

Any storage configuration files are relative to the location of the master configuration file.

Below is an example configuration file with two storage targets defined.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
[Storage bpearl1]
StorageType=BlackPearl
ConfigFile=blackpearl1.conf
LocalFileRegex=/mmfs1/(archive/.+)
RemoteLocationXAttrRegex=blackpearl:(.+)

[Storage awss3]
StorageType=AmazonS3
ConfigFile=awss3.conf
LocalFileRegex=/mmfs1/(active/.+)
RemoteLocationXAttrRegex=awss3:(.+)

Section Keywords

Storage

The Storage section defines how storage targets will be used for migrating and recalling data. Each storage target definition includes a name, type, scope of operation and a reference to the Storage Target configuration file.

Syntax:

[Storage <name>]

Example:

[Storage ngenea-target1]

ConfigFile

Defines the location of the associated configuration file for the Storage entry, either specified as the full path and filename, or path and filename relative to the location of this master configuration file.

Syntax:

ConfigFile=<filename>

Example:

ConfigFile=/opt/arcapix/etc/ngenea-target1.conf

LocalFileRegex

Defines the pathname match, in regex, used to determine which filesystem paths will trigger migrations to this storage target. It also controls the generation of the file path used in the remote storage.

Syntax:

LocalFileRegex=/<mountpoint>/([path filter].+)

The following example will cause this storage target to be used as a migration target for all files on file system /mmfs1/, and store the files in remote storage with their full path minus the /mmfs1/ component:

Example: all filesystem objects can be migrated to this target

LocalFileRegex=/mmfs1/(.+)

The following example will cause this storage target to be used as a migration target for all files underneath /mmfs1/data/fileset1/, and store the files in remote storage with the path data/fileset1/path/to/file:

Example: targeted migration

LocalFileRegex=/mmfs1/(data/fileset1/.+)

RemoteLocationXAttrRegex

Defines the APXrmtlc extended attribute regular expression match. This extended attribute is added to a file upon migration. Any file with the APXrmtlc extended attribute matching this regex will be downloaded from this target when issuing ngrecall:

Syntax:

RemoteLocationXAttrRegex=<identifier>:(.+)

Example:

RemoteLocationXAttrRegex=ngenea-target1:(.+)

StorageType

Defines the supported mode of storage operation. Available options are:

  • BlackPearl - SpectraLogic Black Pearl Spectra S3 target
  • AmazonS3 - Amazon S3, or compatible, protocol target
  • Azure - Microsoft Azure blob storage target
  • FS - POSIX file system target

Syntax:

StorageType=<AmazonS3|BlackPearl|Azure|FS>

Example:

StorageType=AmazonS3