Ngenea HSM requires the following components to correctly function:
- a modified GPFS file system policy;
- a master Ngenea HSM configuration file with at least one storage target; and
- a configuration file for each storage target.
The following sections describe how to apply the above pre-requesites.
GPFS/Spectrum Scale Configuration¶
Transparent Recall file system policy¶
In order to enable transparent recall on a file system, insert the transparent recall rules provided by ArcaStream / Pixit Media at the beginning of the current file system policy.
Note that if no current policy is in place, you will also need to include a default placement policy rule at the end
of the policy, ensuring that new files are written to the correct GPFS storage pool. This is not required if you
only have a
system GPFS storage pool. For example:
/* By default place all data on the sata1 pool */ RULE 'default' SET POOL 'sas1'
Ensure that any future alterations to the GPFS file system policy do not replace, modify or precede any of the Ngenea HSM policy.
In order to provide the maximum number of concurrent transparent recall and migration threads, set the following GPFS cluster configuration parameters on any node which is expected to issue transparent recall requests.
policySystemEvalLimit 64 dmapiWorkerThreads 64
Also, ensure that workerThreads is set to greater than 64:
Master Configuration File¶
The master configuration file defines the available storage targets, and directs migration and recall requests to the appropriate target.
The default location for Ngenea HSM configuration files is within
/opt/arcapix/etc/, and the
default location for the master configuration file is
Any storage configuration files are relative to the location of the master configuration file.
Below is an example configuration file with two storage targets defined.
1 2 3 4 5 6 7 8 9 10 11
[Storage bpearl1] StorageType=BlackPearl ConfigFile=blackpearl1.conf LocalFileRegex=/mmfs1/(archive/.+) RemoteLocationXAttrRegex=blackpearl:(.+) [Storage awss3] StorageType=AmazonS3 ConfigFile=awss3.conf LocalFileRegex=/mmfs1/(active/.+) RemoteLocationXAttrRegex=awss3:(.+)
The Storage section defines how storage targets will be used for migrating and recalling data. Each storage target definition includes a name, type, scope of operation and a reference to the Storage Target configuration file.
Defines the location of the associated configuration file for the Storage entry, either specified as the full path and filename, or path and filename relative to the location of this master configuration file.
Defines the pathname match, in regex, used to determine which filesystem paths will trigger migrations to this storage target. It also controls the generation of the file path used in the remote storage.
The following example will cause this storage target to be used as a migration target for all files on file system
/mmfs1/, and store
the files in remote storage with their full path minus the
Example: all filesystem objects can be migrated to this target
The following example will cause this storage target to be used as a migration target for all files underneath
/mmfs1/data/fileset1/, and store the
files in remote storage with the path
Example: targeted migration
Defines the APXrmtlc extended attribute regular expression match. This extended attribute is added to a file upon migration. Any file with the APXrmtlc extended attribute matching this regex will be downloaded from this target when issuing ngrecall:
Defines the supported mode of storage operation. Available options are:
BlackPearl- SpectraLogic Black Pearl Spectra S3 target
AmazonS3- Amazon S3, or compatible, protocol target
Azure- Microsoft Azure blob storage target
FS- POSIX file system target