Ngenea HSM requires the following components to correctly function:
- a modified GPFS file system policy
- a master Ngenea HSM configuration file with at least one storage target
- a configuration file for each storage target
- active accounts with the appropriate cloud storage providers
The following sections describe how to apply the above prerequisites.
GPFS/Spectrum Scale Configuration¶
Transparent Recall file system policy¶
In order to enable transparent recall on a file system, insert the transparent recall rules provided by ArcaStream / Pixit Media at the beginning of the current file system policy.
Note that if no current policy is in place, you will also need to include a default placement policy rule at the end
of the policy, ensuring that new files are written to the correct GPFS storage pool. This is not required if you
only have a
system GPFS storage pool. For example:
/* By default place all data on the sata1 pool */ RULE 'default' SET POOL 'sas1'
Ensure that any future alterations to the GPFS file system policy do not replace, modify or precede any of the Ngenea HSM policy.
In order to provide the maximum number of concurrent transparent recall and migration threads, set the following GPFS cluster configuration parameters on any node which is expected to issue transparent recall requests.
policySystemEvalLimit 64 dmapiWorkerThreads 64
Also, ensure that workerThreads is set to greater than 64:
Master Configuration File¶
The master configuration file defines the available storage targets, and directs migration and recall requests to the appropriate target.
The default location for Ngenea HSM configuration files is within
/opt/arcapix/etc/, and the
default location for the master configuration file is
Any storage configuration files are relative to the location of the master configuration file.
Below is an example configuration file with two storage targets defined.
1 2 3 4 5 6 7 8 9 10 11
[Storage bpearl1] StorageType=BlackPearl ConfigFile=blackpearl1.conf LocalFileRegex=/mmfs1/(archive/.+) RemoteLocationXAttrRegex=blackpearl:(.+) [Storage awss3] StorageType=AmazonS3 ConfigFile=awss3.conf LocalFileRegex=/mmfs1/(active/.+) RemoteLocationXAttrRegex=awss3:(.+)
|[Storage <storage target name>]|
|[FileMatch <file match set>]|
This optional section specifies general parameters for all storage endpoints.
Section [General] Keywords¶
Specifies the minimum number of successful migrations required in order to report success for a migration to multiple storage endpoints. See Multi-Target Support for an overview of multi-target configuration and operation.
The default value is 0.
MinMigrateEndpointCount=<Minimum number of successful migrations>
Specifies the maximum number of local files to migrate, recall, or delete remote objects for in one batch. If there are more local files to process, ngmigrate or ngrecall automatically creates subsequent batches.
For file recall and remote object deletion operations, ngrecall automatically adjusts the value of this parameter to not exceed the value of "MaxTransparentRecallLockCount" parameter.
The default value is 512.
MaxJobPartFileCount=<Maximum number of files per batch>
Specifies the maximum number of concurrent locks put on distinct local files accessed via DMAPI in file recall and remote object deletion operations.
To minimize the number of such errors, ngrecall automatically adjusts the maximum number of local files to recall or delete remote objects for in one batch specified by the "MaxJobPartFileCount" parameter to not exceed the value of "MaxTransparentRecallLockCount" parameter.
The default value is 19.
MaxTransparentRecallLockCount=<Maximum number of concurrent locks>
Section [Storage <storage target name>]¶
The Storage section defines how a storage target will be used for migrating and recalling data. Each storage target definition includes a name, type, scope of operation and a reference to the Storage Target configuration file.
[Storage <storage target name>]
Section [Storage <storage target name>] Keywords¶
Defines the location of the associated configuration file for the Storage entry, either specified as the full path and filename, or path and filename relative to the location of this master configuration file.
Defines the pathname match, in regex, used to determine which filesystem paths will trigger migrations to this storage target. It also controls the generation of the file path used in the remote storage.
The following example will cause this storage target to be used as a migration target for all files on file system
/mmfs1/, and store
the files in remote storage with their full path minus the
Example: all filesystem objects can be migrated to this target
The following example will cause this storage target to be used as a migration target for all files underneath
/mmfs1/data/fileset1/, and store the
files in remote storage with the path
Example: targeted migration
Defines the APXrmtlc extended attribute regular expression match. This extended attribute is added to a file upon migration. Any file with the APXrmtlc extended attribute matching this regex will be downloaded from this target when issuing ngrecall:
Note that the regular expressions specified by LocalFileRegex and RemoteLocationXAttrRegex match to the beginning of the text. The regular expressions behave as if they are prefixed by the caret anchor
^ (match beginning of text).
Defines the supported mode of storage operation. Available options are:
BlackPearl- SpectraLogic Black Pearl Spectra S3 target
AmazonS3- Amazon S3, or compatible, protocol target
Azure- Microsoft Azure blob storage target
FS- POSIX file system target
Section [FileMatch <file match set>]¶
These sections specify stub sizes for various names and paths of local files.
FileMatch Section Keywords¶
|StubSize||-1 (ie premigrate)||No|
These sections specify stub sizes for various names and paths of local files.
The program ngmigrate applies those stub sizes to local files when migrating them. The program ngrecall applies those stub sizes to local files when reversely stubbing their remote objects.
The header of every file match section must contain the name of a file match set.
[FileMatch <file match set>]
Example: A file match section describing text files might have the header
The names of all file match sets described in a master configuration file must be different.
The programs ngmigrate and ngrecall process file match sections in the order they are present in a master configuration file. If a section matches a local file, ngmigrate or ngrecall applies the section to the file and does not process subsequent file match sections for that file.
The last file match section described in a master configuration file might specify the default stub size to be applied to a local file when no previous sections matched the file. For example, such default file match section might look like this:
Example: A default file match section, specifying a StubSize of 0 bytes
[FileMatch default] ; Use zero stub size for all other files. StubSize=0
Every file match section can contain the parameters "BasenameMask", "PathnameRegex", and "StubSize". The parameters "BasenameMask" and "PathnameRegex" specify conditions joined by logical AND.
Specifies a glob pattern for the base name part of a matched file name.
Separate with '|' (the pipe character) multiple glob patterns joined by logical OR. Example:
If this parameter is absent, all file names will be matched.
Specifies a regular expression for matching against a resolved absolute file name.
If this parameter is absent, all paths will be matched.
Specifies the stub size in bytes.
If the stub size exceeds the size of a file, or if the stub size is -1, ngmigrate or ngrecall will premigrate the file.
StubSize=<stub size in bytes (or -1)>
The default value is -1; i.e. the file will be premigrated.
Cloud Storage Requirements¶
The cloud storage providers have specific requirements for configuring access accounts and providing authentication. These requirements are described below for each storage provider.
Amazon S3 and S3 Compatible Storage¶
An administrator should create a blob storage account through the Azure Portal. A bucket should also be created. The user's authentication details and bucket name must be added to the Amazon S3 configuration file.
The user account should be configured to allow the following permissions for the resource identified by the bucket name:
The bucket should be configured to allow multipart upload - see Multipart Upload API and Permissions
Microsoft Azure Storage¶
A storage account and container should be created through the Azure Portal. The account name, account key, and container name will be copied to the Azure configuration file.