# Storage Target Configuration¶

Ngenea HSM supports Storage Targets compatible with the Amazon S3 protocol, Spectra Logic's BlackPearl Spectra S3 protocol (a superset of the Amazon S3 Protocol) and Microsoft Azure.

Storage Targets are referenced from the Master Configuration File, with configuration data for each target defined in a target-specific configuration file.

The default location for a Storage Target Configuration File is in /opt/arcapix/etc/.

## Native Amazon S3 Type¶

The following section describes how to setup a Storage Target to an S3 compatible storage target.

S3 Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified end point and bucket.

### Example Configuration File¶

[General]
Endpoint=endpoint.host.name
Port=443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
Region=eu-west-2
Scheme=HTTPS
RemoteLocationXAttr=ngenea-target1:$1 RetrieveObjectName=$1
StoreObjectName=$1 DeleteOnRecall=True ### Configuration Keywords¶ #### Endpoint¶ For Amazon AWS S3 targets, do not include this parameter. It will be automatically determined by Amazon based on the region and bucket name. For non-Amazon AWS S3 targets, this parameter defines the Hostname or IP Address whereby Ngenea HSM can access the S3 target. Syntax Endpoint=<Hostname or IP address> Example Endpoint=s3.host.domain #### Port¶ For Amazon AWS S3 targets, do not include this parameter. For non-Amazon AWS S3 targets, this parameter defines the tcp port to connect to on the remote storage. The default value is 443 if this parameter is unspecified. Syntax Port:<TCP port number> Example Port:443 #### AccessKeyId¶ Defines the Account Access Key used to authenticate against the S3 target. For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation. Syntax AccessKeyId=<Amazon Account Access Key> Example AccessKeyId=ABCDEFGHIJLKMNOPQRST #### SecretAccessKey¶ Defines the Secret Key used to authenticate against the S3 target. For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation. Syntax SecretAccessKey=<Amazon Secret Access Key> Example SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm #### Bucket¶ Defines the bucket, or 'container', into which files are migrated. This bucket must exist on the S3 storage target prior to migrating data using Ngenea HSM. For further information regarding Amazon AWS Buckets please refer to the Amazon Documentation. Syntax Bucket=<Name of the S3 bucket> Example Bucket=MyBucketName #### Region¶ Defines the regional endpoint in use for the target bucket. For further information regarding Amazon AWS Regions please refer to the Amazon Documentation. Other S3 targets may have significantly different requirements for how to specify this parameter. Consult your S3 target documentation for further information. Syntax Region=<Name of the S3 Region> Example Region=eu-west-2 #### Scheme¶ Defines the transport protocol used to communicate with the S3 target. ArcaPix only supports HTTPS protocol for Ngenea HSM solutions. Syntax Scheme=<PROTOCOL> Example Scheme=HTTPS #### RemoteLocationXAttr¶ Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file.$1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=ngenea-target1:$1 #### RetrieveObjectName¶ Defines the mapping from the local APXrmtlc to the associated file on the remote storage target.$1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=<RemoteLocationXAttrReference>

Example

RetrieveObjectName=$1 #### StoreObjectName¶ Defines the expression by which the remote filename is composed when migrating to the external storage.$1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

This setting would typically match the RetrieveObjectName above.

Syntax

StoreObjectName=<LocalFileRegexReference>

Example

StoreObjectName=$1 #### MaxUploadSize¶ Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server. This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the S3 server into one object. Set this value to be less than or equal to that supported by the remote storage target. Valid settings are between 5242880 (5MB) and 5368709120 (5GB). This optional parameter defaults to 5GB if not explicitly defined. Syntax MaxUploadSize=<size_in_bytes> Example MaxUploadSize=5368709120 #### DeleteOnRecall¶ Defines whether to delete the data from the remote S3 target when successfully recalled back to the file system. Setting this to False will result in files entering a "premigrated" state on successful recall. Syntax DeleteOnRecall=<True|False> Example DeleteOnRecall=True ## Filesystem Target¶ The following section describes how to setup a Storage Target to a POSIX-compatible mounted file system. The target file system must be mounted on all nodes which will migrate or recall files. ### Example Configuration File¶ [General] StoreObjectName=/mnt/targetfs/$1
RetrieveObjectName=/mnt/targetfs/$1 RemoteLocationXAttr=ngenea-targetfs1:$1
DeleteOnRecall=False
ObjectXAttrManipulationMode=auto
EnsureMountPoint=/mnt/targetfs

### Configuration Keywords¶

#### StoreObjectName¶

Defines the value prepended to the path and filename on migration, which will usually point to the target mount point. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file. Syntax StoreObjectName=/mountpoint/<LocalFileRegexReference> Example StoreObjectName=/mnt/targetfs/$1

#### RetrieveObjectName¶

Defines the value prepended to the path and filename on recall, which will usually point to the target mount point and match the value of StoreObjectName. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file. Syntax RetrieveObjectName=/mountpoint/<LocalFileRegexReference> Example RetrieveObjectName=/mnt/targetfs/$1

#### RemoteLocationXAttr¶

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file. Syntax RemoteLocationXAttr=targetref:<LocalFileRegexReference> Example RemoteLocationXAttr=ngenea-targetfs1:$1

#### DeleteOnRecall¶

Defines whether to immediately delete the data from the POSIX mount target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

#### ObjectXAttrManipulationMode¶

Defines the behaviour for storing the metadata of files migrated to the POSIX mount target.

When native mode is enabled, metadata will be stored as extended attributes on migrated files. If the target POSIX mount does not support extended attributes, then migration and recall will abort with an error.

When shadow mode is enabled, metadata will be stored in an accompanying file which is stored alongside the data file.

Setting this to auto (the default) will result in the ngmigrate process attempting to store metadata in extended attributes, and if that fails it will store the metadata in an accompanying file.

Syntax

Example

ObjectXAttrManipulationMode=auto

#### EnsureMountPoint¶

Specifies the mount point that ngmigrate and ngrecall processes will validate prior to migrating or recalling a file.

Should the specified location not be the root of a mount point, the migration or recall will abort with an error.

This is an important safety feature to ensure that the local disk is not used in the event of a target mount failing.

It is typically set to the root of the mount point.

Syntax

EnsureMountPoint=/mountpoint

Example

EnsureMountPoint=/mnt/targetfs

## Spectra Logic BlackPearl Target¶

ArcaPix's Ngenea HSM software is fully certified by Spectra Logic as an approved BlackPearl client.

Ngenea HSM requires configuration and activation of the BlackPearl's Spectra S3 Service. Please refer to the relevant section of the BlackPearl User Guide.

The following example describes the information required to be present withing the Storage Target Configuration File to enable migration to the specified bucket.

### Example Configuration File¶

[General]
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
RemoteLocationXAttr=blackpearl:$1 RetrieveObjectName=$1
StoreObjectName=$1 CacheFullMaxWaitSeconds=300 CacheFullRetryAfterSeconds=20 CacheLoadRetryAfterSeconds=20 JobCompletionPollSeconds=20 JobCompletionDoWait=true DeleteOnRecall=true ### Configuration Keywords¶ #### Endpoint¶ Defines the Hostname or IP Address, protocol and port whereby Ngenea HSM can access the BlackPearl Appliance. Syntax Endpoint=<http|https>://<Hostname or IP Address>[:port] Example Endpoint=https://blackpearl.example.com:443 #### AccessKeyId¶ Defines the S3 Access ID for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide. Syntax AccessKeyId=<Spectra S3 Access Key> Example AccessKeyId=ABCDEFGHIJLKMNOPQRST #### SecretAccessKey¶ The S3 Secret Key for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide. Syntax SecretAccessKey=<Spectra S3 Secret Key> Example SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm #### Bucket¶ Defines the Bucket Name of the target BlackPearl Bucket. This bucket must exist on the S3 storage target prior to migrating data using Ngenea HSM. Syntax Bucket=<Bucket Name> Example Bucket=MyBucketName #### RemoteLocationXAttr¶ Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file.$1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetname:<LocalFileRegexReference>

Example

RemoteLocationXAttr=blackpearl:$1 #### RetrieveObjectName¶ Defines the mapping from the local APXrmtlc to the associated file on the remote storage target.$1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=<RemoteLocationXAttrReference>

Example

RetrieveObjectName=$1 #### StoreObjectName¶ Defines the exporession by which the remote filename is composed when migrating to the external storage.$1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

This setting would typically match the RetrieveObjectName above.

Syntax

StoreObjectName=<LocalFileRegexReference>

Example

StoreObjectName=$1 #### CacheFullMaxWaitSeconds¶ Defines the maximum number of seconds Ngenea HSM waits for space to be made available in the BlackPearl Appliance cache for migration of new data prior to cancelling the current job if no space is made available. Syntax CacheFullMaxWaitSeconds=<time in seconds> Example CacheFullMaxWaitSeconds=300 #### CacheFullRetryAfterSeconds¶ Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if the cache has free space. Syntax CacheFullRetryAfterSeconds=<time in seconds> Example CacheFullRetryAfterSeconds=20 #### CacheLoadRetryAfterSeconds¶ Defines the Interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if retrieved files are present in the cache. Syntax CacheLoadRetryAfterSeconds=<time in seconds> Example CacheLoadRetryAfterSeconds=20 #### JobCompletionPollSeconds¶ Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if an ongoing job has completed. Syntax JobCompletionPollSeconds=<time in seconds> Example JobCompletionPollSeconds=20 #### JobCompletionDoWait¶ Defines whether Ngenea HSM waits for a BlackPearl Job to complete prior to exiting the current operation. Setting this to 'true' will cause ngmigrate to wait until the BlackPearl has secured data to tape from the BlackPearl cache before completing. Setting this to 'false' will cause ngmigrate to return once the data has been succesfully uploaded to the BlackPearl cache. Syntax JobCompletionDoWait=<true|false> Example JobCompletionDoWait=true #### MaxUploadSize¶ This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate parts equal to this size (except the final part, which may be smaller). The minimum value of this setting is 10485760 (10MB). The maximum value is determined by the BlackPearl target. If this parameter is unset, the part sizes will be determined by the BlackPearl target (this is the recommended configuration). Syntax MaxUploadSize=<size_in_bytes> Example MaxUploadSize=10737418240 #### MaxUploadThreadCount¶ Defines the maximum number of threads an ngmigrate command will use when uploading files to the BlackPearl target. Defaults to the number of CPU cores in the system. Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial. Syntax MaxUploadThreadCount=<# of threads> Example MaxUploadThreadCount=8 #### DeleteOnRecall¶ Defines whether to delete the data from the remote BlackPearl target when successfully recalled back to the file system. Setting this to False will result in files entering a "premigrated" state on successful recall. Syntax DeleteOnRecall=<True|False> Example DeleteOnRecall=True ## Azure Blob Storage Type¶ The following section describes how to setup an Azure storage target. Azure Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and container. ### Example Configuration File¶ [General] StorageAccount=MyStorageAccount AccessKey=MyAccessKey Container=MyContainerName Scheme=HTTPS RemoteLocationXAttr=ngenea-target1:$1
RetrieveObjectName=$1 StoreObjectName=$1
DeleteOnRecall=True

### Configuration Keywords¶

#### StorageAccount¶

Defines the storage account name used to authenticate to Azure.

For further information regarding Azure storage accounts please refer to the Microsoft Azure Documentation.

Syntax

StorageAccount=<Account Name>

Example

StorageAccount=MyStorageAccount

#### AccessKey¶

Defines the storage account access key used to authenticate to Azure.

For further information regarding Azure Account Access Keys please refer to the Microsoft Azure Documentation.

Syntax

AccessKey=<Azure Account Access Key>

Example

AccessKeyId=ABCDEFGHIJLKMNOPQRST

#### Container¶

Defines the 'container' into which files are migrated. The container will be created on the Azure storage target if it does not exist already.

For further information regarding Azure Conainers and Blobs please refer to the Microsoft Azure Documentation.

Syntax

Container=<Name of the Azure container>

Example

Container=MyContainerName

#### Scheme¶

Defines the transport protocol used to communicate with the Azure target.

ArcaPix only supports HTTPS protocol for Ngenea HSM solutions.

This optional parameter defaults to HTTPS if not explicitly defined.

Syntax

Scheme=<PROTOCOL>

Example

Scheme=HTTPS

#### RemoteLocationXAttr¶

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file. Syntax RemoteLocationXAttr=targetref:<LocalFileRegexReference> Example RemoteLocationXAttr=ngenea-target1:$1

#### RetrieveObjectName¶

Defines the mapping from the local APXrmtlc to the associated file on the remote storage target. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file. Syntax RetrieveObjectName=<RemoteLocationXAttrReference> Example RetrieveObjectName=$1

#### StoreObjectName¶

Defines the expression by which the remote filename is composed when migrating to the external storage. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file. This setting would typically match the RetrieveObjectName above. Syntax StoreObjectName=<LocalFileRegexReference> Example StoreObjectName=$1

Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Azure server into one object.

Set this value to be less than or equal to that supported by the remote storage target.

Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).

This optional parameter defaults to 104857600 (100MiB) if not explicitly defined.

Syntax

Example

#### DeleteOnRecall¶

Defines whether to delete the data from the remote Azure target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

## Multi-Target Support¶

The following section describes how to configure Ngenea to migrate and recall to multiple Storage Targets in parallel.

With reference to the Master Configuration File example in General Configuration which demonstrates a configuration to migrate data from two differing directories to associated targets respectively, Ngenea can migrate data from the same directory or even specific individual files to multiple targets.

Such a configuration can help ensure data can be recalled if a storage target is unavailable, for instance due to a disaster or a provider service outage.

### Example Whole Filesystem to Multiple-Targets Configuration¶

 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [Global] MinMigrateEndpointCount=0 [Storage awss3] StorageType=AmazonS3 StorageKey=01 ConfigFile=awss3.conf LocalFileRegex=/mmfs1/(.+) RemoteLocationXAttrRegex=awss3:(.+) [Storage azure] StorageType=Azure StorageKey=02 ConfigFile=azure.conf RemoteLocationXAttrRegex=azure:(.+) LocalFileRegex=/mmfs1/(.+)

### File Matching to Multiple Targets¶

Ngenea matches a file to be migrated against the LocalFileRegex parameter of each Storage Target to determine if the file should be migrated to the target. To migrate files to multiple storage target endpoints ensure that the LocalFileRegex either specifically targets or overlaps the file's path for each relevant storage target.

As per the above example, specifying LocalFileRegex=/mmfs1/(.+) for all targets will ensure that all files within the /mmfs1 directory tree are migrated to both targets.

Alternatively, more granular control can be applied.

Given two storage targets (A) and (B) with differing values for LocalFileRegex parameters:

A: LocalFileRegex=/mmfs1/(.+)

B: LocalFileRegex=/mmfs1/archive/(.+)

After successful migration the first Storage Target (A) would contain all files under the /mmfs1 directory structure. The second Storage Target (B) will only contain files within the /mmfs1/archive directory structure. Thus resulting in identically migrated files within the /mmfs1/archive directory structure on both Storage Targets.

### Recall Behaviour¶

Ngenea will recall migrated files in top-down order of the Storage Target stanzas in the Master Configuration File which match the criteria for the file. Should a storage target not be able to provide the file for recall, Ngenea will use the next matching Storage Target in top-down order - and so on, until all Storage Targets have been exhausted.

### Multi-Target Specific Configuration Keywords¶

#### MinMigrateEndpointCount¶

Defines the number of associated endpoints a file needs to be migrated to in order for the migration to be considered a success.

If the "MinMigrateEndpointCount" parameter is equal to 0 or is greater than or equal to the number of associated storage endpoints for the file, and the file was successfully migrated to all those endpoints, then Ngenea considers the file migrated successfully.

If the "MinMigrateEndpointCount" parameter is greater than 0 and is less than the number of associated storage endpoints for file, and the file was successfully migrated to the minimum number of endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migrated successfully.

If the file was successfully migrated to at least one endpoint associated with the file but less than the minimum number of storage endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migration partially successful.

If the file was not successfully migrated to any endpoint, Ngenea considers the migration of the file as failed.

On exit, Ngenea's ngmigrate utility returns the following exit statuses:

• 0 - all files were migrated successfully
• 1 - all files failed to migrate successfully
• 2 - some files failed to migrate successfully

Syntax

MinMigrateEndpointCount=<Non-negative integer value>

Example

MinMigrateEndpointCount=2

#### StorageKey¶

Defines a custom identifier attributable to the storage endpoint.

Every [Storage] section of the master configuration file can contain the "StorageKey" parameter. The parameter's value must be a two-byte string not equal to the characters "sz". The default value of the parameter is "lc".

When implementing multi-target migration for files, each storage endpoint must have the parameter defined with a unique key.

Syntax

StorageKey=<char><char>

Examples

StorageKey=16
StorageKey=zz
StorageKey=3a