Storage Target Configuration

Ngenea HSM supports Storage Targets compatible with the Amazon S3 protocol, Spectra Logic's BlackPearl Spectra S3 protocol (a superset of the Amazon S3 Protocol) and Microsoft Azure.

Storage Targets are referenced from the Master Configuration File, with configuration data for each target defined in a target-specific configuration file.

The default location for a Storage Target Configuration File is in /opt/arcapix/etc/.

Native Amazon S3 Type

The following section describes how to setup a Storage Target to an S3 compatible storage target.

S3 Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified end point and bucket.

Example Configuration File

[General]
Endpoint=endpoint.host.name
Port=443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
Region=eu-west-2
Scheme=HTTPS
RemoteLocationXAttr=ngenea-target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True

Configuration Keywords

Endpoint

For Amazon AWS S3 targets, do not include this parameter. It will be automatically determined by Amazon based on the region and bucket name.

For non-Amazon AWS S3 targets, this parameter defines the Hostname or IP Address whereby Ngenea HSM can access the S3 target.

Syntax

Endpoint=<Hostname or IP address>

Example

Endpoint=s3.host.domain

Port

For Amazon AWS S3 targets, do not include this parameter.

For non-Amazon AWS S3 targets, this parameter defines the tcp port to connect to on the remote storage.

The default value is 443 if this parameter is unspecified.

Syntax

Port:<TCP port number>

Example

Port:443

AccessKeyId

Defines the Account Access Key used to authenticate against the S3 target.

For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.

Syntax

AccessKeyId=<Amazon Account Access Key>

Example

AccessKeyId=ABCDEFGHIJLKMNOPQRST

SecretAccessKey

Defines the Secret Key used to authenticate against the S3 target.

For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.

Syntax

SecretAccessKey=<Amazon Secret Access Key>

Example

SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm

Bucket

Defines the bucket, or 'container', into which files are migrated. This bucket must exist on the S3 storage target prior to migrating data using Ngenea HSM.

For further information regarding Amazon AWS Buckets please refer to the Amazon Documentation.

Syntax

Bucket=<Name of the S3 bucket>

Example

Bucket=MyBucketName

Region

Defines the regional endpoint in use for the target bucket.

For further information regarding Amazon AWS Regions please refer to the Amazon Documentation.

Other S3 targets may have significantly different requirements for how to specify this parameter. Consult your S3 target documentation for further information.

Syntax

Region=<Name of the S3 Region>

Example

Region=eu-west-2

Scheme

Defines the transport protocol used to communicate with the S3 target.

ArcaPix only supports HTTPS protocol for Ngenea HSM solutions.

Syntax

Scheme=<PROTOCOL>

Example

Scheme=HTTPS

RemoteLocationXAttr

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=ngenea-target1:$1

RetrieveObjectName

Defines the mapping from the local APXrmtlc to the associated file on the remote storage target. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=<RemoteLocationXAttrReference>

Example

RetrieveObjectName=$1

StoreObjectName

Defines the expression by which the remote filename is composed when migrating to the external storage. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

This setting would typically match the RetrieveObjectName above.

Syntax

StoreObjectName=<LocalFileRegexReference>

Example

StoreObjectName=$1

MaxUploadSize

Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the S3 server into one object.

Set this value to be less than or equal to that supported by the remote storage target.

Valid settings are between 5242880 (5MB) and 5368709120 (5GB).

This optional parameter defaults to 5GB if not explicitly defined.

Syntax

MaxUploadSize=<size_in_bytes>

Example

MaxUploadSize=5368709120

DeleteOnRecall

Defines whether to delete the data from the remote S3 target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

Filesystem Target

The following section describes how to setup a Storage Target to a POSIX-compatible mounted file system.

The target file system must be mounted on all nodes which will migrate or recall files.

Example Configuration File

[General]
StoreObjectName=/mnt/targetfs/$1
RetrieveObjectName=/mnt/targetfs/$1
RemoteLocationXAttr=ngenea-targetfs1:$1
DeleteOnRecall=False
ObjectXAttrManipulationMode=auto
EnsureMountPoint=/mnt/targetfs

Configuration Keywords

StoreObjectName

Defines the value prepended to the path and filename on migration, which will usually point to the target mount point. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

StoreObjectName=/mountpoint/<LocalFileRegexReference>

Example

StoreObjectName=/mnt/targetfs/$1

RetrieveObjectName

Defines the value prepended to the path and filename on recall, which will usually point to the target mount point and match the value of StoreObjectName. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=/mountpoint/<LocalFileRegexReference>

Example

RetrieveObjectName=/mnt/targetfs/$1

RemoteLocationXAttr

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=ngenea-targetfs1:$1

DeleteOnRecall

Defines whether to immediately delete the data from the POSIX mount target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

ObjectXAttrManipulationMode

Defines the behaviour for storing the metadata of files migrated to the POSIX mount target.

When native mode is enabled, metadata will be stored as extended attributes on migrated files. If the target POSIX mount does not support extended attributes, then migration and recall will abort with an error.

When shadow mode is enabled, metadata will be stored in an accompanying file which is stored alongside the data file.

Setting this to auto (the default) will result in the ngmigrate process attempting to store metadata in extended attributes, and if that fails it will store the metadata in an accompanying file.

Syntax

ObjectXAttrManipulationMode=<auto|shadow|native>

Example

ObjectXAttrManipulationMode=auto

EnsureMountPoint

Specifies the mount point that ngmigrate and ngrecall processes will validate prior to migrating or recalling a file.

Should the specified location not be the root of a mount point, the migration or recall will abort with an error.

This is an important safety feature to ensure that the local disk is not used in the event of a target mount failing.

It is typically set to the root of the mount point.

Syntax

EnsureMountPoint=/mountpoint

Example

EnsureMountPoint=/mnt/targetfs

Spectra Logic BlackPearl Target

ArcaPix's Ngenea HSM software is fully certified by Spectra Logic as an approved BlackPearl client.

Ngenea HSM requires configuration and activation of the BlackPearl's Spectra S3 Service. Please refer to the relevant section of the BlackPearl User Guide.

The following example describes the information required to be present withing the Storage Target Configuration File to enable migration to the specified bucket.

Example Configuration File

[General]
Endpoint=https://IPADDRESS_OF_BLACKPEARL:443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
RemoteLocationXAttr=blackpearl:$1
RetrieveObjectName=$1
StoreObjectName=$1
CacheFullMaxWaitSeconds=300
CacheFullRetryAfterSeconds=20
CacheLoadRetryAfterSeconds=20
JobCompletionPollSeconds=20
JobCompletionDoWait=true
DeleteOnRecall=true

Configuration Keywords

Endpoint

Defines the Hostname or IP Address, protocol and port whereby Ngenea HSM can access the BlackPearl Appliance.

Syntax

Endpoint=<http|https>://<Hostname or IP Address>[:port]

Example

Endpoint=https://blackpearl.example.com:443

AccessKeyId

Defines the S3 Access ID for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.

Syntax

AccessKeyId=<Spectra S3 Access Key>

Example

AccessKeyId=ABCDEFGHIJLKMNOPQRST

SecretAccessKey

The S3 Secret Key for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.

Syntax

SecretAccessKey=<Spectra S3 Secret Key>

Example

SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm

Bucket

Defines the Bucket Name of the target BlackPearl Bucket. This bucket must exist on the S3 storage target prior to migrating data using Ngenea HSM.

Syntax

Bucket=<Bucket Name>

Example

Bucket=MyBucketName

RemoteLocationXAttr

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetname:<LocalFileRegexReference>

Example

RemoteLocationXAttr=blackpearl:$1

RetrieveObjectName

Defines the mapping from the local APXrmtlc to the associated file on the remote storage target. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=<RemoteLocationXAttrReference>

Example

RetrieveObjectName=$1

StoreObjectName

Defines the exporession by which the remote filename is composed when migrating to the external storage. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

This setting would typically match the RetrieveObjectName above.

Syntax

StoreObjectName=<LocalFileRegexReference>

Example

StoreObjectName=$1

CacheFullMaxWaitSeconds

Defines the maximum number of seconds Ngenea HSM waits for space to be made available in the BlackPearl Appliance cache for migration of new data prior to cancelling the current job if no space is made available.

Syntax

CacheFullMaxWaitSeconds=<time in seconds>

Example

CacheFullMaxWaitSeconds=300

CacheFullRetryAfterSeconds

Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if the cache has free space.

Syntax

CacheFullRetryAfterSeconds=<time in seconds>

Example

CacheFullRetryAfterSeconds=20

CacheLoadRetryAfterSeconds

Defines the Interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if retrieved files are present in the cache.

Syntax

CacheLoadRetryAfterSeconds=<time in seconds>

Example

CacheLoadRetryAfterSeconds=20

JobCompletionPollSeconds

Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if an ongoing job has completed.

Syntax

JobCompletionPollSeconds=<time in seconds>

Example

JobCompletionPollSeconds=20

JobCompletionDoWait

Defines whether Ngenea HSM waits for a BlackPearl Job to complete prior to exiting the current operation.

Setting this to 'true' will cause ngmigrate to wait until the BlackPearl has secured data to tape from the BlackPearl cache before completing.

Setting this to 'false' will cause ngmigrate to return once the data has been succesfully uploaded to the BlackPearl cache.

Syntax

JobCompletionDoWait=<true|false>

Example

JobCompletionDoWait=true

MaxUploadSize

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate parts equal to this size (except the final part, which may be smaller).

The minimum value of this setting is 10485760 (10MB). The maximum value is determined by the BlackPearl target.

If this parameter is unset, the part sizes will be determined by the BlackPearl target (this is the recommended configuration).

Syntax

MaxUploadSize=<size_in_bytes>

Example

MaxUploadSize=10737418240

MaxUploadThreadCount

Defines the maximum number of threads an ngmigrate command will use when uploading files to the BlackPearl target.

Defaults to the number of CPU cores in the system.

Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.

Syntax

MaxUploadThreadCount=<# of threads>

Example

MaxUploadThreadCount=8

DeleteOnRecall

Defines whether to delete the data from the remote BlackPearl target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

Azure Blob Storage Type

The following section describes how to setup an Azure storage target.

Azure Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and container.

Example Configuration File

[General]
StorageAccount=MyStorageAccount
AccessKey=MyAccessKey
Container=MyContainerName
Scheme=HTTPS
RemoteLocationXAttr=ngenea-target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True

Configuration Keywords

StorageAccount

Defines the storage account name used to authenticate to Azure.

For further information regarding Azure storage accounts please refer to the Microsoft Azure Documentation.

Syntax

StorageAccount=<Account Name>

Example

StorageAccount=MyStorageAccount

AccessKey

Defines the storage account access key used to authenticate to Azure.

For further information regarding Azure Account Access Keys please refer to the Microsoft Azure Documentation.

Syntax

AccessKey=<Azure Account Access Key>

Example

AccessKeyId=ABCDEFGHIJLKMNOPQRST

Container

Defines the 'container' into which files are migrated. The container will be created on the Azure storage target if it does not exist already.

For further information regarding Azure Conainers and Blobs please refer to the Microsoft Azure Documentation.

Syntax

Container=<Name of the Azure container>

Example

Container=MyContainerName

Scheme

Defines the transport protocol used to communicate with the Azure target.

ArcaPix only supports HTTPS protocol for Ngenea HSM solutions.

This optional parameter defaults to HTTPS if not explicitly defined.

Syntax

Scheme=<PROTOCOL>

Example

Scheme=HTTPS

RemoteLocationXAttr

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=ngenea-target1:$1

RetrieveObjectName

Defines the mapping from the local APXrmtlc to the associated file on the remote storage target. $1 represents the use of the filename and path matching the ( ) section of the RemoteLocationXAttrRegex setting for this storage target in the master configuration file.

Syntax

RetrieveObjectName=<RemoteLocationXAttrReference>

Example

RetrieveObjectName=$1

StoreObjectName

Defines the expression by which the remote filename is composed when migrating to the external storage. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the master configuration file.

This setting would typically match the RetrieveObjectName above.

Syntax

StoreObjectName=<LocalFileRegexReference>

Example

StoreObjectName=$1

MaxUploadSize

Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Azure server into one object.

Set this value to be less than or equal to that supported by the remote storage target.

Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).

This optional parameter defaults to 104857600 (100MiB) if not explicitly defined.

Syntax

MaxUploadSize=<size_in_bytes>

Example

MaxUploadSize=104857600

DeleteOnRecall

Defines whether to delete the data from the remote Azure target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

Multi-Target Support

The following section describes how to configure Ngenea to migrate and recall to multiple Storage Targets in parallel.

With reference to the Master Configuration File example in General Configuration which demonstrates a configuration to migrate data from two differing directories to associated targets respectively, Ngenea can migrate data from the same directory or even specific individual files to multiple targets.

Such a configuration can help ensure data can be recalled if a storage target is unavailable, for instance due to a disaster or a provider service outage.

Example Whole Filesystem to Multiple-Targets Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[Global]
MinMigrateEndpointCount=0

[Storage awss3]
StorageType=AmazonS3
StorageKey=01
ConfigFile=awss3.conf
LocalFileRegex=/mmfs1/(.+)
RemoteLocationXAttrRegex=awss3:(.+)

[Storage azure]
StorageType=Azure
StorageKey=02
ConfigFile=azure.conf
RemoteLocationXAttrRegex=azure:(.+)
LocalFileRegex=/mmfs1/(.+)

File Matching to Multiple Targets

Ngenea matches a file to be migrated against the LocalFileRegex parameter of each Storage Target to determine if the file should be migrated to the target. To migrate files to multiple storage target endpoints ensure that the LocalFileRegex either specifically targets or overlaps the file's path for each relevant storage target.

As per the above example, specifying LocalFileRegex=/mmfs1/(.+) for all targets will ensure that all files within the /mmfs1 directory tree are migrated to both targets.

Alternatively, more granular control can be applied.

Given two storage targets (A) and (B) with differing values for LocalFileRegex parameters:

A: LocalFileRegex=/mmfs1/(.+)

B: LocalFileRegex=/mmfs1/archive/(.+)

After successful migration the first Storage Target (A) would contain all files under the /mmfs1 directory structure. The second Storage Target (B) will only contain files within the /mmfs1/archive directory structure. Thus resulting in identically migrated files within the /mmfs1/archive directory structure on both Storage Targets.

Recall Behaviour

Ngenea will recall migrated files in top-down order of the Storage Target stanzas in the Master Configuration File which match the criteria for the file. Should a storage target not be able to provide the file for recall, Ngenea will use the next matching Storage Target in top-down order - and so on, until all Storage Targets have been exhausted.

Multi-Target Specific Configuration Keywords

MinMigrateEndpointCount

Defines the number of associated endpoints a file needs to be migrated to in order for the migration to be considered a success.

If the "MinMigrateEndpointCount" parameter is equal to 0 or is greater than or equal to the number of associated storage endpoints for the file, and the file was successfully migrated to all those endpoints, then Ngenea considers the file migrated successfully.

If the "MinMigrateEndpointCount" parameter is greater than 0 and is less than the number of associated storage endpoints for file, and the file was successfully migrated to the minimum number of endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migrated successfully.

If the file was successfully migrated to at least one endpoint associated with the file but less than the minimum number of storage endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migration partially successful.

If the file was not successfully migrated to any endpoint, Ngenea considers the migration of the file as failed.

On exit, Ngenea's ngmigrate utility returns the following exit statuses:

  • 0 - all files were migrated successfully
  • 1 - all files failed to migrate successfully
  • 2 - some files failed to migrate successfully

Syntax

MinMigrateEndpointCount=<Non-negative integer value>

Example

MinMigrateEndpointCount=2

StorageKey

Defines a custom identifier attributable to the storage endpoint.

Every [Storage] section of the master configuration file can contain the "StorageKey" parameter. The parameter's value must be a two-byte string not equal to the characters "sz". The default value of the parameter is "lc".

When implementing multi-target migration for files, each storage endpoint must have the parameter defined with a unique key.

Syntax

StorageKey=<char><char>

Examples

StorageKey=16
StorageKey=zz
StorageKey=3a