Storage Target Configuration¶
ArcaPix Ngenea HSM supports Storage Targets compatible with the Amazon S3 protocol, Spectra Logic's BlackPearl Spectra S3 protocol (a superset of the Amazon S3 Protocol), Microsoft Azure and Google Cloud.
Storage Targets are referenced from the Master Configuration File, with configuration data for each target defined in a target-specific configuration file.
The default location for a Storage Target Configuration File is in /opt/arcapix/etc/.
Amazon S3 Type¶
The following section describes how to setup a Storage Target to an S3 compatible storage target.
S3 Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified end point and bucket.
Example Configuration File¶
[General]
Endpoint=endpoint.host.name
Port=443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
Region=eu-west-2
Scheme=HTTPS
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True
Configuration Keywords (Amazon S3)¶
Keyword |
Default |
Required |
|---|---|---|
Endpoint |
Null |
No |
Port |
80 |
No |
Scheme |
"https" |
No |
SSLVerify |
True |
No |
AccessKeyId |
None |
Yes |
SecretAccessKey / CredentialsFile |
None |
Yes |
Bucket |
None |
Yes |
Region |
Null |
Yes |
StorageClass |
Null |
No |
RetrieveObjectBasePath |
"" |
No |
RetrieveObjectName |
"$0" |
No |
StoreObjectBasePath |
"" |
No |
StoreObjectName |
"$0" |
No |
RemoteLocationXAttr |
"$0" |
No |
DeleteOnRecall |
False |
No |
MaxUploadSize |
20 MB |
No |
MaxDownloadSize |
20 MB |
No |
MaxUploadThreadCount |
Number of cores |
No |
MaxDownloadThreadCount |
Number of cores |
No |
MaxXAttrNativeMetadataSize |
0 |
No |
MaxKeysListObjects |
1000 |
No |
ConnectionTimeoutMs |
3000 |
No |
RequestTimeoutMs |
3000 |
No |
MaxRetryCount |
10 |
No |
EscapeNames |
True |
No |
ShadowFolderMetadataSave |
False |
No |
ShadowFolderMetadataMode |
auto |
No |
SymlinkMode |
object |
No |
ACLSave |
False |
No |
SDKLogging |
False |
No |
Endpoint¶
For Amazon AWS S3 targets, do not include this parameter. It will be automatically determined by Amazon based on the region and bucket name.
For non-Amazon AWS S3 targets, this parameter defines the Hostname or IP Address whereby Ngenea can access the S3 target.
Syntax
Endpoint=<Hostname or IP address>
Example
Endpoint=s3.host.domain
Port¶
For Amazon AWS S3 targets, do not include this parameter.
For non-Amazon AWS S3 targets, this parameter defines the tcp port to connect to on the remote storage.
The default value is 443 if this parameter is unspecified.
Syntax
Port:<TCP port number>
Example
Port:443
AccessKeyId¶
Defines the Account Access Key used to authenticate against the S3 target.
For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.
Syntax
AccessKeyId=<Amazon Account Access Key>
Example
AccessKeyId=ABCDEFGHIJLKMNOPQRST
SecretAccessKey / CredentialsFile¶
Defines the Secret Key used to authenticate against the S3 target.
For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.
The Secret Key may be defined in the storage target configuration file with the keyword SecretAccessKey, or in a separate file referenced by the keyword CredentialsFile.
Syntax
SecretAccessKey=<Amazon Secret Access Key>
or
CredentialsFile=<file-name>
Example
SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm
or
CredentialsFile=access_key
Bucket¶
Defines the bucket, or 'container', into which files are migrated. This bucket must exist on the S3 storage target prior to migrating data using Ngenea.
For further information regarding Amazon AWS Buckets please refer to the Amazon Documentation.
Syntax
Bucket=<Name of the S3 bucket>
Example
Bucket=MyBucketName
Region¶
Defines the regional endpoint in use for the target bucket.
For further information regarding Amazon AWS Regions please refer to the Amazon Documentation.
Other S3 targets may have significantly different requirements for how to specify this parameter. Consult your S3 target documentation for further information.
Syntax
Region=<Name of the S3 Region>
Example
Region=eu-west-2
StorageClass¶
Defines a storage class for objects uploaded to the target bucket.
Valid storage classes include STANDARD, STANDARD_IA, REDUCED_REDUNDANCY, and GLACIER.
Example
StorageClass=GLACIER
Scheme¶
Defines the transport protocol used to communicate with the S3 target.
ArcaStream / Pixit Media only supports HTTPS protocol for Ngenea solutions.
Syntax
Scheme=<PROTOCOL>
Example
Scheme=HTTPS
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=target1:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
MaxUploadSize¶
Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the S3 server into one object.
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 5242880 (5MB) and 5368709120 (5GB).
This optional parameter defaults to 20MB if not explicitly defined.
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=10485760
MaxDownloadSize¶
The maximum size of a downloaded file part. The ngrecall command can download multiple file parts in parallel. If a network error is experienced while downloading a file part, ngrecall retries downloading the failed file part.
Valid settings are between 5242880 (5MB) and 5368709120 (5GB).
This optional parameter defaults to 20MB if not explicitly defined.
Syntax
MaxDownloadSize=<size_in_bytes>
Example
MaxDownloadSize=10485760
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "get bucket" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "get bucket" requests.
The default value for this parameter is 1000. A value specified for this parameter should be less than or equal to that default value.
SSLVerify¶
Controls the behaviour of TLS certificate-related checks.
If the value of this parameter is True, the checks are performed.
Default parameter value is True.
Syntax
SSLVerify=<True|False>
Example
SSLVerify=False
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the S3 target. Each thread uploads an individual file part.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxDownloadThreadCount¶
Defines the maximum number of threads an ngrecall command will use when downloading files from the S3 target. Each thread downloads an individual file part.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.
Syntax
MaxDownloadThreadCount=<# of threads>
Example
MaxDownloadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements.
Ngenea supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for S3 is 2048.
A recommended setting for MaxXAttrNativeMetadataSize is 1024 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
DeleteOnRecall¶
Defines whether to delete the data from the remote S3 target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
ConnectionTimeoutMs¶
Specifies the timeout (in milliseconds) for opening a socket connection to an S3 storage endpoint.
Default parameter value is 3000.
Syntax
ConnectionTimeoutMs=<timeout in milliseconds>
Example
ConnectionTimeoutMs=2000
RequestTimeoutMs¶
Specifies the timeout (in milliseconds) for transferring data from or to an S3 storage endpoint.
If the speed of a data transfer via a socket connection is too slow during that time period, then the data transfer is aborted, and the socket connection is closed.
Default parameter value is 3000.
Syntax
RequestTimeoutMs=<timeout in milliseconds>
Example
RequestTimeoutMs=10000
MaxRetryCount¶
Specifies the number of retries of operations to access an S3 storage endpoint.
Every subsequent attempt to perform an operation is done after an increasing delay.
By default, a failed operation is retried up to 10 times.
Syntax
MaxRetryCount=<number of retries>
Example
MaxRetryCount=10
EscapeNames¶
Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.
If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.
Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.
Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.
The default value is True.
Syntax
EscapeNames=<True|False>
Example
EscapeNames=False
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects are located at the same hierarchy level as remote folders and have the names FOLDERNAME/ or .FOLDERNAME.xattr, where FOLDERNAME is a remote folder name.
The parameter ShadowFolderMetadataMode controls the naming scheme for shadow metadata objects.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories, and ngrecall will recreate them with default file mode, owner, group, and ACL.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
ShadowFolderMetadataMode¶
Defines the naming scheme of shadow metadata objects for remote folders.
In the following description FOLDERNAME is the remote folder name.
The following values are permitted:
"native" --- use shadow metadata objects with names
FOLDERNAME/"shadow" --- use shadow metadata objects with names
.FOLDERNAME.xattr. For MinIO targets, "shadow" is recommended."auto" --- try using a shadow metadata object with the name
FOLDERNAME/, then with the name.FOLDERNAME.xattr
Shadow metadata objects are located at the same hierarchy level as remote folders.
The default value is "auto".
Syntax
ShadowFolderMetadataMode=<auto|native|shadow>
Example
ShadowFolderMetadataMode=shadow
SymlinkMode¶
Defines whether Ngenea stores the target of a symbolic link as data content within an object or within an object's metadata at the storage target.
If the parameter SymlinkMode is object, the local target of the symlink is stored as plaintext data within the named symlink object at the storage target.
If the parameter SymlinkMode is metadata, the local target of the symlink is stored within the apxsymln metadata key for the object at the storage target.
The default value is object.
Syntax
SymlinkMode=<object|metadata>
Example
SymlinkMode=metadata
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
SDKLogging¶
Specifies a log level for the AWS C++ SDK library. If a master configuration file defines multiple S3 storage endpoints, then a log level specified for the first accessed S3 storage endpoint will be in effect.
Log messages are prefixed "AWS: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<fatal|error|warn|info|debug|trace|false>
Boolean "True" is equivalent to "trace".
Example
SDKLogging=True
The default value is "False" which disables logging.
Security¶
The permissions listed in the table below must be enabled to allow Ngenea to correctly function. See Amazon S3 API Reference.
Action |
Description |
|---|---|
AbortMultipartUpload |
Abort a multipart upload |
CompleteMultipartUpload |
Complete a multipart upload |
CopyObject |
Creates a copy of an existing object |
CreateMultipartUpload |
Initiate a multipart upload |
DeleteObject |
Delete an object |
DeleteObjects |
Delete multiple objects from a bucket |
GetObject |
Retrieve an object |
HeadObject |
Retrieves metadata from an object |
ListBucket |
List objects in a bucket |
ListObjectsV2 |
List objects in a bucket |
PutObject |
Adds an object to a bucket |
UploadPart |
Uploads a part in a multipart upload |
Filesystem Target¶
The following section describes how to setup a Storage Target to a POSIX-compatible mounted file system.
The target file system must be mounted on all nodes which will migrate or recall files.
Example Configuration File¶
[General]
RetrieveObjectBasePath=/mnt/targetfs
RetrieveObjectName=$1
StoreObjectBasePath=/mnt/targetfs
StoreObjectName=$1
RemoteLocationXAttr=targetfs1:$1
DeleteOnRecall=False
ObjectXAttrManipulationMode=auto
EnsureMountPoint=/mnt/targetfs
Configuration Keywords (Filesystem)¶
Keyword |
Default |
Required |
|---|---|---|
RetrieveObjectBasePath |
"" |
No |
RetrieveObjectName |
None |
Yes |
StoreObjectBasePath |
"" |
No |
StoreObjectName |
None |
Yes |
RemoteLocationXAttr |
"$0" |
No |
DeleteOnRecall |
False |
No |
EnsureMountPoint |
None |
Yes |
ObjectXAttrManipulationMode |
"auto" |
No |
ShadowMetadataStore |
"samedir" |
No |
ACLSave |
False |
No |
RetrieveObjectBasePath¶
Defines a base path for copying files from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a file to copy. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
The empty base path is interpreted as /.
If a base path does not end with /, then / is implicitly appended to the base path.
The base path ordinarily specifies a mount point or a directory on the mount point. Such a base path is the root of a filesystem sub-tree representing the remote storage target. When listing objects on the remote storage target, ngscan lists files in this sub-tree by default.
The default value for this parameter is /.
This value is inconvenient for using with ngscan: by default, it will list files starting at the root of the entire filesystem.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=/mnt/targetfs
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated file on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a file to retrieve (copy from).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
It is recommended to strip a directory prefix from the value of this parameter and specify the prefix in the RetrieveObjectBasePath parameter. Otherwise, ngscan will ignore that directory prefix when listing files on the remote storage target.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
StoreObjectBasePath¶
Defines a base path for copying files to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a file to copy. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
The empty base path is interpreted as /.
If a base path does not end with /, then / is implicitly appended to the base path.
The base path ordinarily specifies a mount point or a directory on the mount point. Such a base path is the root of a filesystem sub-tree representing the remote storage target. When listing objects on the remote storage target, ngscan lists files in this sub-tree.
The default value for this parameter is /.
This value is inconvenient for using with ngscan: it will list files starting at the root of the entire filesystem.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=/mnt/targetfs
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated file on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a file to store (copy to).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
It is recommended to strip a directory prefix from the value of this parameter and specify the prefix in the StoreObjectBasePath parameter. Otherwise, ngscan will ignore that directory prefix when listing files on the remote storage target.
The default value for this parameter is $0.
Example
StoreObjectName=$1
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=targetfs1:$1
DeleteOnRecall¶
Defines whether to immediately delete the data from the POSIX mount target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
ObjectXAttrManipulationMode¶
Defines the behaviour for storing and retrieving the metadata of files and directories migrated to and recalled from the POSIX mount target.
When native mode is selected, metadata is stored and retrieved as extended attributes on migrated files and directories.
If the target POSIX mount does not support extended attributes, then migration and recall will abort with an error.
When shadow mode is selected, metadata is stored to and retrieved from a file stored alongside the data file or directory.
When disable mode is selected, metadata will not be used.
Setting this parameter to auto (the default) results in the ngmigrate/ngrecall process attempting to store metadata to
or retrieve metadata from extended attributes, and if the storing or retrieving operation fails, ngmigrate/ngrecall uses an accompanying file for storing or retrieving metadata.
Syntax
ObjectXAttrManipulationMode=<auto|shadow|native|disable>
Example
ObjectXAttrManipulationMode=disable
ShadowMetadataStore¶
Defines a location to store shadow metadata files.
If this parameter is equal to "samedir", shadow metadata files for remote objects and folders are stored in the same directory as the remote objects and folders.
If this parameter is equal to "subdir", shadow metadata files for remote objects and folders are stored in the subfolder .ngenea_metadata of a remote folder containing the remote objects and folders.
If this parameter is equal to "altdir:PATH", shadow metadata files for remote objects and folders are stored in the directory PATH, where PATH should be absolute one (i.e., start with /).
Ngenea tools automatically create missing directories for placing the shadow metadata file of a remote object or folder.
Directories created under PATH correspond to parent folders of the remote object or folder.
Parent folders are replicated until reaching the longest common ancestor path calculated for PATH and the absolute path of the remote object or folder.
E.g., if PATH is "/mnt/fs_storage/ngenea_metadata", and the path of a remote object is "/mnt/fs_storage/path/to/file", the longest common ancestor path is "/mnt/fs_storage".
Therefore Ngenea tools will create the shadow metadata file "/mnt/fs_storage/ngenea_metadata/path/to/.file.xattr" for the remote object.
Warning
If a user manually deletes remote objects or folders, Ngenea tools will not delete corresponding shadow metadata files under PATH.
Over time, the directory PATH may contain orphaned shadow metadata files.
The parameter ShadowMetadataStore is applicable to the value "auto" or "shadow" of ObjectXAttrManipulationMode parameter.
The default value is "samedir".
Syntax
ShadowMetadataStore=<samedir|subdir|altdir:<PATH>>
Example
ShadowMetadataStore=subdir
EnsureMountPoint¶
Specifies the mount point that ngmigrate and ngrecall processes will validate prior to migrating or recalling a file.
Should the specified location not be the root of a mount point, the migration or recall will abort with an error.
This is an important safety feature to ensure that the local disk is not used in the event of a target mount failing.
It is typically set to the root of the mount point.
Syntax
EnsureMountPoint=/mountpoint
Example
EnsureMountPoint=/mnt/targetfs
ACLSave¶
Defines whether ngmigrate copies the ACLs of local files and directories to remote files directories or whether ACLs are saved metadata remote files and directories.
If the filesystem of a storage endpoint is of type GPFS, ngmigrate copies the ACLs of local files and directories to the ACLs of files and directories in the storage endpoint
If the filesystem of a storage endpoint is not of type GPFS and metadata is stored in native xattrs, ngmigrate saves the ACLs of local files and directories in native xattrs of files and directories in the storage endpoint
If the filesystem of a storage endpoint is not of type GPFS and metadata is stored in shadow metadata files, ngmigrate saves the ACLs of local files and directories in shadow metadata files for data files and directories in the storage endpoint
Refer to ObjectXAttrManipulationMode to specify where the metadata of files and directories is saved.
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
Spectra Logic BlackPearl Target¶
ArcaPix Ngenea HSM software is fully certified by Spectra Logic as an approved BlackPearl client.
Ngenea requires configuration and activation of the BlackPearl's Spectra S3 Service. Please refer to the relevant section of the BlackPearl User Guide.
The following example describes the information required to be present withing the Storage Target Configuration File to enable migration to the specified bucket.
Example Configuration File¶
[General]
Endpoint=https://IPADDRESS_OF_BLACKPEARL:443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
RemoteLocationXAttr=blackpearl:$1
RetrieveObjectName=$1
StoreObjectName=$1
CacheFullMaxWaitSeconds=300
CacheFullRetryAfterSeconds=20
CacheLoadRetryAfterSeconds=20
JobCompletionPollSeconds=20
JobCompletionDoWait=true
DeleteOnRecall=true
Configuration Keywords (Spectra Logic BlackPearl)¶
Keyword |
Default |
Required |
|---|---|---|
Endpoint |
None |
Yes |
AccessKeyId |
None |
Yes |
SecretAccessKey |
None |
Yes |
Bucket |
None |
Yes |
RetrieveObjectBasePath |
"" |
No |
RetrieveObjectName |
"$0" |
No |
StoreObjectBasePath |
"" |
No |
StoreObjectName |
"$0" |
No |
RemoteLocationXAttr |
"$0" |
No |
DeleteOnRecall |
False |
No |
MaxUploadSize |
10 MB |
No |
MaxUploadThreadCount |
Number of cores |
No |
MaxKeysListObjects |
1000 |
No |
MaxXAttrNativeMetadataSize |
0 |
No |
CacheLoadRetryAfterSeconds |
300 secs |
No |
CacheFullRetryAfterSeconds |
300 secs |
No |
CacheFullMaxWaitSeconds |
1800 secs |
No |
JobCompletionPollSeconds |
20 secs |
No |
JobCompletionDoWait |
False |
No |
ShadowFolderMetadataSave |
False |
No |
ACLSave |
False |
No |
SDKLogging |
False |
No |
Endpoint¶
Defines the Hostname or IP Address, protocol and port whereby Ngenea can access the BlackPearl Appliance.
Syntax
Endpoint=<http|https>://<Hostname or IP Address>[:port]
Example
Endpoint=https://blackpearl.example.com:443
AccessKeyId¶
Defines the S3 Access ID for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.
Syntax
AccessKeyId=<Spectra S3 Access Key>
Example
AccessKeyId=ABCDEFGHIJLKMNOPQRST
SecretAccessKey¶
The S3 Secret Key for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.
Syntax
SecretAccessKey=<Spectra S3 Secret Key>
Example
SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm
Bucket¶
Defines the Bucket Name of the target BlackPearl Bucket. This bucket must exist on the BlackPearl storage target prior to migrating data using Ngenea. THe bucket must be configured with the Versioning type set to Keep Latest. (see the Create a Data Policy section of the BlackPearl User Guide).
Syntax
Bucket=<Bucket Name>
Example
Bucket=MyBucketName
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetname:<LocalFileRegexReference>
Example
RemoteLocationXAttr=blackpearl:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
CacheFullMaxWaitSeconds¶
Defines the maximum number of seconds Ngenea waits for space to be made available in the BlackPearl Appliance cache for migration of new data prior to cancelling the current job if no space is made available.
Syntax
CacheFullMaxWaitSeconds=<time in seconds>
Example
CacheFullMaxWaitSeconds=300
CacheFullRetryAfterSeconds¶
Defines the interval in seconds upon which Ngenea will poll the BlackPearl Appliance to verify if the cache has free space.
Syntax
CacheFullRetryAfterSeconds=<time in seconds>
Example
CacheFullRetryAfterSeconds=20
CacheLoadRetryAfterSeconds¶
Defines the Interval in seconds upon which Ngenea will poll the BlackPearl Appliance to verify if retrieved files are present in the cache.
Syntax
CacheLoadRetryAfterSeconds=<time in seconds>
Example
CacheLoadRetryAfterSeconds=20
JobCompletionPollSeconds¶
Defines the interval in seconds upon which Ngenea will poll the BlackPearl Appliance to verify if an ongoing job has completed.
Syntax
JobCompletionPollSeconds=<time in seconds>
Example
JobCompletionPollSeconds=20
JobCompletionDoWait¶
Defines whether Ngenea waits for a BlackPearl Job to complete prior to exiting the current operation.
Setting this to 'true' will cause ngmigrate to wait until the BlackPearl has secured data to tape from the BlackPearl cache before completing.
Setting this to 'false' will cause ngmigrate to return once the data has been successfully uploaded to the BlackPearl cache.
Syntax
JobCompletionDoWait=<true|false>
Example
JobCompletionDoWait=true
MaxUploadSize¶
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate parts equal to this size (except the final part, which may be smaller).
The minimum value of this setting is 10485760 (10MB). The maximum value is determined by the BlackPearl target.
If this parameter is unset, the part sizes will be determined by the BlackPearl target (this is the recommended configuration).
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=10737418240
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the BlackPearl target.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "get bucket" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "get bucket" requests.
The default value for this parameter is 1000.
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file or directory as metadata elements.
Ngenea supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for BlackPearl is 4096.
A recommended setting for MaxXAttrNativeMetadataSize is 2048 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
DeleteOnRecall¶
Defines whether to delete the data from the remote BlackPearl target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
SDKLogging¶
Specifies a log level for the Spectra DS3 C SDK library.
Log messages are prefixed "BlackPearl: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<error|warn|info|debug|trace|false>
Boolean "True" is equivalent to "trace".
Example
SDKLogging=True
The default value is "False" which disables logging.
Azure Blob Storage Type¶
The following section describes how to setup an Azure storage target.
Azure Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and container.
Example Configuration File¶
[General]
StorageAccount=MyStorageAccount
AccessKey=MyAccessKey
Container=MyContainerName
Scheme=HTTPS
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True
Configuration Keywords (Azure Blob Storage)¶
Keyword |
Default |
Required |
|---|---|---|
AccessKey / CredentialsFile |
None |
Yes |
ACLSave |
False |
No |
Container |
None |
Yes |
DeleteOnRecall |
False |
No |
MaxDownloadSize |
20 MB |
No |
MaxDownloadThreadCount |
Number of cores |
No |
MaxKeysListObjects |
5000 |
No |
MaxRetryCount |
3 |
No |
EscapeNames |
True |
No |
MaxUploadSize |
20 MB |
No |
MaxUploadThreadCount |
Number of cores |
No |
MaxXAttrNativeMetadataSize |
0 |
No |
Port |
Null |
No |
RemoteLocationXAttr |
"$0" |
No |
RetrieveObjectBasePath |
"" |
No |
RetrieveObjectName |
"$0" |
No |
Scheme |
"https" |
No |
SDKLogging |
False |
No |
ShadowFolderMetadataSave |
False |
No |
StorageAccount |
None |
Yes |
StoreObjectBasePath |
"" |
No |
StoreObjectName |
"$0" |
No |
AccessKey / CredentialsFile¶
Defines the storage account access key used to authenticate to Azure.
For further information regarding Azure Account Access Keys please refer to the Microsoft Azure Documentation.
The AccessKey may be defined in the storage target configuration file with the keyword AccessKey, or in a separate file referenced by the keyword CredentialsFile.
Syntax
AccessKey=<Azure Account Access Key>
or
CredentialsFile=<file-name>
Example
AccessKey=ABCDEFGHIJLKMNOPQRST
or
CredentialsFile=access_key
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
Container¶
Defines the 'container' into which files are migrated. This container must exist on the Azure storage target prior to migrating data using Ngenea.
For further information regarding Azure Containers and Blobs please refer to the Microsoft Azure Documentation.
Syntax
Container=<Name of the Azure container>
Example
Container=MyContainerName
DeleteOnRecall¶
Defines whether to delete the data from the remote Azure target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
MaxDownloadSize¶
As for the uploads, downloads also use a multi-part transfer process. Files larger than 1MiB are downloaded in parts. This will split files into multiple parts and transfer each part separately.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller).
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).
This optional parameter defaults to 20MiB if not explicitly defined.
Syntax
MaxDownloadSize=<size_in_bytes>
Example
MaxDownloadSize=134217728
MaxDownloadThreadCount¶
Defines the maximum number of threads a ngrecall command will use when downloading files from the Azure Storage target. The threads download multiple files in parallel and download multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.
Syntax
MaxDownloadThreadCount=<# of threads>
Example
MaxDownloadThreadCount=8
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "list blobs" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "list blobs" requests.
The default value for this parameter is 5000.
MaxRetryCount¶
Specifies the number of retries of operations to access an Azure storage endpoint.
Every subsequent attempt to perform an operation is done after an increasing delay.
By default, a failed operation is retried up to 3 times.
Syntax
MaxRetryCount=<number of retries>
Example
MaxRetryCount=5
EscapeNames¶
Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.
If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.
Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.
Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.
The default value is True.
Syntax
EscapeNames=<True|False>
Example
EscapeNames=False
MaxUploadSize¶
Files larger than 5MiB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Azure server into one object.
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).
This optional parameter defaults to 20MiB if not explicitly defined.
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=104857600
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the Azure target. The threads upload multiple files in parallel and upload multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements.
Ngenea supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for Azure is 8192.
A recommended setting for MaxXAttrNativeMetadataSize is 4096 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
Port¶
For Azure Storage targets, do not include this parameter.
For non-Azure Storage targets, this parameter defines the TCP port to connect to on the remote storage.
The default value is 443 if this parameter is unspecified.
Syntax
Port:<TCP port number>
Example
Port:443
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=target1:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
Scheme¶
Defines the transport protocol used to communicate with the Azure target.
ArcaStream / Pixit Media only supports HTTPS protocol for Ngenea solutions.
This optional parameter defaults to HTTPS if not explicitly defined.
Syntax
Scheme=<PROTOCOL>
Example
Scheme=HTTPS
SDKLogging¶
Specifies a log level for the Azure Storage C++ SDK library. If a master configuration file defines multiple Azure storage endpoints, then a log level specified for the first accessed Azure storage endpoint will be in effect.
Log messages are prefixed "Azure: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<error|warn|info|verbose|false>
Boolean "True" is equivalent to "verbose".
The default value is "False" which disables logging.
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
StorageAccount¶
Defines the storage account name used to authenticate to Azure.
For further information regarding Azure storage accounts please refer to the Microsoft Azure Documentation.
Syntax
StorageAccount=<Account Name>
Example
StorageAccount=MyStorageAccount
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
Security¶
The permissions listed in the table below must be enabled to allow Ngenea to correctly function. These permissions are included in Azure's built-in role Storage Blob Data Contributor.
Blob Service Operation |
Description |
|---|---|
List Blobs |
read (limited to the container) |
Put Blob |
write |
Get Blob |
read blob |
Get Blob Properties |
read blob properties (blob size, MIME type} |
Set Blob Properties |
write or update blob properties (MIME type) |
Get Blob Metadata |
read Ngenea-specific attributes |
Set Blob Metadata |
write / update Ngenea-specific attributes |
Delete Blob |
delete the blob |
Put Block |
write a block (part of a multi-block operation) |
Put Block List |
submit blocklist (completes a multi-block operation) |
Google Cloud Storage Type¶
The following section describes how to setup a Google Cloud Storage target.
Google Cloud Storage targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and bucket.
Example Configuration File¶
[General]
CredentialsFile=MyCredentialsFile
( or CredentialsJSON="multi-line JSON" )
Bucket=MyBucketName
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True
Configuration Keywords (Google Cloud Storage)¶
Keyword |
Default |
Required |
|---|---|---|
Endpoint |
Null |
No |
OAuth2Endpoint |
Null |
No |
CredentialsFile / CredentialsJSON |
None |
No |
CredentialsJSON |
None |
No |
Bucket |
None |
Yes |
StorageClass |
Null |
No |
RetrieveObjectBasePath |
"" |
No |
RetrieveObjectName |
"$0" |
No |
StoreObjectBasePath |
"" |
No |
StoreObjectName |
"$0" |
No |
RemoteLocationXAttr |
"$0" |
No |
DeleteOnRecall |
False |
No |
MaxUploadSize |
20 MB |
No |
MaxDownloadSize |
20 MB |
No |
MaxUploadThreadCount |
Number of cores |
No |
MaxDownloadThreadCount |
Number of cores |
No |
MaxXAttrNativeMetadataSize |
0 |
No |
RequestTimeoutMs |
300000 |
No |
EscapeNames |
True |
No |
ShadowFolderMetadataSave |
False |
No |
ACLSave |
False |
No |
Endpoint¶
This parameter defines the Hostname to be used by Ngenea to access on-premises Google Storage.
For access to Google's public cloud storage, do not include this parameter.
Syntax
Endpoint=<Hostname>
Example
Endpoint=storage-vialink1.p.googleapis.com
OAuth2Endpoint¶
This parameter defines the Hostname to be used by Ngenea to perform OAuth2 authentication.
The Endpoint parameter must also be included.
Syntax
Endpoint=<Hostname>
OAuth2Endpoint=<OAuth-Hostname>
Example
Endpoint=storage-vialink1.p.googleapis.com
OAuth2Endpoint=oauth2-vialink1.p.googleapis.com
CredentialsFile / CredentialsJSON¶
Defines the Google Cloud Storage authentication credentials. CredentialsFile specifies the path to a file containing the credentials (a JSON file). CredentialsJSON specifies the credentials as a multi-line JSON string.
If neither of these keywords is specified, the Ngenea applications use the file specified in the environment variable GOOGLE_APPLICATION_CREDENTIALS.
For further information regarding Google Cloud Storage accounts please refer to the Google Cloud Storage Documentation.
Syntax
CredentialsFile=<path to credentials file>
or
CredentialsJSON=<multi-line JSON credentials>
Example:
CredentialsFile=/user/cred.json
or
CredentialsJSON={
"type": "service_account",
...
}
Bucket¶
Defines the 'bucket' into which files are migrated. This bucket must exist on the Google Cloud Storage target prior to migrating data using Ngenea.
For further information regarding Google Cloud Storage Buckets and Blobs please refer to the Google Cloud Storage Documentation.
Syntax
Bucket=<Name of the Google Cloud Storage bucket>
Example
Bucket=MyBucketName
StorageClass¶
Defines a storage class for objects uploaded to the target bucket.
Valid storage classes include STANDARD, NEARLINE, COLDLINE, and ARCHIVE.
Example
StorageClass=ARCHIVE
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=target1:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
MaxUploadSize¶
Files larger than 5MiB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Google Cloud Storage server into one object.
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 5242880 (5MiB) and 5368709120 (5GiB).
This optional parameter defaults to 20MiB if not explicitly defined.
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=134217728
MaxDownloadSize¶
As for the uploads, downloads also use a multi-part transfer process. Files larger than 5MiB are downloaded in parts. This will split files into multiple parts and transfer each part separately.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller).
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 5242880 (5MiB) and 5368709120 (5GiB).
This optional parameter defaults to defaults to 20MiB if not explicitly defined.
Syntax
MaxDownloadSize=<size_in_bytes>
Example
MaxDownloadSize=134217728
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the Google Cloud Storage target. The threads upload multiple files in parallel and upload multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxDownloadThreadCount¶
Defines the maximum number of threads a ngrecall command will use when downloading files from the Google Cloud Storage target. The threads download multiple files in parallel and download multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.
Syntax
MaxDownloadThreadCount=<# of threads>
Example
MaxDownloadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements.
Ngenea supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for Google Cloud Storage is 8192.
A recommended setting for MaxXAttrNativeMetadataSize is 4096 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
RequestTimeoutMs¶
Specifies the timeout (in milliseconds) for transferring data from or to a Google storage endpoint.
If the speed of a data transfer via a socket connection is too slow during that time period, then the data transfer is aborted, and the socket connection is closed.
Default parameter value is 300000.
Syntax
RequestTimeoutMs=<timeout in milliseconds>
Example
RequestTimeoutMs=60000
DeleteOnRecall¶
Defines whether to delete the data from the remote Google Cloud Storage target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
EscapeNames¶
Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.
If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.
Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.
Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.
The default value is True.
Syntax
EscapeNames=<True|False>
Example
EscapeNames=False
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
Security¶
The permissions listed in this table must be enabled to allow Ngenea to correctly function.
See Google Cloud Storage - APIs & reference (IAM permissions for Cloud Storage).
Object Permission Name |
Description |
|---|---|
storage.buckets.get |
Read bucket metadata, excluding IAM policies. |
storage.objects.create |
Add new objects to a bucket. |
storage.objects.delete |
Delete objects. |
storage.objects.get |
Read object data and metadata, excluding ACLs. |
storage.objects.list |
List objects in a bucket. Also read object metadata, |
excluding ACLs, when listing. |
|
storage.objects.update |
Update object metadata, excluding ACLs. |
Multi-Target Support¶
The following section describes how to configure Ngenea to migrate and recall to multiple Storage Targets in parallel.
With reference to the Master Configuration File example in General Configuration which demonstrates a configuration to migrate data from two differing directories to associated targets respectively, Ngenea can migrate data from the same directory or even specific individual files to multiple targets.
Such a configuration can help ensure data can be recalled if a storage target is unavailable, for instance due to a disaster or a provider service outage.
Example Whole Filesystem to Multiple-Targets Configuration¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | [General]
MinMigrateEndpointCount=0
[Storage awss3]
StorageType=AmazonS3
StorageKey=01
ConfigFile=awss3.conf
LocalFileRegex=/mmfs1/(.+)
RemoteLocationXAttrRegex=awss3:(.+)
[Storage azure]
StorageType=Azure
StorageKey=02
ConfigFile=azure.conf
RemoteLocationXAttrRegex=azure:(.+)
LocalFileRegex=/mmfs1/(.+)
|
File Matching to Multiple Targets¶
Ngenea matches a file to be migrated against the LocalFileRegex parameter of each Storage Target to determine if the file should be migrated to the target. To migrate files to multiple storage target endpoints ensure that the LocalFileRegex either specifically targets or overlaps the file's path for each relevant storage target.
As per the above example, specifying LocalFileRegex=/mmfs1/(.+) for all targets will ensure that all files within the /mmfs1 directory tree are migrated to both targets.
Alternatively, more granular control can be applied.
Given two storage targets (A) and (B) with differing values for LocalFileRegex parameters:
A: LocalFileRegex=/mmfs1/(.+)
B: LocalFileRegex=/mmfs1/archive/(.+)
After successful migration the first Storage Target (A) would contain all files under the /mmfs1 directory structure. The second Storage Target (B) will only contain files within the /mmfs1/archive directory structure. Thus resulting in identically migrated files within the /mmfs1/archive directory structure on both Storage Targets.
Recall Behaviour¶
Ngenea will recall migrated files in top-down order of the Storage Target stanzas in the Master Configuration File which match the criteria for the file. Should a storage target not be able to provide the file for recall, Ngenea will use the next matching Storage Target in top-down order - and so on, until all Storage Targets have been exhausted.
Multi-Target Specific Configuration Keywords¶
MinMigrateEndpointCount
StorageKey
MinMigrateEndpointCount¶
Defines the number of associated endpoints a file needs to be migrated to in order for the migration to be considered a success.
If the "MinMigrateEndpointCount" parameter is equal to 0 or is greater than or equal to the number of associated storage endpoints for the file, and the file was successfully migrated to all those endpoints, then Ngenea considers the file migrated successfully.
If the "MinMigrateEndpointCount" parameter is greater than 0 and is less than the number of associated storage endpoints for file, and the file was successfully migrated to the minimum number of endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migrated successfully.
If the file was successfully migrated to at least one endpoint associated with the file but less than the minimum number of storage endpoints as defined by "MinMigrateEndpointCount", then Ngenea considers the file migration partially successful.
If the file was not successfully migrated to any endpoint, Ngenea considers the migration of the file as failed.
On exit, Ngenea ngmigrate utility returns the following exit statuses:
0 - all files were migrated successfully
1 - all files failed to migrate successfully
2 - some files failed to migrate successfully
Syntax
MinMigrateEndpointCount=<Non-negative integer value>
Example
MinMigrateEndpointCount=2
StorageKey¶
Defines a custom identifier attributable to the storage endpoint.
Every [Storage] section of the Master Configuration File can contain the "StorageKey" parameter. The parameter's value must be a two-byte string not equal to the characters "sz". The default value of the parameter is "lc".
When implementing multi-target migration for files, each storage endpoint must have the parameter defined with a unique key.
Syntax
StorageKey=<char><char>
Examples
StorageKey=16
StorageKey=zz
StorageKey=3a