Policy Based Migration¶
A Spectrum Scale/GPFS policy is typically used to migrate files. A policy will select candidates for migration based on various criteria, and then call ngmigrate in the execution phase to migrate files to external storage.
Example Migration Policy¶
This simple example policy will migrate files which have not been accessed in 180 days to free up space.
define( exclude_list, ( PATH_NAME LIKE '%/.ctdb/%' OR NAME LIKE 'user.quota%' OR NAME LIKE 'fileset.quota%' OR NAME LIKE 'group.quota%' ) ) define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%')) /* All files use the ngenea.conf configuration file:*/ RULE EXTERNAL POOL 'NGENEA_DEFAULT' EXEC '/var/mmfs/etc/mmpolicyExec-ngenea-hsm' OPTS '--config-file=/opt/arcapix/etc/ngenea.conf -v1' RULE 'ngenea_migrate' MIGRATE TO POOL 'NGENEA_DEFAULT' /* If Filesystem is running out of space (more than 85% full) reduce usage to 70% */ THRESHOLD(85,70,70) /* Choose files least recently accessed */ WEIGHT(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) /* but only migrate files > 1MB in size */ WHERE KB_ALLOCATED > 1024 /* Don't migrate anything which has been accessed in the last 180 days */ AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) > 180) AND NOT (is_migrated) AND NOT (exclude_list)
A more comprehensive example can be found here.
ngmigrate [--config-file=FILE] FILE1 ... FILEn
Migrates files from the local GPFS file system to a Storage Target.
--config-file=FILE path to a configuration file to use. Default: /opt/arcapix/etc/ngenea.conf --default-stub-size=LENGTH default approximate length of a beginning file segment that should be retained during file migration. This setting can be overridden in a configuration file. Default: 0 (free up the entire file content). --force-stub-size=LENGTH retain a segment of every migrated file starting from its beginning and having a specified approximate length in bytes. --help display this help and exit. -p, --premigrate retain the content of every migrated file and do not set the OFFLINE flag for the file. -v, --verbose[=LEVEL] verbosity level: 0 = standard messages (also used when this option is absent); 1 = additional informational and warning messages (default); 2 = debug messages; 3 = enable core dump and messages related to file locking; print PID and current time with microsecond precision. -V, --version display version information and exit.
To migrate a file to the associated Storage Target:
To migrate a file to a storage target, overriding the current config with (E.G. a custom storage target):
ngmigrate --config-file=/path/to/custom.conf /mmfs1/data/file1
To migrate multiple files to their associated Storage Target:
ngmigrate /mmfs1/data/file1 /mmfs1/data/file2
To migrate all files in directory
/mmfs1/data/ starting with name
file to the associated Storage Target:
To migrate all files starting with name
file and name
newfile in directory
/mmfs1/data/ to the associated Storate Target:
ngmigrate /mmfs1/data/file* /mmfs1/data/newfile*
To migrate all files, except hidden ("dot") files, within a directory to the associated Storage Target:
To migrate all files, including hidden ("dot") files, within a directory to the associated Storage Target:
To migrate all files, except hidden ("dot") files, within two different directories to their associated Storage Target(s):
ngmigrate /mmfs1/data/dir1/* /mmfs1/data/dir2/*