Migrating

Policy Based Migration

A Spectrum Scale/GPFS policy is typically used to migrate files. A policy will select candidates for migration based on various criteria, and then call ngmigrate in the execution phase to migrate files to external storage.

Example Migration Policy

This simple example policy will migrate files which have not been accessed in 180 days to free up space.

define(
    exclude_list,
    (
        PATH_NAME LIKE '%/.ctdb/%'
        OR NAME LIKE 'user.quota%'
        OR NAME LIKE 'fileset.quota%'
        OR NAME LIKE 'group.quota%'
    )
)

define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%'))

/* All files use the ngenea.conf configuration file:*/
RULE EXTERNAL POOL 'NGENEA_DEFAULT'
    EXEC '/var/mmfs/etc/mmpolicyExec-ngenea-hsm'
    OPTS '--config-file=/opt/arcapix/etc/ngenea.conf -v1'

RULE 'ngenea_migrate' MIGRATE TO POOL 'NGENEA_DEFAULT'
/* If Filesystem is running out of space (more than 85% full)
    reduce usage to 70% */
THRESHOLD(85,70,70)

/* Choose files least recently accessed */
WEIGHT(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))

/* but only migrate files > 1MB in size */
WHERE KB_ALLOCATED > 1024
/* Don't migrate anything which has been accessed in the last 180 days */
AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) > 180)

AND NOT (is_migrated)
AND NOT (exclude_list)

A more comprehensive example can be found here.

ngmigrate

Synopsis

ngmigrate [-vLEVEL] [--config-file=FILE] [-p]
          [-E PATTERN1] [--endpoint-exclude=PATTERN2] FILE1 ... FILEn
ngmigrate [-vLEVEL] [--config-file=FILE] --default-stub-size=LENGTH
          [-E PATTERN1] [--endpoint-exclude=PATTERN2] FILE1 ... FILEn
ngmigrate [-vLEVEL] [--config-file=FILE] --force-stub-size=LENGTH
          [-E PATTERN1] [--endpoint-exclude=PATTERN2] FILE1 ... FILEn

Description

Migrates files from the local GPFS file system to a Storage Target.

Options

--config-file=FILE
                path to a configuration file to use.
                Default: /opt/arcapix/etc/ngenea.conf
--default-stub-size=LENGTH
                default approximate length of a beginning file segment that
                should be retained during file migration. This setting can be
                overridden in a configuration file.
                Default: 0 (free up the entire file content).
-E, --endpoint=PATTERN
                restrict the set of endpoints to interact with to endpoints
                with aliases specified by an extended glob pattern.
--endpoint-exclude=PATTERN
                remove endpoints with aliases specified by an extended glob
                pattern from the set of endpoints to interact with.
--force-stub-size=LENGTH
                retain a segment of every migrated file starting from its
                beginning and having a specified approximate length in bytes.
--help          display this help and exit.
-p, --premigrate
                retain the content of every migrated file and do not set the
                OFFLINE flag for the file.
-v, --verbose[=LEVEL]
                verbosity level:
                0 = standard messages (also used when this option is absent);
                1 = additional informational and warning messages (default);
                2 = debug messages;
                3 = enable core dump and messages related to file locking;
                    print PID and current time with microsecond precision.
-V, --version   display version information and exit.

Examples

To migrate a file to the associated Storage Target:

ngmigrate /mmfs1/data/file1

To migrate a file to a storage target, using a custom configuration file (which may redefine the storage target) to replace the default configuration file:

ngmigrate --config-file=/path/to/custom.conf /mmfs1/data/file1

To migrate multiple files to their associated Storage Targets:

ngmigrate /mmfs1/data/file1 /mmfs1/data/file2

To migrate all files in directory /mmfs1/data/ starting with name file to the associated Storage Target:

ngmigrate /mmfs1/data/file*

To migrate all files starting with name file and name newfile in directory /mmfs1/data/ to the associated Storage Target:

ngmigrate /mmfs1/data/file* /mmfs1/data/newfile*

To migrate all files, except hidden ("dot") files, within a directory to the associated Storage Target:

ngmigrate /mmfs1/data/*

To migrate all files, including hidden ("dot") files, within a directory to the associated Storage Target:

ngmigrate /mmfs1/data/{.??,}*

To migrate all files, except hidden ("dot") files, within two different directories to their associated Storage Targets:

ngmigrate /mmfs1/data/dir1/* /mmfs1/data/dir2/*

Matching Storage Endpoints

The options --endpoint / --endpoint-exclude are used to include / exclude some endpoints.

Example, to migrate files to those Storage Targets which match an extended glob pattern :

ngmigrate --endpoint='awss3-*|fs-*' /mmfs1/data/dir1/*

Only storage endpoints which match the pattern will be taken into consideration.

To migrate files to those Storage Targets which fail to match an extended glob pattern :

ngmigrate --endpoint=exclude='awss3-*|fs-*' /mmfs1/data/dir1/*

Only storage endpoints which don't match the pattern will be taken into consideration.

See Also