Migrating

Policy Based Migration

A Spectrum Scale/GPFS policy is typically used to migrate files. A policy will select candidates for migration based on various criteria, and then call ngmigrate in the execution phase to migrate files to external storage.

Example Migration Policy

This simple example policy will migrate files which have not been accessed in 180 days to free up space.

define(
    exclude_list,
    (
        PATH_NAME LIKE '%/.ctdb/%'
        OR NAME LIKE 'user.quota%'
        OR NAME LIKE 'fileset.quota%'
        OR NAME LIKE 'group.quota%'
    )
)

define(is_migrated, (MISC_ATTRIBUTES LIKE '%V%'))

/* All files use the ngenea.conf configuration file:*/
RULE EXTERNAL POOL 'NGENEA_DEFAULT'
    EXEC '/var/mmfs/etc/mmpolicyExec-ngenea-hsm'
    OPTS '--config-file=/opt/arcapix/etc/ngenea.conf -v1'

RULE 'ngenea_migrate' MIGRATE TO POOL 'NGENEA_DEFAULT'
/* If Filesystem is running out of space (more than 85% full)
    reduce usage to 70% */
THRESHOLD(85,70,70)

/* Choose files least recently accessed */
WEIGHT(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))

/* but only migrate files > 1MB in size */
WHERE KB_ALLOCATED > 1024
/* Don't migrate anything which has been accessed in the last 180 days */
AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) > 180)

AND NOT (is_migrated)
AND NOT (exclude_list)

A more comprehensive example can be found here.

ngmigrate

Synopsis

ngmigrate [--config-file=FILE] FILE1 ... FILEn

Description

Migrates files from the local GPFS file system to a Storage Target.

Options

--config-file=FILE
                path to a configuration file to use.
                Default: /opt/arcapix/etc/ngenea.conf
--default-stub-size=LENGTH
                default approximate length of a beginning file segment that
                should be retained during file migration. This setting can be
                overridden in a configuration file.
                Default: 0 (free up the entire file content).
--force-stub-size=LENGTH
                retain a segment of every migrated file starting from its
                beginning and having a specified approximate length in bytes.
--help          display this help and exit.
-p, --premigrate
                retain the content of every migrated file and do not set the
                OFFLINE flag for the file.
-v, --verbose[=LEVEL]
                verbosity level:
                0 = standard messages (also used when this option is absent);
                1 = additional informational and warning messages (default);
                2 = debug messages;
                3 = enable core dump and messages related to file locking;
                    print PID and current time with microsecond precision.
-V, --version   display version information and exit.

Examples

To migrate a file to the associated Storage Target:

ngmigrate /mmfs1/data/file1

To migrate a file to a storage target, overriding the current config with (E.G. a custom storage target):

ngmigrate --config-file=/path/to/custom.conf /mmfs1/data/file1

To migrate multiple files to their associated Storage Target:

ngmigrate /mmfs1/data/file1 /mmfs1/data/file2

To migrate all files in directory /mmfs1/data/ starting with name file to the associated Storage Target:

ngmigrate /mmfs1/data/file*

To migrate all files starting with name file and name newfile in directory /mmfs1/data/ to the associated Storate Target:

ngmigrate /mmfs1/data/file* /mmfs1/data/newfile*

To migrate all files, except hidden ("dot") files, within a directory to the associated Storage Target:

ngmigrate /mmfs1/data/*

To migrate all files, including hidden ("dot") files, within a directory to the associated Storage Target:

ngmigrate /mmfs1/data/{.??,}*

To migrate all files, except hidden ("dot") files, within two different directories to their associated Storage Target(s):

ngmigrate /mmfs1/data/dir1/* /mmfs1/data/dir2/*

See Also