Querying

Listing file status

The ngenea status of files can be easily queried with the ngls tool.

For example:

# ngls sample.txt
  SIZE    RSS ST FILE NAME
    29      0 m sample.txt

The information displayed shows:

  • SIZE: file size (in bytes)
  • RSS: resident size on disk (in bytes)
  • ST: migration state: m for migrated, - for resident, p for premigrated (resident at both ends)
  • FILE NAME: file name

ngls

Synopsis

ngls [-h | --help] [-V | --version] [-H | --header] [-n | -a] [ -l ] [-d N | --depth=N]
    | [--output='format-specifier' || --output=ALL ]
    | [-m pattern | --match=pattern] file-or-dir-paths

Description

Lists Ngenea status of files.

Options

-n, -a                  natural(-n) or alphabetical (-a) sort of file names
                        natural sort is alphabetical, with embedded version
                        numbers sorted numerically
-l                      display remote location
-d                      same as "-d1" or "-depth=1"
-dNNN, --depth=NNN      descend directories to a depth of NNN
-D                      same as "-d15"
-H, --header            don't print the directory header line
--output='format-specifier'
                        output in the specified format, using printf-style
                        format specifiers
--output=ALL            same as --output='%10s %6r %8p %08w %S %-50n %l'
                           %s file size
                            %r resident size
                            %p file permissions
                            %w win attrs
                            %S migration state ('-', 'm=migrated', 'p=premigrated')
                            %n file name
                            %l remote location
                        default is --output='%10s %6r %S %-50n'
                        or '%10s %6r %S %-50n %l' if -l also selected
-m'glob',--match='glob' match the filename against the glob string
-r'regex', --regex='regex'
                        match the filename against the regular expression
                        (note: directory names are not matched)

-?, --help                  display this help and exit.
-V, --version               display version information and exit.

Examples

Default Output Format

Display data for jpg files in the current directory

ngls *.jpg

Display data for JPG files in the current directory and one level below

ngls -d --match='*.jpg'

Display data for JPG files in the current directory and 5 levels below

ngls -d5 --match='*.jpg'

Display all data for files in the current directory.

ngls -o all
    SIZE    RSS     PERM   ATTRS  ST FILE NAME                        REMOTE
. : 7 items
1048576     1024   100644 00000001  p File_1.dat                      tg1:data/File_1.dat
1048576        0   100644 00000101  m File_2.dat                      tg1:data/File_2.dat
    <dir>      1    40755 00000008  - images
    <dir>      4    40755 00000008  - data
36375247   35584   100644 00000001  - sample_data.tgz

Natural / Alphabetic sort

The default sort option is -n - "natural" sort order. This is useful when listing filenames containing a version number. Compare:

ngls images
    SIZE    RSS ST FILE NAME
images : 5 items
2226725   2176 - images/IMG_V3.5.jpg
2371737   2432 - images/IMG_V3.23.jpg
2068990   2048 - images/IMG_V3.47.jpg
1958986   1920 - images/IMG_V9.1.jpg
2706537   2688 - images/IMG_V12.1.jpg

with an alphabetic sort (-a option):

ngls -a images
    SIZE    RSS ST FILE NAME
images : 5 items
2706537   2688 - images/IMG_V12.1.jpg
2371737   2432 - images/IMG_V3.23.jpg
2068990   2048 - images/IMG_V3.47.jpg
2226725   2176 - images/IMG_V3.5.jpg
1958986   1920 - images/IMG_V9.1.jpg

Custom Output Format

Display all data with '||' separator

ngls -H -o '%s||%r||%p||%w||%S||%n||%l'
1048576||1024||  100644||00000001 ||p||File_1.dat||tg1:data/File_1.dat
1048576||0||  100644||00000101 ||m||File_2.dat||tg1:data/File_2.dat
<dir>||1||   40755||00000008 ||-||images||
<dir>||4||   40755||00000008 ||-||data||
36375247||35584||  100644||00000001 ||-||sample_data.tgz||

The -H option suppresses the directory header.

Display selected items in comma-separated format, including sub-directories

ngls --output='%n,%s,%r' -d
. : 7 items
File_1.dat,1048576,1024
File_2.dat,1048576,0
images,<dir>,1
data,<dir>,4
sample_data.tgz,36375247,35584
images : 0 items
data : 5 items
data/IMG_4625.jpg,2706537,2688
data/IMG_4626.jpg,1958986,1920
data/IMG_4650.jpg,2226725,2176
data/IMG_4686.jpg,2068990,2048
data/IMG_4694.jpg,2371737,2432

See Also

Querying extended attributes

All ngenea metadata associated with a file's migration are stored as extended attributes within the file inode.

As such, standard Spectrum Scale/GPFS tools can be used to show this information.

For example - mmlsattr can be used to show the migration information about a file:

# mmlsattr -d -L sample.txt
file name:            sample.txt
metadata replication: 2 max 2
data replication:     1 max 3
immutable:            no
appendOnly:           no
flags:
storage pool name:    sas1
fileset name:         root
snapshot name:
creation time:        Wed May 30 16:54:18 2018
Misc attributes:      ARCHIVE OFFLINE
Encrypted:            no
dmapi.APXguuid:       "a7e82a60-62ae-4c25-8da6-91d05f265714"
dmapi.APXrmtlc:       "ngenea_target1:test/sample.txt"
dmapi.APXsh512:       "0cfcd93fb5eae3f5c9962117b38882de79ea4f9c432d574e8c9a4b1b65cded4620d25dc1b29ef0e60d238219797cb4f7d3946ce3a0f718b849cff2ae41fffd1c"
dmapi.APXrmtsz:       "29"
gpfs.dmapi.region:    0xFFFFFFFFFFFFFFFF00000000000000000700000000000000

The relevant extended attributes include:

  • Misc attributes: the presence of the OFFLINE status here shows that the file is migrated, and is no longer resident on the file system
  • dmapi.APXguuid: Unique ID for the file, used to authoritatively match an inode to a remote object
  • dmapi.APXrmtlc: Remote location of the file, as defined in the RemoteLocationXAttrRegex setting for the activated storage target in the Master Configuration File, at the point it was migrated.
  • dmapi.APXsh512: SHA512 checksum of the file at the point it was migrated.
  • dmapi.APXrmtsz: Size of the object sent to the external storage.
  • gpfs.dmapi.region: Internal Spectrum Scale/GPFS attribute which reflects the offline status of the file

Note that these attributes must not be changed, except under explicit direction from ArcaStream / Pixit Media support. Doing so may result in data becoming unavailable, and may even result in data corruption/loss.