Querying¶
Listing file status¶
The ngenea status of files can be easily queried with the ngls tool.
For example:
# ngls sample.txt
SIZE RSS ST FILE NAME
29 0 m sample.txt
The information displayed shows:
- SIZE: file size (in bytes)
- RSS: resident size on disk (in bytes)
- ST: migration state: m for migrated, - for resident, p for premigrated (resident at both ends)
- FILE NAME: file name
ngls¶
Synopsis¶
ngls [-h | --help] [-V | --version] [-H | --header] [-n | -a] [ -l ] [-d N | --depth=N]
| [--output='format-specifier' || --output=ALL ]
| [-m pattern | --match=pattern] file-or-dir-paths
Description¶
Lists Ngenea status of files.
Options¶
-n, -a natural(-n) or alphabetical (-a) sort of file names
natural sort is alphabetical, with embedded version
numbers sorted numerically
-l display remote location
-d same as "-d1" or "-depth=1"
-dNNN, --depth=NNN descend directories to a depth of NNN
-D same as "-d15"
-H, --header don't print the directory header line
--output='format-specifier'
output in the specified format, using printf-style
format specifiers
--output=ALL same as --output='%10s %6r %8p %08w %S %-50n %l'
%s file size
%r resident size
%p file permissions
%w win attrs
%S migration state ('-', 'm=migrated', 'p=premigrated')
%n file name
%l remote location
default is --output='%10s %6r %S %-50n'
or '%10s %6r %S %-50n %l' if -l also selected
-m'glob',--match='glob' match the filename against the glob string
-r'regex', --regex='regex'
match the filename against the regular expression
(note: directory names are not matched)
-?, --help display this help and exit.
-V, --version display version information and exit.
Examples¶
Default Output Format¶
Display data for jpg files in the current directory
ngls *.jpg
Display data for JPG files in the current directory and one level below
ngls -d --match='*.jpg'
Display data for JPG files in the current directory and 5 levels below
ngls -d5 --match='*.jpg'
Display all data for files in the current directory.
ngls -o all
SIZE RSS PERM ATTRS ST FILE NAME REMOTE
. : 7 items
1048576 1024 100644 00000001 p File_1.dat tg1:data/File_1.dat
1048576 0 100644 00000101 m File_2.dat tg1:data/File_2.dat
<dir> 1 40755 00000008 - images
<dir> 4 40755 00000008 - data
36375247 35584 100644 00000001 - sample_data.tgz
Natural / Alphabetic sort¶
The default sort option is -n - "natural" sort order. This is useful when listing filenames containing a version number. Compare:
ngls images
SIZE RSS ST FILE NAME
images : 5 items
2226725 2176 - images/IMG_V3.5.jpg
2371737 2432 - images/IMG_V3.23.jpg
2068990 2048 - images/IMG_V3.47.jpg
1958986 1920 - images/IMG_V9.1.jpg
2706537 2688 - images/IMG_V12.1.jpg
with an alphabetic sort (-a option):
ngls -a images
SIZE RSS ST FILE NAME
images : 5 items
2706537 2688 - images/IMG_V12.1.jpg
2371737 2432 - images/IMG_V3.23.jpg
2068990 2048 - images/IMG_V3.47.jpg
2226725 2176 - images/IMG_V3.5.jpg
1958986 1920 - images/IMG_V9.1.jpg
Custom Output Format¶
Display all data with '||' separator
ngls -H -o '%s||%r||%p||%w||%S||%n||%l'
1048576||1024|| 100644||00000001 ||p||File_1.dat||tg1:data/File_1.dat
1048576||0|| 100644||00000101 ||m||File_2.dat||tg1:data/File_2.dat
<dir>||1|| 40755||00000008 ||-||images||
<dir>||4|| 40755||00000008 ||-||data||
36375247||35584|| 100644||00000001 ||-||sample_data.tgz||
The -H option suppresses the directory header.
Display selected items in comma-separated format, including sub-directories
ngls --output='%n,%s,%r' -d
. : 7 items
File_1.dat,1048576,1024
File_2.dat,1048576,0
images,<dir>,1
data,<dir>,4
sample_data.tgz,36375247,35584
images : 0 items
data : 5 items
data/IMG_4625.jpg,2706537,2688
data/IMG_4626.jpg,1958986,1920
data/IMG_4650.jpg,2226725,2176
data/IMG_4686.jpg,2068990,2048
data/IMG_4694.jpg,2371737,2432
Querying extended attributes¶
All ngenea metadata associated with a file's migration are stored as extended attributes within the file inode.
As such, standard Spectrum Scale/GPFS tools can be used to show this information.
For example - mmlsattr can be used to show the migration information about a file:
# mmlsattr -d -L sample.txt
file name: sample.txt
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sas1
fileset name: root
snapshot name:
creation time: Wed May 30 16:54:18 2018
Misc attributes: ARCHIVE OFFLINE
Encrypted: no
dmapi.APXguuid: "a7e82a60-62ae-4c25-8da6-91d05f265714"
dmapi.APXrmtlc: "ngenea_target1:test/sample.txt"
dmapi.APXsh512: "0cfcd93fb5eae3f5c9962117b38882de79ea4f9c432d574e8c9a4b1b65cded4620d25dc1b29ef0e60d238219797cb4f7d3946ce3a0f718b849cff2ae41fffd1c"
dmapi.APXrmtsz: "29"
gpfs.dmapi.region: 0xFFFFFFFFFFFFFFFF00000000000000000700000000000000
The relevant extended attributes include:
- Misc attributes: the presence of the OFFLINE status here shows that the file is migrated, and is no longer resident on the file system
- dmapi.APXguuid: Unique ID for the file, used to authoritatively match an inode to a remote object
- dmapi.APXrmtlc: Remote location of the file, as defined in the RemoteLocationXAttrRegex setting for the activated storage target in the Master Configuration File, at the point it was migrated.
- dmapi.APXsh512: SHA512 checksum of the file at the point it was migrated.
- dmapi.APXrmtsz: Size of the object sent to the external storage.
- gpfs.dmapi.region: Internal Spectrum Scale/GPFS attribute which reflects the offline status of the file
Note that these attributes must not be changed, except under explicit direction from ArcaStream / Pixit Media support. Doing so may result in data becoming unavailable, and may even result in data corruption/loss.