Querying¶
Listing file status¶
The Ngenea status of files can be easily queried with the ngls tool.
For example:
# ngls sample.txt
SIZE RSS ST FILE NAME
29 0 m sample.txt
The information displayed shows:
SIZE: file size (in bytes)
RSS: resident size on disk (in bytes)
ST: migration state: m for migrated, - for resident, p for premigrated (resident at both ends)
FILE NAME: file name
ngls¶
Synopsis¶
ngls [--help] [-V | --version] [-H | --no-header] [-n | -a]
[-d N | --depth=N] [--output='format-specifier' | --output=ALL ]
[--quoting-style=escape] [--json[=pretty]]
[-h | --human-readable ] [ --si ]
[ -l ] [--print-xattrs] [--block-size=BlockSize|auto|0]
[--size-width=n] || [--rss-width=n] | [--width=n]
[-m pattern | --match=pattern] [--type=f|d]
[--state=STRING ] file-or-dir-paths
Description¶
Lists Ngenea status of files.
Options¶
-n, -a natural(-n) or alphabetical (-a) sort of file names
natural sort displays embedded version numbers numerically
-l display remote locations
--print-xattrs display all extended attributes
-dNNN, --depth=NNN descend directories to a depth of NNN
-D same as "-d15"
-h, --human-readable output file size in human-readable form
auto-scaling unless blocksize is specified
--si select human-readable form, with SI units for file sizes
1K=1000 bytes, 1M=1,000,000 etc
-z,--blocksize=<size> Specifies the units for SIZE and RSS columns
Default is 1K
<size> may be "auto" or 0, to scale to the largest unit
or n, to set the unit to n bytes
or nX , X is K,M,G,T,P 1K=1024 bytes, 1M=1048576 etc
or nXB , to use SI units
--size-width=n Set the width of the SIZE column to n
--rss-width=n Set the width of the RSS column to n
--width=n Set the widths of the SIZE and RSS columns to n
-H, --no-header don't print the directory header line
--output='format-specifier' output in the specified format, using printf-style format specifiers
--output=ALL same as --output='%20s %10r %8p %08w %1S %-50n %l'
%s file size
%r resident size
%p file permissions
%w win attrs
%S migration state ('-', 'm=migrated', 'p=premigrated')
%n file name
%l remote location
default is --output='%20s %10r %S %-50n'
or '%20s %10r %S %-50n %l' if -l also selected
--quoting-style=escape|shell-escape
use a specified quoting style for listed file names:
"escape" - quoting unprintable characters using `\'
"shell-escape" - shell-safe quoting using $'string'
default is "shell-escape"
--json[=pretty] output in JSON format
On passing "pretty", produce indented multiline output for JSON
objects; otherwise, produce single-line output for JSON objects.
-m'glob', --match='glob' match the filename against the glob string
-r'regex', --regex='regex' match the filename against the regular expression
(note: directory names are not matched)
--state=STRING filter listed files to include only files in states:
'm' - migrated;
'p' - premigrated;
'-' - unmanaged.
Separate these characters by ',' to specify multiple states
--type=f|d filter listed entities to include only:
'f' - regular files;
'd' - directories
-?, --help display this help and exit.
-V, --version display version information and exit.
Examples¶
Default Output Format¶
Display data for jpg files in the current directory
ngls *.jpg
Display data for JPG files in the current directory and one level below
ngls -d1 --match='*.jpg'
Display data for JPG files in the current directory and 5 levels below
ngls -d5 --match='*.jpg'
Display all data for files in the current directory.
ngls -o all
SIZE RSS PERM ATTRS ST FILE NAME REMOTE
. : 7 items
1048576 1024 100644 00000001 p File_1.dat tg1:data/File_1.dat
1048576 0 100644 00000101 m File_2.dat tg1:data/File_2.dat
<dir> 1 40755 00000008 - images
<dir> 4 40755 00000008 - data
36375247 35584 100644 00000001 - sample_data.tgz
Natural / Alphabetic sort¶
The default sort option is -n - "natural" sort order. This is useful when listing filenames containing a version number. Compare:
ngls images
SIZE RSS ST FILE NAME
images : 6 items
2226725 2176K - images/IMG_V3.5.jpg
2371737 2432K - images/IMG_V3.23.jpg
2068990 2048K - images/IMG_V3.47.jpg
1958986 1920K - images/IMG_V9.1.jpg
2706537 2688K - images/IMG_V12.1.jpg
1729382256910270464 0K - images/Temp_E1.5
with an alphabetic sort (-a option):
ngls -a images
SIZE RSS ST FILE NAME
images : 6 items
2706537 2688K - images/IMG_V12.1.jpg
2371737 2432K - images/IMG_V3.23.jpg
2068990 2048K - images/IMG_V3.47.jpg
2226725 2176K - images/IMG_V3.5.jpg
1958986 1920K - images/IMG_V9.1.jpg
1729382256910270464 0K - images/Temp_E1.5
Human-readable format¶
The human-readable option (-h, or --human-readable) converts the file size (SIZE column) and resident size (RSS column) to a human-readable form, based on the international standard, or the SI equivalent, if selected.
Unit
Value
SI Value
K
kilo 1024 or 2**10
10**3
M
mega 1024**2 or 2**20
10**6
G
giga 1024**3 or 2**30
10**3
T
tera 1024**4 or 2**40
10**3
P
peta 1024**5 or 2**50
10**3
E
exa 1024**6 or 2**60
10**3
ngls -h images
SIZE RSS ST FILE NAME
images : 6 items
2.1M 2.1M - images/IMG_V3.5.jpg
2.3M 2.4M - images/IMG_V3.23.jpg
2.0M 2.0M - images/IMG_V3.47.jpg
1.9M 1.9M - images/IMG_V9.1.jpg
2.6M 2.6M - images/IMG_V12.1.jpg
1.5E 0 - images/Temp_E1.5
ngls --si images
SIZE(SI) RSS ST FILE NAME
images : 6 items
2.2M 2.2M - images/IMG_V3.5.jpg
2.4M 2.5M - images/IMG_V3.23.jpg
2.1M 2.1M - images/IMG_V3.47.jpg
2.0M 2.0M - images/IMG_V9.1.jpg
2.7M 2.8M - images/IMG_V12.1.jpg
1.7E 0 - images/Temp_E1.5
Custom Output Format¶
Display all data with '||' separator
ngls -H -o '%s||%r||%p||%w||%S||%n||%l'
1048576||1024||100644||00000001||p||File_1.dat||tg1:data/File_1.dat
1048576||0||100644||00000101||m||File_2.dat||tg1:data/File_2.dat
<dir>||1||40755||00000008||-||images||
<dir>||4||40755||00000008||-||data||
36375247||35584||100644||00000001||-||sample_data.tgz||
The -H option suppresses the directory header.
Display selected items in comma-separated format, including sub-directories
ngls --output='%n,%s,%r' -d1
. : 7 items
File_1.dat,1048576,1024
File_2.dat,1048576,0
images,<dir>,1
data,<dir>,4
sample_data.tgz,36375247,35584
images : 0 items
data : 5 items
data/IMG_4625.jpg,2706537,2688
data/IMG_4626.jpg,1958986,1920
data/IMG_4650.jpg,2226725,2176
data/IMG_4686.jpg,2068990,2048
data/IMG_4694.jpg,2371737,2432
See Also¶
Querying extended attributes¶
All Ngenea metadata associated with a file's migration are stored as extended attributes within the file inode.
As such, standard PixStor tools can be used to show this information.
For example - mmlsattr can be used to show the migration information about a file:
# mmlsattr -d -L sample.txt
file name: sample.txt
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sas1
fileset name: root
snapshot name:
creation time: Wed May 30 16:54:18 2018
Misc attributes: ARCHIVE OFFLINE
Encrypted: no
dmapi.APXguuid: "a7e82a60-62ae-4c25-8da6-91d05f265714"
dmapi.APXrmtlc: "target1:test/sample.txt"
dmapi.APXsh512: "0cfcd93fb5eae3f5c9962117b38882de79ea4f9c432d574e8c9a4b1b65cded4620d25dc1b29ef0e60d238219797cb4f7d3946ce3a0f718b849cff2ae41fffd1c"
dmapi.APXrmtsz: "29"
gpfs.dmapi.region: 0xFFFFFFFFFFFFFFFF00000000000000000700000000000000
The relevant extended attributes include:
Misc attributes: the presence of the OFFLINE status here shows that the file is migrated, and is no longer resident on the file system
dmapi.APXguuid: Unique ID for the file, used to authoritatively match an inode to a remote object
dmapi.APXrmtlc: Remote location of the file, as defined in the RemoteLocationXAttrRegex setting for the activated storage target in the Master Configuration File, at the point it was migrated.
dmapi.APXsh512: SHA512 checksum of the file at the point it was migrated.
dmapi.APXrmtsz: Size of the object sent to the external storage.
gpfs.dmapi.region: Internal PixStor attribute which reflects the offline status of the file
The ngls tool called with the option --print-xattrs also lists extended attributes:
# ngls --print-xattrs sample.txt
SIZE RSS ST FILE NAME
29 0K m sample.txt
Extended attributes:
dmapi.APXguuid: "a7e82a60-62ae-4c25-8da6-91d05f265714"
dmapi.APXrmtlc: "target1:test/sample.txt"
dmapi.APXsh512: "0cfcd93fb5eae3f5c9962117b38882de79ea4f9c432d574e8c9a4b1b65cded4620d25dc1b29ef0e60d238219797cb4f7d3946ce3a0f718b849cff2ae41fffd1c"
dmapi.APXrmtsz: "29"
gpfs.dmapi.region: 0xFFFFFFFFFFFFFFFF00000000000000000700000000000000
Note that these attributes must not be changed, except under explicit direction from ArcaStream / Pixit Media support. Doing so may result in data becoming unavailable, and may even result in data corruption/loss.