File¶
The File object represents a file on the filesystem. It can be used to look-up and change attributes of a file.
This class can’t be used to create, destroy, or edit the contents of files.
Important
All time-based attributes are provided as naive datetime objects. However, creation time is given in the localtime zone whereas all other times are in Universal Coordinate Time (UTC). This is a quirk of PixStor. To convert all times to the same timezone, consider using the pytz library
Description¶
-
class
arcapix.fs.gpfs.file.RsFlags¶ Flags which may be passed to the
File restripemethod:- criticalFromSuspended
- allFromSuspended
- repairPoolPlacement
- repairReplicaConflicts
- repairReadOnlyConflicts
- rebalance
- strictRebalance
- relocateBlocks
- runDeferedCompression
-
class
arcapix.fs.gpfs.file.File(path)¶ File objects represent files in the GPFS filesystem
Create a File object
Parameters: path (str) – path to a file within the GPFS filesystem Note: This is just a representation of an existing file. This does not create a new file on the filesystem.
-
change(**kwargs)¶ Change certain file attributes
Parameters: - action – ‘yes’ or ‘defer’ (default=’yes’)
- storagepoolName – Name of pool to move file to
- dataReplicas (int) – number of data replicas to make of file
- maxDataReplicas (int) – maximum number of data replicas to make of file
- metadataReplicas (int) – number of metadata replicas to make of file
- maxMetadataReplicas (int) – maximum number of metadata replics to make of file
- immutable (bool) – set the (im)mutability of the file
- appendOnly (bool) – set whether the file should be append only
- compression (str) –
compress or decompress the file with a given library. One of
- yes: compress using ‘z’ (see below) (4.2.1+ only)
- no: decompress (4.2.1+ only)
- z: Cold data. Favors compression efficiency over access speed. (5.0.0+ only)
- lz4: Active, nonspecific data. Favors access speed over compression efficiency. (5.0.0+ only)
- zfast: Active genomic data in FASTA, SAM, or VCF format. (5.0.3+ only)
- alphae: Active genomic data in FASTQ format. Slightly favors compression efficiency over access speed. (5.0.3+ only)
- alphah: Active genomic data in FASTQ format. Slightly favors access speed over compression efficiency. (5.0.3+ only)
Note
maxDataReplicasandmaxMetadataReplicascan only be changed for zero length files.
-
restripe(flag)¶ Perform a restripe action on the file.
Parameters: flag – The restripe action to perform. One of RsFlags
-
acl¶ Returns access control information for the file.
Return type: arcapix.fs.gpfs.acl.Acl
-
maxMetadataReplicas¶ Returns the maximum number of metadata replicas that can be created of the file
Return type: int
-
maxDataReplicas¶ Returns the maximum number of data replicas that can be created of the file
Return type: int
-
flags¶ Return any flags associated with the file
Return type: list
-
poolName¶ Returns the name of the storage pool which the file resides in
Synonym for
storagepoolNameReturn type: str
-
snapshotName¶ Returns the name of the snapshot which the file belongs to
Returns None if not a snapshot file
Return type: str
-
creationTime¶ Returns the file’s creation time (local timezone)
Return type: datetime
-
miscAttributes¶ Returns miscellaneous attributes of the file such as whether the file is ARCHIVE or READONLY.
Older versions of GPFS call these ‘Windows Attributes’
Return type: list
-
compression¶ Returns the compression library used on the file (if any)
New in GPFS 4.2.1 - older versions return ‘no’
For GPFS < 5.0.0 the specific library isn’t available, so only ‘yes’ is returned if the file is compressed
One of [yes, no, z, lz4, zfast, alphae, alphah]
Return type: str
-
accessTime¶ Returns the time when the file was last accessed (UTC)
Return type: datetime
-
modificationTime¶ Returns the time when the file (UTC) was last modified
Return type: datetime
-
changeTime¶ Returns time that the file metadata (UTC) was last changed.
Return type: datetime
-
sqlMiscAttributes¶ Returns various micalleneous attributes of the file
This is a string of characters, which indicate properties such as object type.
See MISC_ATTRIBUTES for a explanation of the returned string
Return type: str
-
generation¶ Returns the file’s generation number
A number that is incremented when an inode number is reused.
Return type: int
-
-
arcapix.fs.gpfs.file.setNoUpdateCtime(enable=True)¶ Don’t update file change time (ctime) when xattrs are created, updated, or deleted.
-
class
arcapix.fs.gpfs.file.XAttrs(path)¶ Class representing the extended attributes of a file
Note
Only user extended attributes can be created, deleted, or changed.
You do not need to specify
user.*-
new(key, value=None, force=False)¶ Set a user extended attribute
>>> f.xattrs.new('foo', 'bar') >>> f.xattrs.user['foo'] bar
Parameters: - key – Name of new extended attribute
- value – Value for new extended attribute
- force – Whether this attribute should overwite any existing with the same name (default=False)
Raises: GPFSExecuteException If attribute with name ‘key’ already exists and ‘force’ isn’t True
-
change(key, value=None)¶ Change a user extended attribute
Parameters: - key – Name of the user extended attribute to be changed
- value – New value for the extended attribute
Raises: KeyError If attribute with name ‘key’ doesn’t exist
-
destroy(key)¶ Delete a user extended attribute
Parameters: key – Name of the user extended attribute to delete Raise: KeyError If attribute with name ‘key’ doesn’t exist
-
Examples¶
Make a file immutable¶
>>> from arcapix.fs.gpfs import File
>>>
>>> # create a new File object
... f = File('/mmfs/data/myfile.txt')
>>>
>>> # check the file's mutability
... f.immutable
False
>>>
>>> # make the file immutable
... f.change(immutable=True)
>>>
>>> f.immutable
True
Change the data replication for all .txt files using a Policy¶
>>> from arcapix.fs.gpfs import ManagementPolicy, ListProcessingRule, Criteria
>>>
>>> # define the processing function
... def processor(files):
... files_changed = 0
... for f in files:
... f.change(dataReplicas=1)
... files_change += 1
... return files_changed
...
>>> # create a policy
... p = ManagementPolicy()
>>>
>>> # create the processing rule
... r = p.rules.new(ListProcessingRule, 'txt_repls', processor=processor)
>>>
>>> # add a criteria to only select txt files
... r.criteria.new(Criteria.like('NAME', '*.txt'))
>>>
>>> # Run the policy
... p.run('mmfs1')
{'txt_repls': 345578}
>>>
>>> f = File('/mmfs1/data/myfile.txt')
>>> f.dataReplicas
1
Restripe a File After a Defered Move¶
>>> from arcapix.fs.gpfs.file import File, RsFlags
>>>
>>> f = File('/mmfs1/test.tmp")
>>>
>>> # perform a defered move of the file to sas1
... f.change(storagepoolName='sas1', action='defer')
>>>
>>> # the file is now 'illplaced'
... print(f.flags)
['illplaced']
>>>
>>> # perform a restripe on the file
... f.restripe(RsFlags.repairPoolPlacement)
>>>
>>> # the file is no longer illplaced
... print(f.flags)
[]