ChangeLog¶
1.28.1 - 2024-04-05¶
Bug Fixes:
[RNDNGHSM-1033] Remote location xattr not stamped on a live file if a snapshot file has a remote location xattr
[RNDNGHSM-1036] Segfault in ngscan on passing '--unique --json'
1.28.0 - 2024-03-01¶
New Feature:
[RNDNGHSM-1017] Byte based partial file recall
Improvement:
[RNDNGHSM-1008] AWS Partner Metadata Tagging
1.27.0 - 2023-12-20¶
New Feature:
[RNDNGHSM-1004] Create the ngunmanage tool
Improvements:
[RNDNGHSM-991] Provide option to not store sidecar files on POSIX file system targets
[RNDNGHSM-993] Provide the ability to store sidecar files in a specific folder within the directory
[RNDNGHSM-994] Provide the ability to store sidecar files in an alternative location
[RNDNGHSM-999] Add ngenea metadata to files migrated from snapshots
[RNDNGHSM-1001] Skip mime-type detection for POSIX targets
Bug Fixes:
[RNDNGHSM-1005] Shadow metadata files might become corrupted
1.26.1 - 2023-10-20¶
Bug Fix:
[RNDNGHSM-1000] Segfault on passing a nonexistent file in the root directory
1.26.0 - 2023-10-06¶
Improvements:
[RNDNGHSM-888] Print vendor metadata
[RNDNGHSM-972] Improve file system target performance
[RNDNGHSM-974] Support premigrating files from snapshots
1.25.1 - 2023-11-30¶
Bug Fix:
[RNDNGHSM-1016] GCS: failure status of a metadata put operation is lost
1.25.0 - 2023-07-28¶
Improvements:
[RNDNGHSM-886] Provide the ability to recall to a different location
[RNDNGHSM-955] Support '--ignore-rmtlc' by ngrecall
[RNDNGHSM-956] Honor remote location xattrs by ngrecall on '--sync-metadata'
[RNDNGHSM-970] Perform content-only check on '--fail-on-mismatch' by default
Bug Fixes:
[RNDNGHSM-961] Reading a file being created with ngrecall -p results in corrupt data reads
[RNDNGHSM-965] Truncating a file causes "Operation not permitted" with a transparent recall policy installed
1.24.1 - 2023-06-05¶
Bug Fix:
[RNDNGHSM-966] Segfault on reading an empty .xattr file
1.24.0 - 2023-06-01¶
Improvement:
[RNDNGHSM-959] Support '--sync-metadata' by ngmigrate
1.23.0 - 2023-05-11¶
Improvements:
[RNDNGHSM-203] Add timestamp for migration time to migrated file extended attributes
[RNDNGHSM-572] Ensure GCS handles indefinite retries properly
[RNDNGHSM-844] Support ds3_sdk 5.4.0
[RNDNGHSM-887] Provide the ability to migrate to a different location
[RNDNGHSM-951] Honor remote location xattrs by ngscan
Bug Fixes:
[RNDNGHSM-952] ngscan fails to report shadow metadata when it has a key called symln
[RNDNGHSM-954] Migration fails with large xattrs
1.22.0 - 2023-03-20¶
Improvements:
[RNDNGHSM-793] Support overriding config params with command line options
[RNDNGHSM-936] Update retry policy for GCS protocol
Bug Fixes:
- [RNDNGHSM-923] Resolve issue with GCS performance whereby requests to the same endpoint
use the same caching slot causing sequential execution of tasks
[RNDNGHSM-924] Concurrency reduction with GCS PSC OAuth2 connections types
1.21.0 - 2023-02-23¶
New Feature:
[RNDNGHSM-931] Provide option to not migrate a file if the remote object is different
Bug Fixes:
[RNDNGHSM-921] Unexpected debug output when uploading objects to GCS
[RNDNGHSM-934] Error "unexpected end of file" may result in process termination
[RNDNGHSM-938] Cloud object deleted but not recalled
1.20.1 - 2023-02-06¶
Improvement:
[RNDNGHSM-929] Include source code location information in "out of memory" error messages
1.20.0 - 2023-01-25¶
Improvements:
[RNDNGHSM-736] Provide the ability to target specific storage classes for Azure protocol
[RNDNGHSM-789] Add CredentialsFile option to Azure config
[RNDNGHSM-803] Bash completion
[RNDNGHSM-827] Add EscapeNames support for Azure
[RNDNGHSM-861] Add support for local path in ngscan output
[RNDNGHSM-872] Don't rewrite Ngenea xattrs if they haven't changed
[RNDNGHSM-873] Sync metadata on file recall
[RNDNGHSM-882] Ensure ngenea configurations are secure
[RNDNGHSM-883] Add CredentialsFile option in AWS
[RNDNGHSM-901] Restricting restored symlinks
[RNDNGHSM-904] Saving/restoring ctime for symlinks
[RNDNGHSM-905] Recursive pulling of all symlinks in a path
[RNDNGHSM-908] Log file processed when --delete-remote is used on a premigrated file
[RNDNGHSM-911] Provide sample utility to remove remotely orphaned files
Bug Fixes:
[RNDNGHSM-455] ngls incorrectly lists files with unprintable characters in the name
[RNDNGHSM-835] Inconsistent information in Ngenea storage target configuration page
[RNDNGHSM-875] Resolve issue with policy driven migration and -- in file paths
[RNDNGHSM-885] ngrecall -L doesn't recall the specifically requested remote
[RNDNGHSM-906] Can't migrate to S3 a file with special characters in an xattr name
[RNDNGHSM-907] DeleteOnRecall=true can cause deletions from DeleteOnRecall=false targets when multi-target setup is in use
1.19.2 - 2023-01-23¶
Bug Fix:
[RNDNGHSM-920] Revert behaviour of GCS connection reuse to 1.18 behaviour due to observed performance decrease
1.19.1 - 2023-01-23¶
Bug Fix:
[RNDNGHSM-900] Error "directory restoring failed" dependent on arguments order
1.19.0 - 2022-10-04¶
New Features:
[RNDNGHSM-538] Updating the metadata of already migrated remote objects
[RNDNGHSM-854] Updating the status of local files based on remote object metadata
Improvements:
[RNDNGHSM-424] Ensure user defined xattr are reflected in endpoints
[RNDNGHSM-831] Improve performance of Google uploads
[RNDNGHSM-842] Support storing a symlink value as a metadata element in S3
[RNDNGHSM-858] Reduce the probability of Google error 503 when migrating
1.18.0 - 2022-07-14¶
Improvements:
[RNDNGHSM-824] Add EscapeNames support for GCS
[RNDNGHSM-841] Support additional file mode bits
1.17.1 - 2022-06-27¶
Bug Fix:
[RNDNGHSM-840] Assertion failure when recalling with '-r' a directory containing a symlink
1.17.0 - 2022-06-22¶
Improvements:
[RNDNGHSM-766] Google OAuth2 Compliance
[RNDNGHSM-805] Provide a mechanism to update atime and mtime
Bug Fixes:
[RNDNGHSM-690] GCS_AddMimeType storage: object doesn't exist
[RNDNGHSM-787] Ensure that GCS Go goes not retain file in memory for duration of transfer
[RNDNGHSM-802] ngls lists files twice
[RNDNGHSM-804] Scanning + reverse stubbing/premigration fails on objects with inconsistently percent-encoded characters in names
[RNDNGHSM-833] Assertion failure in apiod_perc_decode
[RNDNGHSM-834] Update GCS SDK to resolve error "Could not find default credentials"
1.16.0 - 2022-05-10¶
Improvements:
[RNDNGHSM-672] ngscan --list-shadow
[RNDNGHSM-673] ngls --state and --type filters
[RNDNGHSM-742] Provide quiet output for ngscan to enable fast listing of object(s) existence returning only exit status
[RNDNGHSM-769] Support passing local filenames to ngscan
[RNDNGHSM-773] Add Endpoint and Port to Azure config
[RNDNGHSM-777] Provide JSON output from ngscan
[RNDNGHSM-778] Provide JSON output from ngls
[RNDNGHSM-779] Provide additional metadata from endpoint object for ngscan
[RNDNGHSM-780] Provide extended attr metadata from local object for ngls
Bug Fixes:
[RNDNGHSM-775] Ensure azure block ids are identical for subsequent uploads of a non-committed file
[RNDNGHSM-782] Premigrating a file to azure with overwrite creates 2x new versions in Azure
[RNDNGHSM-792] Unsafe functions called from a signal handler
[RNDNGHSM-799] Cope with pre-existing uncommitted blocks uploaded by other tools
1.15.0 - 2022-03-01¶
Improvement:
[RNDNGHSM-678] Single-line message output in JSON log
Bug Fix:
[RNDNGHSM-784] Incorrect unescaping JSON string literals
1.14.0 - 2022-02-10¶
New Features:
[RNDNGHSM-719] Filter entity types listed by ngscan
[RNDNGHSM-722] Support storing/retrieving symbolic links as links instead of targets they reference
Improvements:
[RNDNGHSM-757] Remove the 25 connections limit for S3
[RNDNGHSM-759] Include total part count in log messages
[RNDNGHSM-761] Support recalling remote folders created by third-party tools
Bug Fix:
[RNDNGHSM-762] Fix migrating/recalling folders for Minio
1.13.0 - 2021-12-13¶
Improvements:
[RNDNGHSM-539] Implement minimal DMAPI locking mode
[RNDNGHSM-727] Implement a timeout to automatically terminate ngenea in the event that external operations are not progressing
[RNDNGHSM-737] Provide the ability to target specific storage classes for GCS protocol
[RNDNGHSM-738] Provide GPFS Policy ILM samples
[RNDNGHSM-747] Add man pages for ngscan and ngreconcile
Bug Fixes:
[RNDNGHSM-699] Running ngls on a long path causes memory to become corrupt
[RNDNGHSM-706] Ngenea GCP permissions docs missing required rights
[RNDNGHSM-733] Sample script migrate.py calls a wrong function to enact migration
[RNDNGHSM-741] Google StorageClass migration fails if lowercase
1.12.1 - 2021-10-22¶
Improvement:
[RNDNGHSM-720] Overwrite remote folder metadata on '--overwrite-remote'
Bug Fix:
[RNDNGHSM-721] apiod_migrate: assertion "n_filtered2<n_filtered1" when migrating empty directories
1.12.0 - 2021-10-15¶
New Features:
[RNDNGHSM-634] Possibility of migrating and recalling empty directories
[RNDNGHSM-687] Support for storing and retrieving ACLs
Improvements:
[RNDNGHSM-680] Support Black Pearl SDK 5.3.0
[RNDNGHSM-683] Emit the signal exit code when ngmigrate or ngrecall is killed by signal
[RNDNGHSM-702] Provide a user-friendly message for gpfs_putacl error 95
[RNDNGHSM-716] Ensure maxfiles migration for python scripts are set to a sensible default
Bug Fix:
[RNDNGHSM-688] Support UTF-8 for storing user and group names
1.11.0 - 2021-07-02¶
New Feature:
[RNDNGHSM-573] Run migrate without running hash before
Improvement:
[RNDNGHSM-167] Do not call an external program to cancel a current BlackPearl job by Ctrl-C
Bug Fix:
[RNDNGHSM-679] Deadlock in JSON log mode when printing a string not matching a LC_CTYPE locale
1.10.0 - 2021-05-07¶
New Feature:
[RNDNGHSM-646] - Structured Log Output
Improvements:
[RNDNGHSM-464] - Support connecting to Google's private or restricted API services
[RNDNGHSM-563] - Use Intel ISA-L acceleration library for sha-512 acceleration
[RNDNGHSM-595] - Set MIME type for Google uploads
[RNDNGHSM-596] - Set MIME type for Azure uploads
[RNDNGHSM-621] - Improve parallelism of hashing for AWS and Azure
[RNDNGHSM-643] - Improve GCS thread management
Bug Fixes:
[RNDNGHSM-628] - ngmigrate terminates with what(): Resource temporarily unavailable
[RNDNGHSM-661] - Ngenea assert() on zero byte object in Azure blob
1.9.2 - 2021-03-24¶
Improvements:
[RNDNGHSM-645] - Release all acquired DMAPI locks on SIGTERM or SIGINT
1.9.1 - 2021-03-09¶
Improvements:
[RNDNGHSM-641] - Add the parameter "ShadowFolderMetadataSave" for S3, Azure, GCS, and BlackPearl
Bug Fixes:
[RNDNGHSM-640] - .xattr files for folders are escaped even with EscapeNames=false
1.9.0 - 2021-02-11¶
New Feature:
[RNDNGHSM-605] - Scan local files and fix their premigrate flag
Improvements:
[RNDNGHSM-529] - Fetch metadata once when processing scanned remote objects and deleting recalled ones
[RNDNGHSM-540] - Add an ngmigrate option to forcibly overwrite remote objects
[RNDNGHSM-587] - Support passing a filelist file to ngmigrate/ngrecall
[RNDNGHSM-610] - Provide the ability to not escape character sequences
[RNDNGHSM-615] - Store both string and numeric user and group identifiers
Bug Fixes:
[RNDNGHSM-206] - ngls does not consistently list offline status when run inside a snapshot
[RNDNGHSM-481] - ngls -V returns non-zero
[RNDNGHSM-484] - Docs show ngls -? as valid option despite this not working
[RNDNGHSM-618] - Ngls dm_init_service error output for non-root users
1.8.0 - 2020-11-06¶
Improvements:
[RNDNGHSM-428] - Store times in metadata with nanosecond precision
[RNDNGHSM-593] - Improve Azure multi-threaded upload and download
Bug Fixes:
[RNDNGHSM-597] - Double free or corruption while migrating files to BlackPearl
[RNDNGHSM-599] - ngmigrate does not set "premigrated" file status visible in a GPFS policy
[RNDNGHSM-604] - migrate.c: line 1130: apiod_migrate: assertion "0" failed
1.7.2 - 2020-09-23¶
Bug Fixes:
[RNDNGHSM-601] - Migration failure when unable to resolve uids and gids. getgrgid_r error observed.
1.7.1 - 2020-08-07¶
Bug Fixes:
[RNDNGHSM-598] - Eliminate a race condition causing "attribute name length 6 is not equal to 8"
1.7.0 - 2020-07-13¶
Improvements:
[RNDNGHSM-328] - Honor the options "--skip-check-uuid" and "--skip-check-hash" during file recall operations
[RNDNGHSM-347] - Allow uuid and hashing override to enable a 'twin-tailed' stub file workflow
[RNDNGHSM-557] - GCS: Increase performance by downloading part files in parallel
[RNDNGHSM-575] - Ensure ngrecall with --no-recursion-remote flag skips files in subdirs
[RNDNGHSM-576] - GCS: Remove duplication of buffer during download
[RNDNGHSM-582] - Reduce frequency of flock errors on reverse stubbing
[RNDNGHSM-584] - Support reverse stubbing/premigrating all object instances rather than the latest
[RNDNGHSM-586] - GCS: Increase upload performance
Bug Fixes:
[RNDNGHSM-452] - GCS: a result of dm_write_invis isn't handled correctly
[RNDNGHSM-578] - ngrecall scans and reversely stubs/premigrates ".*.xattr" files
1.6.0 - 2020-05-26¶
Improvements:
[RNDNGHSM-480] - Support signing payloads to AWS S3
[RNDNGHSM-544] - S3: make threads upload file parts rather than files
[RNDNGHSM-545] - S3: make threads download file parts rather than files
[RNDNGHSM-549] - Sign payloads to BlackPearl
[RNDNGHSM-551] - Sign payloads to Azure
[RNDNGHSM-561] - Ensure logging behaviour is consistent for all SDKs
Bug Fixes:
[RNDNGHSM-442] - Migration to Azure fails for large files
[RNDNGHSM-451] - Azure: result of dm_read_invis and dm_write_invis isn't handled correctly
1.5.0 - 2020-04-14¶
Improvements:
[RNDNGHSM-534] - Release migration file locks as soon as reasonably possible
[RNDNGHSM-536] - Release recall file locks as soon as reasonably possible
1.4.0 - 2020-02-14¶
New Feature:
[RNDNGHSM-48] - Automatically ingest and reverse stub pre-existing external data
[RNDNGHSM-446] - Ensure ngmigrate sets the correct MIME type for files uploaded to S3
[RNDNGHSM-471] - Python API examples to drive ngmigrate and ngrecall
Improvements:
[RNDNGHSM-365] - Remove info level output from ngmigrate to google object store with no verbose flags set
[RNDNGHSM-435] - Ensure failed to recall partially recalled files are removed
[RNDNGHSM-482] - Reduce memory usage for GCS migrations
[RNDNGHSM-494] - Create missing directories with permission modes for reversely stubbed/premigrated files
[RNDNGHSM-518] - Provide logging output to syslog for ngmigrate & ngrecall
[RNDNGHSM-530] - Provide direct migration to all S3 Storageclasses
1.3.5 - 2020-01-06¶
Bug Fixes:
[RNDNGHSM-499] - Fixes a problem whereby if a dmapi right is acquired on a file, and the file gets deleted, then releasing the right would cause a waiter to appear
1.3.4 - 2019-12-02¶
Improvements:
[RNDNGHSM-478] - Ensure that failed recalls from S3 reset the prior state of the stub file
Bug Fixes:
[RNDNGHSM-489] - Fixes an issue whereby ngls and ngrecall trigger a gpfs crash if there are non-ngenea managed APX dmapi xattrs present on a file
1.3.3 - 2019-11-22¶
Improvements:
[RNDNGHSM-429] - ngscan should find latest object instances by default
Bug Fixes:
[RNDNGHSM-470] - failed ngrecall from aws s3 bucket where file has been moved to glacier changes the local file size
[RNDNGHSM-475] - ngrecall not failing cleanly
1.3.2 - 2019-11-14¶
Feature Enhancements:
[RNDNGHSM-357] - Add the option "-r, --recursion-local" to ngrecall
[RNDNGHSM-392] - Ensure that ngrecall does not change the source atime of a file
[RNDNGHSM-394] - Detect changing a source file while migrating it and abort migration on file change
[RNDNGHSM-395] - Ensure reverse stubbing uses the latest file instance
[RNDNGHSM-406] - ngls now supports multi-targets
[RNDNGHSM-414] - Reverse stubbing a file with multiple endpoints uses any available endpoint
Bug Fixes:
[RNDNGHSM-382] - Updating metadata on large files (>5GB) fails on AWS S3 Targets
[RNDNGHSM-397] - ngrecall cannot reversely stub/premigrate a remote object with special characters or spaces
[RNDNGHSM-426] - ngmigrate hanging whole fileset
[RNDNGHSM-431] - ngls -d does not work
[RNDNGHSM-438] - Exit with error in the case that ngrecall attempts (and fails) to write to a full file system
[RNDNGHSM-440] - failure to move large files >400g from pixelogic up to snoball.. maybe others
[RNDNGHSM-441] - SAMBA unresponsive when ngmigrate is in progress
[RNDNGHSM-450] - ngrecall hanging ngls
1.3.1 - 2019-08-05¶
Bug Fixes:
[RNDNGHSM-407] - ngrecall causes a gpfs hang when recalling from multiple targets, where one target has changed the file
[RNDNGHSM-417] - ngrecall doesn't check whether the size of a stub file and a remote object are equal
[RNDNGHSM-418] - memory leak in ngls
1.3.0 - 2019-05-28¶
Feature Enchancements:
[RNDNGHSM-353] - Support restricting paths of processed remote objects in a storage endpoint
[RNDNGHSM-355] - Add storage-specific parameters specifying prefixes for "RetrieveObjectName" and "StoreObjectName"
[RNDNGHSM-356] - Add the option "-r, --recursion-local" to ngmigrate
[RNDNGHSM-362] - Create the "ngscan" tool
[RNDNGHSM-375] - Stop deleting empty parent directories if such a directory is one specified in "RetrieveObjectBasePath"
Bug Fixes:
[RNDNGHSM-364] - ngls doesn't show remote location for multiple endpoints
[RNDNGHSM-366] - ngls should display file sizes in human-readable format.
[RNDNGHSM-370] - Provide informative error message if Scheme target config setting is not present
[RNDNGHSM-380] - ngrecall with "-L REMOTE_LOC" uses a computed local filename to determine a remote object location
1.2.0 - 2019-02-13¶
Feature Enchancements:
[RNDNGHSM-262] - Allow ngenea to migrate to Google Cloud Storage
[RNDNGHSM-279] - Add support for BlackPearl SDK v5
[RNDNGHSM-330] - IBM Cloud Object Support Validation
Bug Fixes:
[RNDNGHSM-243] - GPFS crashes when running mmlsattr against a specific file with transparent policy enabled
[RNDNGHSM-338] - On errors when migrating a batch of files to Azure, the APXrmtXX xattr isn't set for successfully uploaded files
[RNDNGHSM-341] - Complementary file locking functions are not thread-safe
[RNDNGHSM-348] - If Azure access key length is not a multiple of 4, ngmigrate aborts prematurely
1.1.1 - 2018-09-27¶
Feature Enhancements:
[RNDNGHSM-308] - Multiple Target Support
Bug Fixes:
[RNDNGHSM-124] - Multi-part upload memory improvements (S3)
1.1.0 - 2018-06-08¶
Feature Enhancements:
[RNDNGHSM-31] - Pre-migration capability
[RNDNGHSM-60] - ngls utility
[RNDNGHSM-135] - Support for POSIX targets
[RNDNGHSM-166] - Support for Azure Blob targets
[RNDNGHSM-179] - Variable stub sizing capability
[RNDNGHSM-189] - Reverse stub creation capability
1.0.0 - 2017-09-22¶
Initial release