#########
Changelog
#########

1.2.0 - 2024-02-01
==================

Bug fixes:

    * RNDEAS-1475: Validation error for valid GeoPoint data

Feature enhancements:

    * HUB2-1172: Tags plugin


1.1.0 - 2023-10-26
==================

Bug fixes:

    * RNDEAS-1460: Handle unicode error for invalid DPX film edge codes
    * RNDEAS-1462: Remove unsupported float values from extracted metadata
    * RNDEAS-1464: Remove invalid GPS values from the location plugin
    * RNDEAS-1467: Last ingest time improperly formatted in verify-file
    * RNDEAS-1471: 500 error from projection on unknown field

Improvements:

    * RNDEAS-1465: Ability to configure the google cloud plugin poll interval

Feature enhancement:

    * RNDEAS-1473: Support for per-field AND filters


1.0.0 - 2022-05-26
==================

Refactoring:

    * RNDEAS-1446: Migrated to Python 3
    * Deprecated commands (finder, etc.) have been removed

Bug fixes:

    * RNDEAS-1451: elasticsearch service status reporting on PixStor 6

Improvements:

    * RNDEAS-1434: Change video preview audio encoding to vorbis

Breaking Changes:

    * Renamed plugin 'async' method to 'is_async'
    * Any user plugins which are not compatible with Python 3 will no longer work.
      See :doc:`migrate` for more details.


0.15.0 - 2022-05-17
===================

Bug fixes:

    * RNDEAS-1426: UnicodeDecodeError in prune directory plugin
    * RNDEAS-1435: Ingest auth token expires after 24 hours

Feature enhancement:

    * RNDEAS-1360: Extract more DPX metadata
    * RNDEAS-1363: Plugin to extract endian information
    * RNDEAS-1365: Adobe Premier project plugin
    * RNDEAS-1374: Plugin to extract Maya fileinfo metadata
    * RNDEAS-1427: Plugin development walk-throughs
    * RNDEAS-1430: Add plugin check for missing schema
    * RNDEAS-1432: Searchctl command to verify ingest status of individual files
    * RNDEAS-1448: Ngenea metadata plugins
    * Vantage video proxy plugin

Improvements:

    * RNDEAS-1431: Sort plugins with the same priority alphabetically


0.14.0 - 2021-03-30
===================

Updated EULA

Bug fixes:

    * RNDEAS-1376: More informative error when trying to ingest single file
    * RNDEAS-1377: Reason not listed in searchctl jobs for memory errors
    * RNDEAS-1386: Exiftool can't identify some formats with -fast3
    * RNDEAS-1398: Revert filters link name in C+J
    * RNDSRCHUI-249: TypeError for aggregation of non-range numeric filter

Feature enhancements:

    * RNDEAS-1344: Support for generating null-terminated lists from verify-ingest
    * RNDEAS-1385: Use policy output processor during ingest
    * RNDEAS-1390: Colorized searchctl jobs table
    * RNDEAS-1392: Searchctl command for checking service statuses

Refactoring:

    * RNDEAS-1391: Update to use rights instead of roles


0.13.0 - 2021-01-08
===================

Bug fixes:

    * RNDEAS-1185: Prune directory error if no 'core.inode' data
    * RNDEAS-1242: Paths aren't properly quoted in HATEOAS links
    * RNDEAS-1300: Unicode error for ingest with unicode target path
    * RNDEAS-1303: Don't allow stopping all jobs in searchctl
    * RNDEAS-1304: Memory error from verbose ingest logging
    * RNDEAS-1328: Improve error reporting for broker user config
    * RNDEAS-1330: Rephrase cleandb logging messages
    * RNDEAS-1339: Dynamic plugins errors from unicode paths
    * RNDEAS-1349: Unicode error in searchctl jobs

Feature enhancements:

    * RNDEAS-985: Support for multiple ingest locks
    * RNDEAS-1211: Manpages for searchctl single-file tools
    * RNDEAS-1261: Support for setting ImageMagick temp dir in search.yaml
    * RNDEAS-1269: Support for running stat-only in parallel with rich ingest
    * RNDEAS-1279: Record logging settings in searchctl jobs
    * RNDEAS-1291: Document how to install plugin dependencies
    * RNDEAS-1304: Show ingest logging in searchctl ingest
    * RNDEAS-1313: Sort jobs by status change time in searchctl
    * RNDEAS-1326: Plugin for curating duplicate entries by path
    * RNDEAS-1351: Add plugin schema endpoint to Swagger docs
    * RNDEAS-1358: Mark locking errors as SKIPPED in searchctl jobs


0.12.0 - 2020-09-25
===================

Bug fixes:

    * RNDEAS-1310: Support pre 1900 dates in metadata fields
    * RNDEAS-1311: Don't skip exiftool mimetype for large files
    * RNDEAS-1312: Wrong encoding of non-ascii characters in retry lists

Feature enhancements:

    * RNDEAS-1091: Support for 'follow' mode in searchctl logs
    * RNDEAS-1287: Dynamic plugin filter


0.11.0 - 2020-08-21
===================

New Features:

    * RNDEAS-1228: Ability to set a size limit for plugins
    * RNDEAS-1257: Include site name in email alert subject line

Improvements:

    * RNDEAS-937: SIGTERM / SIGINT on apsearch-ingest should propagate to all children
    * RNDEAS-950: Show headers when using 'pxs_file_list' search CLI utility
    * RNDEAS-1045: Change ingest output to report multiple invalid plugins filters
    * RNDEAS-1049: Validate plugin - support for dynamic plugins
    * RNDEAS-1095: Ability to keep policy lists
    * RNDEAS-1145: Tweaks for searchctl plugins list
    * RNDEAS-1148: Searchctl ingest could inform of log at start of ingest
    * RNDEAS-1163: Implement plugin ingest timeout
    * RNDEAS-1171: searchctl filter jobs statuses
    * RNDEAS-1172: searchctl add total ingest time
    * RNDEAS-1200: Support APLOGTARGET in search.yaml
    * RNDEAS-1201: Ensure mimetype tools do not read huge files
    * RNDEAS-1213: Ensure ingest worklists are contained within their own directory
    * RNDEAS-1241: Ability to change column widths in searchctl jobs
    * RNDEAS-1254: Increase speed and robustness of PDF Proxying
    * RNDEAS-1273: rpmvenv deployment for middleware
    * RNDEAS-1298: Search 0.11 is converting TIFs via ImageMagick, not OpenImageIO
    * RNDEAS-1217: Ability to match plugins on mimetype only
    * RNDEAS-1218: Handle single frame videos
    * RNDEAS-1219: Xsym plugin
    * RNDEAS-1220: Exiftool processes are left running when ingest dies
    * RNDEAS-1230: Invalid metadata values from exiftool
    * RNDEAS-1235: Handle video files with no video stream
    * RNDEAS-1237: Camera plugin error for invalid flash value
    * RNDEAS-1238: Image fix orientation error for invalid orientation
    * RNDEAS-1239: Investigate cause of 422 errors on EG+
    * RNDEAS-1274: wsgi-searchserver.py needs to be in $PATH
    * RNDEAS-1275: Bundle modified systemd unit in RPM
    * RNDEAS-1277: Binary stripping causes ELF errors
    * RNDEAS-1278: Update sudoers to match new paths
    * RNDEAS-1209: Support for yaml with searchctl ingest

Bug fixes:

    * RNDEAS-627: Unallowed white balance
    * RNDEAS-948: `pxs_file_list --filter` search CLI utility does not work
    * RNDEAS-989: Condor job put on hold if files that require async job are ingested by root user
    * RNDEAS-1075: Unicode compatible paths are ingested with no issue but do not appear in UI
    * RNDEAS-1103: Unicode error for pxs_file_list
    * RNDEAS-1143: proxy: mimetype of generated proxy "video/mp4" doesn't match mimetype specified by plugin "video/mpeg"
    * RNDEAS-1146: Log message states to retry directory None
    * RNDEAS-1164: Regex searching returns 500
    * RNDEAS-1167: searchctl plugins check does not work alike validate_plugin
    * RNDEAS-1173: ConflictError: TransportError(409, u'version_conflict_engine_exception
    * RNDEAS-1182: VideoThumbnail - IOError: cannot identify image file <cStringIO.StringI object
    * RNDEAS-1193: double CTRL-C required to stop ingest
    * RNDEAS-1196: Per plugin logs print out unicode, summary log prints out proper filename
    * RNDEAS-1199: CommonAttributesPlugin common.creator: must be of string type
    * RNDEAS-1206: PhotoshopMetaDataPlugin - TypeError: Struct() argument 1 must be string, not unicode
    * RNDEAS-1215: Imagmagick tmp can fill OS filesystem
    * RNDEAS-1216: Assorted plugin errors at EG+
    * RNDEAS-1222: Quote paths in queries for prunedirectory
    * RNDEAS-1227: Prune directory query error when path contains double quotes
    * RNDEAS-1236: Broken pipe after exiftool crash
    * RNDEAS-1243: Setting security utils to true breaks the DB
    * RNDEAS-1255: 0.11b3 cannot import name abc
    * RNDEAS-1258: Handle exceptions in videopreview
    * RNDEAS-1266: Remove video/x-ms-asf from imagepreview
    * RNDEAS-1270: PDF Preview thumbnail aspect stretched
    * RNDEAS-1272: systemd service file has wrong variable names
    * RNDEAS-1281: Ensure non-root users can ingest with searchctl
    * RNDEAS-1290: find_duplicates not working on 10.132.0.175

Breaking changes:

    * RNDEAS-948: rename pxs_file_list flags --single-field to --output-field
        and --field to --filter-by-field

0.10.1 - 2020-06-18
===================

Bug fixes:

    * RNDEAS-1223: Change searchctl shebang to use env Python 2.7
    * RNDEAS-1224: Change wsgi script shebang to use env Python 2.7

0.10.0 - 2020-05-14
===================

Bug fixes:

    * RNDEAS-1085: Truncate logging of ffmpeg output
    * RNDEAS-1101: Don't use tuples in plugin docs and core plugins
    * RNDEAS-1108: Error when searchctl logs level isn't set
    * RNDEAS-1109: apsearch-ingest returned exit code 1 on success
    * RNDEAS-1110: Ignore trailing slash when filtering jobs by target
    * RNDEAS-1113: Some MXF proxies are transparent
    * RNDEAS-1117: Profile plugin backwards compatibility
    * RNDEAS-1165: Error when PDF metadata plugin doesn't extract any data

Feature enhancements:

    * RNDEAS-823: Prevent ingest from changing file timestamps (atime, mtime, ctime)
    * RNDEAS-956: Tool for recommending ingest configurations for stat-only ingest
    * RNDEAS-957: Manpages for searchctl tools
    * RNDEAS-994: Support for benchmarking plugins against multiple files
    * RNDEAS-1016: Tool for listing and stopping active ingests
    * RNDEAS-1037: Support for resetting file access times (atime) on ingest
    * RNDEAS-1063: Generate proxies for more image and video types
    * RNDEAS-1071: Plugin filter for 'core' plugins
    * RNDEAS-1087: Convert existing CLI tools to searchctl commands
    * RNDEAS-1088: Tool for listing enabled plugins
    * RNDEAS-1093: Display searchctl logs in a pager
    * RNDEAS-1099: PDF metadata and proxy plugins
    * RNDEAS-1100: Add MXF plugin to Lab
    * RNDEAS-1104: Document how to perform range queries
    * RNDEAS-1106: Dynamic plugin to show duplicate files
    * RNDEAS-1107: Add descriptions for all core plugins
	* RNDEAS-1155 Parent navigation with Nav plugin
    * RNDSRCHUI-202: Dynamic Flags plugin to show real-time on/offline file status

Refactoring:

    * The existing CLI tools (finder, etc) are deprecated in favour of equivalent 'searchctl' tools.
      The old tools still exist in this release for compatibility, but will be removed in a future release.
    * RNDEAS-1137: Create Arcapix Lab plugin location


0.9.0 - 2020-02-10
==================

Bug fixes:

    * RNDEAS-993: Prune directory fails for directories bigger than 10k files
    * RNDEAS-999: Async proxy update fails if proxy already exists
    * RNDEAS-1010: Validation error for creation time of filesystem mountpoint
    * RNDEAS-1019: Don't warn about root ingest for stat-only
    * RNDEAS-1031: Error when filtering core plugins by mimetype
    * RNDEAS-1032: More descriptive error for bad plugin filters
    * RNDEAS-1033: Bad error message from metadata validation error
    * RNDEAS-1035: Don't try to calculate hash for directories
    * RNDEAS-1040: apsearch-ingest doesn't show full logging output
    * RNDEAS-1042: Validate plugin tool doesn't recognise 'remote' type
    * RNDEAS-1065: Make sure 'broker' user can 'see' deleted files in search
    * RNDEAS-1069: Skip paths containing non-unicode bytes

Feature enhancements:

    * RNDEAS-242: Support for ingest with selected plugins
    * RNDEAS-863: Support for hiding proxies from selected users
    * RNDEAS-995: Method for checking if a core plugin already generated proxies for a file
    * RNDEAS-996: OpenAPI (Swagger) config for the REST API
    * RNDEAS-1008: Infer date parse order from locale + support for explicit configuration
    * RNDEAS-1030: File access security takes into account user roles
    * RNDEAS-1054: Set ingest syslog identifier to 'arcapix.search.tools.finder'
    * RNDEAS-1064: Support for OPTIONS requests to check endpoint permissions
    * RNDEAS-1053: Support for incremental ingest using file change time (ctime)

Refactoring:

    * RNDEAS-1039: Make use of Eve roles configs


0.8.0 - 2019-11-22
==================

Bug fixes:

    * RNDEAS-585: Plugin validate - supress misleading tracebacks
    * RNDEAS-805: Reduce unnecessary logger warnings
    * RNDEAS-933: Mimetype incorrectly identified for directories
    * RNDEAS-963: Validation error when sha512 hash can't be calculated
    * RNDEAS-988: Prune directory plugin deletes renamed files
    * RNDEAS-1003: Tweak to offline file check

Feature enhancements:

    * RNDEAS-428: Optimised single-file ingest
    * RNDEAS-442: Support for multi-node ingest
    * RNDEAS-625: Support for exiftool 'large file support'
    * RNDEAS-651: Plugin validate - check value precision
    * RNDEAS-654: Plugin validate - check schema against elasticsearch
    * RNDEAS-800: Support for 'wand' as an image converter
    * RNDEAS-882: Plugin validate - add profiling information
    * RNDEAS-907: Record per-plugin last successful ingest times
    * RNDEAS-935: Support for ingesting files from list
    * RNDEAS-940: Support for only re-ingesting failed and out-dated plugins
    * RNDEAS-943: Flexible last ingest timestamp support
    * RNDEAS-975: Support for recording lists of files which failed to ingest

Performance enhancements:

    * RNDEAS-251: Use bulk inserts in ingest
    * RNDEAS-441: Use asynchronous REST calls in ingest
    * RNDEAS-883: Use persistent ExifTool
    * RNDEAS-906: Implement mime.types as a method for determining mimetype
    * RNDEAS-942: Support for specifying preferred image converters
    * RNDEAS-972: Plugin profiling tool

Refactoring:

    * RNDEAS-824: apsearch-ingest script re-written to use yaml config
    * RNDEAS-844: Incremental ingest no longer uses snapshots
    * RNDEAS-895: Move file_security to apcore-utils

Incompatible changes:

    * PillowThumbnail was renamed to CoreImageThumbnail.
        Any plugins which import it - including ExtendedImageThumbnail - will need updating


0.7.1 - 2019-08-05
==================

Bug fixes:

    * RNDEAS-874: Allow proxies to inherit acls
    * RNDEAS-913: Import error for async user plugins
    * RNDEAS-916: Only handle mov files based on mimetype
    * RNDEAS-918: Don't error on lock cleanup if lock file is missing
    * RNDEAS-919: Recursion error from cleandb HTTPError
    * RNDEAS-920: Provide logging output for cleandb and other tools

Feature enhancements:

    * RNDEAS-901: Tool for locating proxies on the filesystem
    * RNDEAS-922: Safety check to verify mimetype for video plugins


0.7.0 - 2019-05-30
==================

Bug fixes:

    * RNDEAS-571: Handle max results error nicely
    * RNDEAS-632: Improve results reporting from finder
    * RNDEAS-705: Cleandb error when directory contains more than 10,000 entries
    * RNDEAS-730: Sudoers file to specify what commands need to run as root
    * RNDEAS-768: Rename common plugin to reduce confusion re whether it is optional
    * RNDEAS-787: Handle KeyboardInterrupt nicely in find_duplicates
    * RNDEAS-843: 500 error when proxy wasn't found
    * RNDEAS-845: Improved error reporting when ETAG can't be fetched during ingest
    * RNDEAS-856: Support for PSD, PSB, and EXR file types
    * RNDEAS-870: Stat-only ingest doesn't change access time (atime)
    * RNDEAS-894: Allow users with admin role to see all files

Feature enhancements:

    * Add profiling information for plugins

Incompatible changes:

    * RNDEAS-796: Logging module was removed. Standalone 'aplogging' package is required instead.


0.6.2 - 2019-02-28
==================

Documentation Update:

    * Correct URL's in documentation after change to /api


0.6.1 - 2019-02-25
==================

Bug fixes:

    * RNDEAS-793: Error on desktop connector dynamic import


0.6.0 - 2019-02-19
==================

Feature enhancements:

    * RNDEAS-707: Support for Dynamic Plugins and Desktop Connector
    * RNDEAS-688: Support for boolean operators in '_all' queries
    * RNDEAS-738: Support for incoming Collection+JSON
    * Client tools for EDL Export, File List Export, and Attribute adjustment (BETA)

Refactoring:

    * RNDEAS-640: Move non-stat plugins out of core


0.5.0 - 2018-11-14
==================

Bug fixes:

    * RNDEAS-655: Finder stop tries to read pid from lock file
    * RNDEAS-675: Error when snapshot history file in manually cleared
    * RNDEAS-676: Refine finder include/exclude matching
    * RNDEAS-679: Permission error when checking if files are offline

Feature enhancements:

    * Support for combining database queries


0.4.0 - 2018-10-03
==================

Bug fixes:

    * RNDEAS-474: File security check error for unicode filenames
    * RNDEAS-559: Text fields being sorted by tokenised values
    * RNDEAS-562: PickleError from 'finder remove'
    * RNDEAS-575: Make file security checks nGeanea-safe
    * RNDEAS-599: Make sure async job engine can import user plugins
    * RNDEAS-618: Add missing config documentation
    * RNDEAS-621: Exiftool fails for large video files
    * RNDEAS-628: Ingest dies if file can't be read
    * RNDEAS-629: Security checker randomly fails
    * RNDEAS-630: Validate plugin reports 'hint' field as unknown field error

Feature Enhancements:

    * RNDEAS-175: Support for offline files
    * RNDEAS-447: Tool for cleaning up orphaned proxy files
    * RNDEAS-519: Support for interfacing with external resources
    * RNDEAS-576: Add file-type (core.type) metadata field
    * RNDEAS-637: Search recursively for plugins in plugin path

Refactoring:

    * RNDEAS-504: APConfig and APAuth split into separate packages
    * RNDEAS-523: Configurable lock path
    * RNDEAS-534: Remove misleading logging of exceptions on proxy move
    * RNDEAS-557: Increase pagination limit to 1000
    * RNDEAS-558: Make date aggregations return ints rather than floats
    * RNDEAS-573: Configurable proxy-gen work directory
    * RNDEAS-579: Remove default value for proxy-gen work directory - must be configured explicitly
    * RNDEAS-591: Convert classmethod ``generate_temp_filename`` to standalone utility function
    * RNDEAS-613: Async jobs only run on HTCondor nodes configured for apsearch
    * RNDEAS-622: Remove misleading logging of exception when Pillow can't open an image
    * RNDEAS-625: Improved handling of errors from exiftool
    * RNDEAS-649: Change type of core.user.id and core.group.id to long


0.3.1 - 2018-04-13
==================

Bug fixes:

    * RNDEAS-565: issues generating docs for rpm build

Feature enhancement:

    * RNDEAS-487: move example plugins to public repo
    * RNDEAS-569: example plugin for extended image thumbnail support


0.3.0 - 2018-02-01
==================

Bug fixes:

    * RNDGPA-234: fix uid/gid lookup on SSSD
    * RNDEAS-445: proxygen fails when proxy already exists

Feature enhancements:

    * RNDEAS-465: add plugin validation tool
    * RNDEAS-499: support for incremental ingest (scan) without snapshots
    * RNDEAS-501: add ingest locking
    * RNDEAS-501: add support for email alert on ingest error
    * RNDEAS-501: add ``stop`` command to kill ingest gracefully
    * RNDEAS-502: improve snapshot rotation for ingest
    * RNDEAS-506: database cleanup tool

Incompatible changes:

    * RNDEAS-268: logging controlled via ``APDEBUGLOG`` and ``APDEBUGLEVEL`` environment variables
    * RNDEAS-500: rename ``monitor`` to ``finder``
    * Various database schema changes - NOTE: you will need to delete any existing Elastic DB
    * Integrated ``eve-easysearch`` package - NOTE: you may need to remove the existing package manually


0.2 - 2017-07-14
================

    * Initial release