Configuration directives

Overview

The PixStor Search system is controlled by various configuration directives. These are represented in a filesystem heirarchy, and manifest themselves as dot seperated properties. The content of the actual property values is JSON formatted. More complex properties (e.g. dicts/maps) are also used for some properties.

Important

It is especially important to note that raw strings are NOT valid JSON, therefore all string values must be appropriately quoted

The base location for ArcaPix configuration values is /opt/arcapix/etc (see below). Thus, to set the value of arcapix.search.authserver to https://localhost:5000/ one would write a file containing precisely

"https://localhost:5000"

into /opt/arcapix/etc/arcapix/search/authserver

Location

The default location for configuration values is /opt/arcapix/etc/. However, this can be overridden by setting the ARCAPIX_CONFIG environment variable to point to a suitable directory - this maybe useful for example to centralise configuration values.

Defaults

The system does not have any “default” values - all configuration values must be explictly set. This is normally done as part of the installation process.

Using Python to write the config values

One of the most convenient methods for setting configuration values is to use the supplied python library.

from arcapix.config import config
config["arcapix.search.authserver"] = "https://localhost:5001"

Configuration properties

The following properties are currently defined for search.

Note

there may be other properties present in the /opt/arcapix/etc/ heirarchy - these are for other arcapix products not connected with search

arcapix.alerts.email.from

If something goes wrong during ingest, an email alert may be sent (see below)

This config specifies the email address these alert emails should be sent from (replyto)

Type String
Suggested default value root@localhost

arcapix.alerts.email.to

If something goes wrong during ingest, an email alert may be sent. This config specifies a list of email addresses to send alterts to.

If the config is None or an empty list, no emails alerts will be sent.

Type List of Strings
Suggested default value []

arcapix.search.finder.lock.directory

Directory in which lock files are stored for the finder (ingest). Lock files prevent multiple concurrent ingests on the same filesystem.

Note - the user running finder must have write permission for the configured directory.

Type Integer
Suggested default value “/var/lock”

arcapix.search.finder.pid.directory

Directory in which pid files are stored for the finder (ingest). Pid files are used to facilitate the finder stop functionality.

Note - the user running finder must have write permission for the configured directory.

Type Integer
Suggested default value “/var/run”

arcapix.search.logs.path

The location where asynchonous jobs write their log output. Sub-directories will be created to ensure that no one directory is overloaded.

Type String
Suggested default value “/mmfs1/.policy_tmp/condor/logs”

arcapix.search.metadata.broker.username

The username which the broker will use to authenticate to the system. In theory, this could be a system user, but in practice, it makes sense to use one from the arcapix.search.server.authserver.passwords property described above

Type String
Suggested default value “broker”

arcapix.search.metadata.plugins.path

The path into which user plugins are installed. See Plugin Installation for more information.

Type String
Suggested default value “/opt/arcapix/usr/share/apsearch/plugins”

arcapix.search.monitor.exclude

A list of patterns that should always be excluded. Patterns are the same as would be passed to the monitor command line tool.

This config can be used to exclude files, directories, or file types that should never be ingested, e.g.

File: /mmfs1/path/to/file Directory: /mmfs1/directory/* File Type: *.tmp

Paths can be either absolute or relative (to the ingest target directory)

Type List of Strings
Suggested default value [“.policytmp/”, “.ctdb/”]

arcapix.search.monitor.gpfs.snapprefix

As part of the reconciliation process to detect deleted files, a snapshot is created. This setting determines the prefix used for the snapshot, and should be set to something which will not clash with any other usage of snapshots in your tools.

Type String
Suggested default value

“__EAS_47145616346_”

(or any suitably obfuscated string - the number has no significance at all)

arcapix.search.proxies.path

The location where proxy files will be stored. Ideally, this should be on the same filesystem/fileset as the proxy generators work with.

Type String
Suggested default value “/mmfs1/.proxies”

arcapix.search.proxies.preview.size

The size in pixels (width, height) which preview proxies are made at by the standard proxy generator tool. NB. User created proxies need not be this size (though they may be resized for display in the UI if not).

Type 2-tuple/list of integers
Suggested default value (400, 300)

arcapix.search.proxies.thumbnail.size

The size in pixels (width, height) for thumbnails.

Type 2-tuple/list of integers
Suggested default value (150, 150)

arcapix.search.proxies.retain

Specify whether proxy files should be kept when their corresponding originals are deleted from the filesystem/database.

Type Boolean
Suggested default value False

arcapix.search.proxies.workdir

The location where proxy files are generated (for builtin plugins). Once successfully generated, the proxies are moved to the proxy store.

Ideally, this should be on the same filesystem/fileset as the proxy store.

Type String
Suggested default value “/mmfs1/apsearch/proxies/.proxytmp”

arcapix.search.server.authserver.passwords

A dictionary of username/password pairings which the authentication server will also consider to be valid for the system. These can be used by automated tools to enable them to authenticate without a system account.

Type Dictonary of string => string key/values
Suggested default value

{“broker”: “7dsfdsfsrdsfsadcvxc98ds98SEg4tn,slicvxc”}

(Password should be any randomly generated string.)

Warning

The filesystem file associated with this property should be properly secured using file system permissions, such that only the automated tools and the auth server itself can read it. Otherwise, any user who can read the file can authenticate.

Warning

Do not use the password included in the suggested default value above - you should change it to some other suitable randomised string.

arcapix.search.server.authserver.url

This value defines the server which the search server can use to verify access tokens. This server _should_ use https: rather than http:, but this is not mandated.

If the auth server is behind a reverse proxy, such as nginx, this config should point to the proxied url.

Type String
Suggested default value

https://localhost/

(This url will have /oauth2/token_info appended to it automatically.)

arcapix.search.server.searchserver.url

This value defines the server which the broker will use to update the metadata in the database. Often it is useful to have this point at the “nearest” instance of the search server. If both the broker and search server are running on all nodes, using localhost is suitable.

Alternatively, if the search server is behind a reverse proxy, such as nginx, it might make sense to set this to the proxied url. That way, the broker might take advantage of load-balancing.

Type String
Suggested default value http://localhost:5000/

arcapix.search.utils.security.disable

This property determines whether file access security is enabled. With this option disabled, any authenticated user can see metadata and proxies for any file on the system, not just those which they would have access to.

Type Boolean
Suggested default value False

Warning

It is strongly recommended that this key should be secured against writing by non-root users, since otherwise they can disable the security checks, assuming they can trigger a service restart

arcapix.snapshot.rotation.path

The snapshot rotation path is a special directory which holds data used by snapshot rotation, for example, a record of the last snapshot(s) used for incremental ingest.

Type String
Suggested default value “/mmfs1/.rotate”

executables.exiftool

Metadata extraction for images and videos makes use of the exiftool exectuable. This property allows the location to be set. Depending on whether you have installed a system version, or compiled a specific one, the location may change.

Type String
Suggested default value “/usr/bin/exiftool”

executables.ffprobe

Metadata extraction for videos makes use of the ffprobe exectuable. This property allows the location to be set. Depending on whether you have installed a system version, or compiled a specific one, the location may change.

Type String
Suggested default value “/usr/bin/ffprobe”