Tasks

Tasks are functions which are called when an object is created (POST) updated (PATCH) or deleted (DELETE).

These tasks are called via database event hooks, and they perform changes to the underlying filesystem, corresponding to the database change being performed.

For example, when a template is deleted, a TemplateDeleter task will remove the corresponding template files from the filesystem.

Tasks are executed via the particular Job Engine being used

API Documentation

ORM Task

ORMTask is the base class for all tasks which affect objects on the filesystem.

class arcapix.management.tasks.ormtask.DBTask

Specialisation of the Task base class for altering the database.

Contains a reference to the database driver from which sessions can be created. This must be set before attempting to use ORM Tasks.

Parameters:
  • type – Class object representing the type of object the task applies to
  • id (int) – numerical id of object the task applies to
  • etag (int) – etag of the object at the time the job was submitted If this changes by the time the job runs, we have an issue

Params are used to look-up the relevant items from the SQL database

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

class arcapix.management.tasks.ormtask.ORMTask(type_, id_, etag=None)

Specialisation of the Task base class for dealing with our ORMed stuff.

This is further sub-classed by the individual tasks that are defined in other parts of this python package

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

Exposer Tasks

class arcapix.management.tasks.exposertasks.CIFSExposerBuilder(type_, id_, etag=None)

Task to create a CIFS exposer.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Add a new share to the SMB config file.

post(resultstatus)

Write updated config back to file.

pre()

Load samba config.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.CIFSExposerDeleter(type_, id_, etag=None)

Task to delete a CIFS exposer.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Remove the relevant share from SMB config file.

post(resultstatus)

Write updated config back to file.

pre()

Load samba config.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.CIFSExposerUpdater(*args, **kwargs)

Task to update a CIFS exposer.

An extras field with a value of None will cause that field to be ‘unset’

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Change the name or path of a share in SMB config.

post(resultstatus)

Write updated config back to file.

pre()

Load samba config.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.CIFSTask(type_, id_, etag=None)

Base class for CIFS tasks.

Handles reading and writing the samba config file.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Write updated config back to file.

pre()

Load samba config.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.NFSExposerBuilder(type_, id_, etag=None)

Task to create a NFS exposer.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Add a new share to the NFS config file.

post(resultstatus)

Write updated config back to file.

pre()

Load NFS config file.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.NFSExposerDeleter(type_, id_, etag=None)

Task to delete a NFS exposer.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Remove the relevant share from NFS config file.

post(resultstatus)

Write updated config back to file.

pre()

Load NFS config file.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.NFSExposerUpdater(*args, **kwargs)

Task to update a NFS exposer.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Change the name or path of a share in NFS config.

post(resultstatus)

Write updated config back to file.

pre()

Load NFS config file.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.exposertasks.NFSTask(type_, id_, etag=None)

Base class for NFS tasks.

Handles reading and writing the nfs config file.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Write updated config back to file.

pre()

Load NFS config file.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

arcapix.management.tasks.exposertasks.sync_configs()

Sync nfs and smb configs using csync.

Adapted from fixstor

Profile Tasks

class arcapix.management.tasks.profiletasks.ProfileBuilder(type_, id_, etag=None)

Task which makes the changes required for a new Profile.

Key things needed are setting up of the relevant placement rule for the first and subsequent ILM steps.

Each step’s source pool is the target pool from the prior step

i.e. (Placement) -> pool_1 -> (Management_1) -> pool_2 -> (Management_2) …

Filesets will use names to match for pool positioning

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Create the relevant rules in the filesystem placement policy.

post(resultstatus)

Save changes to policy.

pre()

Load relevant placement policy.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.profiletasks.ProfileDeleter(type_, id_, etag=None)

Task to delete a profile.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Remove the policy rules corresponding to the profile.

post(resultstatus)

Save changes to policy.

pre()

Load relevant placement policy.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.profiletasks.ProfileTask(type_, id_, etag=None)

Generic Profile task to handle load/save of placement policy.

Doing this in pre/post rather than in perform makes unittesting easier.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Save changes to policy.

pre()

Load relevant placement policy.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.profiletasks.ProfileUpdater(type_, id_, etag=None)

Task to update a profile.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Update the policy rules corresponding to the profile.

Essentially just finds rules in the placement policy corresponding to Profile ILM Steps and modifies their associated inline comments to include the new name

post(resultstatus)

Save changes to policy.

pre()

Load relevant placement policy.

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

Space Tasks

class arcapix.management.tasks.spacetasks.SpaceBuilder(type_, id_, etag=None)

Task to create a new space.

classmethod default_resource_requirements()

List of resources required by this space builder.

get_db()

Gets a database session object for the database associated with the task

get_exposer()

Helper method - find a GPFSNativeExposer in the space’s exposers list.

get_id()

Get the resource id associated with this task.

Get the link path for the space’s native exposer.

Returns None if the space doesn’t have a native exposer (unlinked)

get_placement_filesystem()

Get the space’s placement filesystem.

get_resource()

Get the resource associated with this task.

perform()

Creates an Independent Fileset.

Also sets up any profile, template, etc.

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.spacetasks.SpaceDeleter(type_, id_, etag=None)

Task to delete a space.

This can only work with EVE versions which support SOFT_DELETE (>=0.6)

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_exposer()

Helper method - find a GPFSNativeExposer in the space’s exposers list.

get_id()

Get the resource id associated with this task.

Get the link path for the space’s native exposer.

Returns None if the space doesn’t have a native exposer (unlinked)

get_placement_filesystem()

Get the space’s placement filesystem.

get_resource()

Get the resource associated with this task.

perform()

Unlink and delete the relevant fileset.

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.spacetasks.SpaceUpdater(*args, **kwargs)

Tasks to update Space properties.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_exposer()

Helper method - find a GPFSNativeExposer in the space’s exposers list.

get_id()

Get the resource id associated with this task.

Get the link path for the space’s native exposer.

Returns None if the space doesn’t have a native exposer (unlinked)

get_placement_filesystem()

Get the space’s placement filesystem.

get_resource()

Get the resource associated with this task.

perform()

Change fileset properties as requested.

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

arcapix.management.tasks.spacetasks.get_path_inode_space(filesystem, path)

Get the inode space that an arbitrary path belongs to.

Unlink fileset, with rollback if there’s an error in context.

Snapshots Tasks

class arcapix.management.tasks.snapshottasks.SnapshotBuilder(type_, id_, etag=None)
classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.snapshottasks.SnapshotDeleter(type_, id_, etag=None)
classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Main part of task execution.

Returns:the state of the resource after perform completes. default: ACTIVE
post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

Template Tasks

class arcapix.management.tasks.templatetasks.TemplateBuilder(type_, id_, etag=None)

Task to create a template.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Create the template from a provided path.

Copies the template from a user specified location to {template_store}/{template_name}

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.templatetasks.TemplateDeleter(type_, id_, etag=None)

Task to delete a template.

Warning

You can’t delete a template that is installed in any spaces.

If you attempt to delete an installed template, this task will raise an exception and the template will enter an ERRORED state.

The ERRORED template will still be usuable, however.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Delete the template from the template store directory.

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

class arcapix.management.tasks.templatetasks.TemplateUpdater(*args, **kwargs)

Task to update a template.

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

get_id()

Get the resource id associated with this task.

get_resource()

Get the resource associated with this task.

perform()

Update the template name or replace it from a different target.

This is called before an item is updated

Future values are held in self.todo

post(resultstatus)

Post-flight.

Parameters:resultstatus – the status of the resource, as returned from perform.
pre()

Pre-flight.

Sets the resource’s status to INPROGRESS

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

Executes: pre -> perform -> post

Also takes care of boilerplate, including database interactions and error handling.

arcapix.management.tasks.templatetasks.copy_template(source, dest)

Calls the copy_template script.

The copy_template script should be added to sudoers so copying templates can be done by a non-root user and still preserve file permissions

arcapix.management.tasks.templatetasks.delete_template(location)

Calls the delete_template script.

The delete_template script should be added to sudoers so deleting templates can be done by a non-root user and still preserve file permissions

arcapix.management.tasks.templatetasks.move_template(source, dest)

Calls the move_template script.

The move_template script should be added to sudoers so moving templates can be done by a non-root user and still preserve file permissions

Template Tools

The following are commandline tools which are used by tasks. They need to be run as root to be able to manipulate templates, regardless of the permissions of the objects in the templates.

Important

Ideally, these scripts should be added to the sudoers file for the user which runs the management REST server (i.e. the user that tasks are run as).

Copy Template
class arcapix.management.tasks.copy_template.CopyNotAllowed(*args, **kwargs)

Exception raised when attempting an invalid copy operation.

arcapix.management.tasks.copy_template.copy_gpfs_acl(source, dest)

Copy gpfs acls using mm-commands.

arcapix.management.tasks.copy_template.copy_perms(source, dest)

Copy acls and ownership from one file to another.

arcapix.management.tasks.copy_template.get_inode_space(fs, path)

Get the inode space a path belongs to.

Path must be a fileset link path.

arcapix.management.tasks.copy_template.get_pool(fset)

Polls the filesystem PlacecmentPolicy to try to figure out what pool files in ‘fset’ are assigned to.

Fileset placement can be specified either as “FOR FILESET …” or “WHERE FILESET_NAME …”

If no specific placement rule exists, the default placement rule is used.

Any pool macros are resolved.

Parameters:fset (Filesets object) – fileset to find placement pool for
Returns:pool name
arcapix.management.tasks.copy_template.is_parent_dir(parent, child)

Check if a path is a parent directory of another path.

Delete Template

delete_template – tool for removing a template from the template store.

This is essentially just a way of calling shutil.rmtree with sudo.

However, that is potentially very dangerous, so we have various checks to make sure the script can ONLY be used to uninstall templates

arcapix.management.tasks.delete_template.delete_template(location)

Remove a template from the template store.

Move Template

move_template – tool for moving (renaming) templates in the template store.

This is essentially just a way of calling os.rename with sudo.

However, that is potentially very dangerous, so we have various checks to make sure the script can ONLY be used to move templates WITHIN the configure template store

class arcapix.management.tasks.move_template.MoveNotAllowed(*args, **kwargs)

Exception raised when attempting an invalid move operation.

arcapix.management.tasks.move_template.move_template(source, target)

Remove a template from the template store.

Uninstall Template
class arcapix.management.tasks.uninstall_template.UninstallNotAllowed(*args, **kwargs)
arcapix.management.tasks.uninstall_template.check_installed_template(template_path, space_path)

Check if a template installed in a space is different from the original template.

If the template is the same, it contains no user data and we can safely remove it.

arcapix.management.tasks.uninstall_template.files_match(source, dest)

Compare two files based on stat attributes.

arcapix.management.tasks.uninstall_template.uninstall_template(template_path, space_path)

Remove a template from a space.

Assumes the space has been checked and the installed template is ‘empty’

Populate Tasks

These are methods for populating or reconciling the database based on the current state of the filesystem.

class arcapix.management.tasks.populatetasks.ReconcileDBTask

Task to perform database reconcile

To be run via the job engine

classmethod default_resource_requirements()

Returns a list of resources which tasks need.

get_db()

Gets a database session object for the database associated with the task

perform(session)

Bring the database up to date with the cluster

resource_requirements()

List the resources the task requires.

This can be overridden on a per instance basis using this function. Alternatively, see Task.default_resource_requirements(), which allows override on a per task type basis.

run()

Execute the task.

arcapix.management.tasks.populatetasks.create_default_placement_rules(cluster)

Create a default placement rule for each data pool in each filesystem.

arcapix.management.tasks.populatetasks.populate(db, _every=0, _async=True)

Submit a ReconcileDBTask to the currently configured job engine.

Convenience function

Warning

May be changed or removed in a future release.

Parameters:db – Flask-SQLAlchemy datalayer or SQLAlchemy Engine