REST API Overview

This page gives an overview of PixStor Management REST interface and functionality. For a more hands-on guide, see Walkthrough: Creating a Space

Methods

The operations available on collection endpoints are GET to list the items in a collection, and POST to create new items.

Individual resources support GET, PATCH and DELETE, to show, modify and delete individual items.

Note that “PUT to replace” is not permitted - you cannot replace an existing item with a new one. Instead, you should delete your existing item, then add a new one.

You may not have permission to use all the available methods - see Auth Roles below

Authentication

Before you can use the API, you will need an authentication token. This is done using the APAuth authentication server.

$ curl -X POST https://authserver/oauth2/token \
> -H 'content-type: application/x-www-form-urlencoded' \
> -d 'username=<user>&password=<pass>&grant_type=password'
{
  "access_token":"<token>",
  "token_type":"Bearer",
  "expires_in":86400,
}

Tip

Depending on your local setup, you might need to include the -k/--insecure flag to work with self-signed certificates, and the -L/--location flag to make curl follow redirects, such as from nginx reverse proxy. The standard PixStor installation uses, and thus requires, both of these.

For a detailed discussion of the auth server, see the APAuth documentation.

You must provide your access token with all future requests.

When working in the console, it is usually convenient to store the token in an environment variable - $TOKEN. This $TOKEN can then be provided to curl requests with the -u/--user flag

$ curl https://mypixstorserver -u $TOKEN:

Note

Notice the trailing colon - the token is used as the ‘username’ and the ‘password’ field (after the colon) is intentionally left blank

REST Server

Once you have your access token, the PixStor Management REST server can be reached locally at http://localhost:5101/ or externally at https://mypixstorserver/, where mypixstorserver is the external hostname of your pixstor. (This applies to standard Pixstor installations. Other installations may provide an alternative endpoint)

The base, or ‘billboard’, url with return a list of available resource types

$ curl http://localhost:5101/ -u $TOKEN:
{"collection":
  {"links": [
    {"render": "link", "href": "/datastores/", "prompt": "Datastores", "name": "datastores", "rel": "resource"},
    {"render": "link", "href": "/ilmsteps/", "prompt": "Ilmsteps", "name": "ilmsteps", "rel": "resource"},
    {"render": "link", "href": "/profiles/", "prompt": "Profiles", "name": "profiles", "rel": "resource"},
    {"render": "link", "href": "/spaces/", "prompt": "Spaces", "name": "spaces", "rel": "resource"},
    {"render": "link", "href": "/templates/", "prompt": "Templates", "name": "templates", "rel": "resource"},
    {"render": "link", "href": "/exposers/", "prompt": "Exposers", "name": "exposers", "rel": "resource"},
    {"render": "link", "href": "/snapshots/", "prompt": "Snapshots", "name": "snapshots", "rel": "resource"}
  ]
}}

Note

This billboard endpoint can’t be reached via the https url due to nginx proxying

The default response format is Collection+JSON. Collection+JSON (C+J) was chosen because of it’s ‘explorability’ and usefulness to software clients.

Warning

Familiarity with the semantics of Collection+JSON is advisable before attempting to use the API further.

Accessing Collections

Sending a GET request to a collection endpoint will return all items of that resource type.

For example, to get a listing of all Spaces

$ curl https://mypixstorserver/spaces/ -u $TOKEN:
{"collection": {
    "href": "/spaces/",
    "items": [
        {
            "data": [
                {
                    "name": "name",
                    "prompt": "Name",
                    "value": "root"
                },
                ...
            ],
            "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf",
            ...
        },
        {
            "data": [
                {
                    "name": "name",
                    "prompt": "Name",
                    "value": "new_space"
                },
                ...
            ],
            "href": "/spaces/f4254930-faa5-4914-8f8f-dc5106e35e10",
            ...
        },
    ...
    ],
    "links": [...],
    "queries": [...],
    "template": {
        "data": [...]
    },
    "version": "1.0"
}}

(Some of the response has been omitted for clarity).

Projections, Filtering, and Sorting

It is possible to filter which entities are returned, as well as which attributes of those entities are returned, by appending query strings to the requested URLs. These are in line with the Python EVE query language.

For example:

• return only the space whose name is bob:

  https://mypixstorserver/spaces?where={"name":"bob"}

• return any spaces that have a particular template installed:

  https://mypixstorserver/spaces?where={"templates":"3a78810a-a734-4dc2-b8d9-71998b84c131"}

• return the spaces with a GPFSNativeExposer (filesystem) called ‘mmfs1’:

  https://mypixstorserver/spaces?where={"exposers.name":"mmfs1","exposers.type":"gpfsnative"}

• return only the name attribute for all spaces:

  https://mypixstorserver/spaces?projection={"name":1}

• return spaces sorted by name:

  https://mypixstorserver/spaces?sort=name

• return spaces sorted by profile name:

  https://mypixstorserver/spaces?sort=profile.name

NB. It is rarely worth using the projection system - the protocol overheads are orders of magnitude greater than the amount of data saved by the restriction, since all data is directly retrieved from the database. However, in the case where some “full-text” fields have been included, excluding those fields can help.

Queries

The C+J includes a queries section, which lists a selection of available filters

$ curl https://mypixstorserver/spaces/ -u $TOKEN:
{"collection": {
    "href": "/spaces/",
    ...
    "queries": [
        {
            "data": [
                {
                    "prompt": "Name",
                    "name": "name",
                    "value": ""
                }
            ],
            "href": "/spaces?where={\"name\":\"{name}\"}",
            "prompt": "Search by Name",
            "rel": "search",
            "encoding": "uri-template"
        },
    ...
    "version": "1.0"
}}

The data section describes a particular field, and the href provides a template for constructing a query (filter) on that field.

So to query the name field, we would visit the href with the variable {name} replaced with the specific name we want to search

https://mypixstorserver/spaces?where={"name":"bob"}

Pagination

The result set can be (and is by default) paginated. Whilst it is best to simply allow the REST Server to paginate automatically - the server will be tuned to an appropriate page size based on the complexity of retrieving an item - you can request any particular page size you like. This is done by supplying a max_results parameter, as per the EVE standard.

For example - to return the second page of spaces with 5 results per page:

https://mypixstorserver/spaces?max_results=5&page=2

Links are supplied in the collection+JSON for next, previous, and last in line with IANA standards.

$ curl https://mypixstorserver/spaces/ -u $TOKEN:
{"collection": {
    "href": "/spaces/",
    ...
    "links": [
        {
            "render": "link",
            "href": "/spaces/",
            "prompt": "Spaces",
            "name": "self",
            "rel": "self"
        },
        {
            "render": "link",
            "href": "/spaces?page=25",
            "prompt": "Last",
            "name": "last",
            "rel": "last"
        },
        {
            "render": "link",
            "href": "/spaces?page=2",
            "prompt": "Next",
            "name": "next",
            "rel": "next"
        }
    ],
    "version": "1.0"
}}

Properties

The response may also include a properties section. This provides some metadata about the response itself.

$ curl https://mypixstorserver/spaces/ -u $TOKEN:
{"collection": {
    "href": "/spaces/",
    ...
    "properties": [
        {
            "name": "total",
            "value": 9
        },
        {
            "name": "page",
            "value": 1
        },
        {
            "name": "max_results",
            "value": 25
        }
    ]
}}

Typical properties are:

total:

The total number or results in the collection, or matched by the given query. This value can also be looked up via the X-Total-Count header

page:

For paginated results, this is the current page of results being returned

max_results:

For paginated results, the maximum number of results being returned per page

Accessing Items

If you know an individual item’s id, you can access it directly

$ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN:
{"collection": {
    "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf",
    "items": [
        {
            "data": [
                {
                    "name": "name",
                    "prompt": "Name",
                    "value": "root"
                },
                ...
            ],
            "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf",
            "links": [...]
        }
    ],
    "links": [...],
    "queries": [...],
    "template": {
        "data": [...]
    },
    "version": "1.0"
}}

URLs for individual items are also provided in each item’s ‘href’ field when the collection is queried (see example above)

Standard properties

Descriptions of the properties available for each object and relevant associations are given on the ORM page, and are as on the Object Model diagram. However, there are several properties which all items have:

id:

This is a unique identifier for this item

_etag:

This is used to verify that an object has not been modified between a GET and PATCH/DELETE operation. It must be provided with all PATCH and DELETE requests (see later)

_created:

Date/Time when the object was created in the database - this is likely to differ from when the corresponding filesystem object was created, due to filesystem operations being performed asynchronously

_modified:

Date/Time when the object was modified in the database.

status:

This is an enumerated field which currently supports the following values:

• NEW: The item was accepted into the database. This is the default state.

• PENDING: The item has changed in the database, but the associated action hasn’t started yet

• ACTIVE: The item is in a quiescent state - all associated jobs have completed successfully - and it’s ready for use

• INPROGRESS: A modification is being made to the item - a job is active. Access to the item may be interrupted, and the precise state of the item cannot be determined

• ERRORED: The most recent change to the item failed - typically the result of an error during job execution

(other values are possible, and should not be treated as an error condition

comment:

Some arbitrary comment attached to the object.

extras:

Additional settings or metadata.

Related Items

Some items have related items - for example, a Space will have associated Exposers. The collection+JSON for a Space will include a link to the relevant Exposer(s) in it’s ‘links’ section

$ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN:
{"collection":
    "items": [{
        ...
        "links": [
            ...
            {
              "href": "/exposers?where={\"id\": \"d2a89b81-f348-4bfa-aabd-a09b6327ac48\"}",
              "name": "exposers",
              "prompt": "Exposers",
              "rel": "exposers collection",
              "render": "link"
            }
        ]
    }]
}

Checking Job Status

The progress of asynchronous jobs can be monitored via the job engine REST server.

Jobs can be looked up individually at jobs/<job_id>. Alternatively, a link to all jobs associated with an item is included in the item’s ‘links’ section

$ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN:
{"collection":
    "items": [{
        ...
        "links": [
            ...
            {
              "href": "/jobs?where={\"resource_type\": \"spaces\", \"resource_id\": \"62afc25f-fa84-48ea-a738-6296cb4deebf\"}",
              "name": "jobs",
              "prompt": "Jobs",
              "rel": "jobs collection",
              "render": "link"
            }
        ]
    }]
}

Jobs are returned in id order, so to get the latest jobs you’ll need to modify the url to add e.g. &sort=-_created or &sort=-start_time

Note

job status values differ from those of other ORM objects.

Adding a new item

When you perform a GET on a collection endpoint, included in the collection+JSON response should be a template section.

$ curl https://mypixstorserver/spaces/ -u $TOKEN:
{"collection":
    ...
    "template": {
        "data": [
            {
              "name": "name",
              "prompt": "Space name",
              "value": ""
            },
            ...
        ]
}}

This template provides a listing of what fields should be provided when creating a new item.

$ curl -X POST https://mypixstorserver/spaces/ -u $TOKEN: \
  -H "Content-Type: application/vnd.collection+json" \
  -d '{"template": {
         "data": [
           {
             "name": "name",
             "value": "foo"
           },
           ...
       ]}}'

Important

You must include the Content-Type: application/vnd.collection+json header, as shown above, so that the REST server knows what JSON format is being sent

Important

The trailing slash on the end of the url /spaces/ is required. Without it, the POST request will fail.

If the POST is successful, the newly added item will be returned as the response, with status code 202 (Accepted).

This implies that the object has been submitted for processing, but that it is not yet available to the user.

Since every operation in the API is executed asynchronously via a job engine, it is very unlikely that you will receive a 200 (OK) response, although this is possible.

To determine when the process is complete, you should poll the href given in the Location header, and watch the status property. That property will transition from NEW to PENDING to INPROGRESS to ACTIVE. Once ACTIVE, the process is complete.

The POSTed data will be validated by the REST server. If the data fails validation - for example, if a value is of the wrong type - the server will return 422 (Unprocessable Entity)

If an error occurs during the job execution phase, the object status will be set to ERRORED. The logs from the associated job can be retrieved to determine the cause of the failure.

Polymorphic Objects

For polymorphic objects - objects for which there are different types - there won’t be a template section. Typically, polymorphic objects won’t share all the same fields, so a singular template doesn’t makes sense.

Instead, there will be a templates section, which contains individualised templates for each type of the object.

$ curl https://mypixstorserver/exposers/ -u $TOKEN:
{"collection":
    ...
    "templates": {
    {
      "data": [
      {
        "prompt": "exposer name",
        "name": "name",
        "value": ""
      },
      {
        "prompt": "specific type of the exposer",
        "name": "type",
        "value": "nfs"
      },
      ...
      ],
      "name": "nfs"
    },
    {
      "data": [
      {
        "prompt": "exposer name",
        "name": "name",
        "value": ""
      },
      {
        "prompt": "specific type of the exposer",
        "name": "type",
        "value": "cifs"
      },
      ...
      ],
      "name": "cifs"
    }

As with the singular template, to create a new instance of an object, you just provide values for the fields in the type-specific template

$ curl -X POST https://mypixstorserver/exposers/ -u $TOKEN: \
  -H "Content-Type: application/vnd.collection+json" \
  -d '{"template": {
         "data": [
           {
             "name": "type",
             "value": "cifs",
           {
             "name": "name",
             "value": "bar"
           },
           ...
       ]}}'

Hint

If there is neither a template or templates section in the C+J, that means the endpoint doesn’t support creating new instances.

This can be confirmed by performing an OPTIONS request and checking for POST in the Allow header.

See Auth Roles below for more information

Specifying Related Items

When specifying a related item - for example, a profile to be used with a space - values can be given as either an id, or a URL to the item.

{"template": {
    "data": [
       {
         "name": "profile",
         "value": "profiles/e07a1fd3-b518-4f15-afa6-57fe51e223c7"
       },
       ...
]}}

To specify multiple items - for example, exposers to be used with a space - values should be a comma-separated list of ids or URLs

{"template": {
    "data": [
       {
         "name": "exposers",
         "value": "d2a89b81-f348-4bfa-aabd-a09b6327ac48,ba62e6a6-84f5-4706-9cb1-e41132152ddd"
       },
       ...
]}}

Template Validations

Template items may include extra fields which provide validation hints

$ curl -X POST https://mypixstorserver/spaces/ -u $TOKEN: \
  -H "Content-Type: application/vnd.collection+json" \
  -d '{"template": {
         "data": [
           {
             "name": "name",
             "value": "foo",
             "required": true,
             "regex": "^[\\w\\-]+\\Z"
           },
           ...
       ]}}'

In this example, we see that the name field is required, and that it must match the given regex - which indicates the name must contain one or more alphanumeric characters, underscores, or dashes

These validations are applied in the REST server. If they’re not satisfied, the request will be rejected and the server will return code 422 (Unprocessable Entity)

The validation hints allow you to pre-check your data before sending it to the REST server. This may be useful when developing a client or UI for the REST API

Note

The REST server may perform additional validation checks, so data could still be invalid, even if it satisfies the provided validation hints.

Updating an item

As with adding a new item, to update an item you should use the template returned by the GET request. In this case, you only need to provide the fields you wish to modify.

The update template, as returned by an individual item, may differ from the create template, as returned by the collection endpoint.

This happens when there are fields which can’t be updated once set.

For example, you must provide a name when creating a space. But changing the name of a space is not currently supported. So, the name field will appear in the space create template, but won’t appear in the space update template.

If you try to change one of these ‘immutable’ fields, you will get back a 422 (Unprocessable Entity) error.

If there’s no template section at all in the C+J, that means the endpoint doesn’t support updating the item in any way. This can be confirmed by performing an OPTIONS request and checking for PATCH in the Allow header.

See Auth Roles below for more information

Etags

In order to prevent a race condition - where an item has been modified between GET-ing the current state and PATCH-ing an update - you must supply the item’s etag with the PATCH request.

The object’s etag can be retrieved from a GET request on the item, either in the response header or data.

$ curl -i https://mypixstorserver/spaces/6 -u $TOKEN:

HTTP/1.0 200 OK
Content-Type: application/vnd.collection+json
Content-Length: 6385
ETag: "8788611043145"
Last-Modified: Fri, 02 Nov 2018 13:30:03 GMT
Server: Eve/0.7.2 Werkzeug/0.10.4 HTCondor/8.6.12 Python/3.6
Date: Fri, 22 Feb 2019 10:43:38 GMT

{"collection":
    ...
    "items": [{
        "data": [{
            "name": "_etag",
            "prompt": "_etag",
            "value": "8788611043145"
          },
        ...
        ]
    }]
    "template": {
        "data": [
            {
              "name": "size",
              "prompt": "hard limit on the size of the space in blocks",
              "value": ""
            },
            ...
        ]
}}

The etag is then included as an If-Match header in the PATCH request

$ curl -X PATCH https://mypixstorserver/spaces/4a40b050-0bae-45d8-a554-6198af95e09c -u $TOKEN: \
  -H "Content-Type: application/vnd.collection+json" \
  -H "If-Match: 8788611043145" \
  -d '{"template": {
        "data": [{
          "name": "size",
          "value": 1024}]
      }}'

Again, if the PATCH request is successful, the updated item will be returned as the response, with status code 202 (Accepted).

If you omit the etag, or if the etag doesn’t match the one on the server, the request will be rejected with status code 412 (Precondition Failed).

As with POST, if the PATCHed data fails validation, 422 (Unprocessable Entity) will be returned

Deleting an item

As with PATCH, you need to send the item etag along with a DELETE request. This ensures that operations will not be lost.

$ curl -X DELETE https://mypixstorserver/spaces/4a40b050-0bae-45d8-a554-6198af95e09c -u $TOKEN: -H "If-Match: 574575684686"

If the request is successful, you will get back an empty response with status code 204 (No Content)

Note

Items can only be deleted individually. It is not possible to perform a bulk-delete against a collection endpoint

Auth Roles

Your auth token will have associated with it a collection of auth roles. These auth roles determine whether you will have permission to perform certain operations on a given endpoint.

You can check your auth roles using the auth server’s /tokeninfo endpoint.

Alternatively, you can perform an OPTIONS request against an endpoint and check the Allow header

$ curl -X OPTIONS -i https://mypixstorserver/spaces/ -u $TOKEN:
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Thu, 08 Nov 2018 12:26:44 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Allow: HEAD, GET, POST, OPTIONS

In this case, we see that our token grants us permission to view (GET) and create (POST) spaces. If our token didn’t grant us write access, to the /spaces endpoint we wouldn’t see POST included.

If you performed an OPTIONS request against a specific item, you would also see PATCH and DELETE if you have write permission.

The allowed methods may vary for different endpoints - e.g. you could have permission to create spaces, but not profiles.

The Allow header will also tell you if the resource or items supports a particular method at all, regardless of your auth roles.

For example, datastores can’t be created, updated, or deleted. So their Allow headers will never contain POST, PATCH, or DELETE, even if you have an auth role which would otherwise allow you to perform such operations.