################# REST API Overview ################# This page gives an overview of PixStor Management REST interface and functionality. For a more hands-on guide, see :doc:`space-walkthru` Methods ======= The operations available on collection endpoints are ``GET`` to list the items in a collection, and ``POST`` to create new items. Individual resources support ``GET``, ``PATCH`` and ``DELETE``, to show, modify and delete individual items. Note that "PUT to replace" is not permitted - you cannot replace an existing item with a new one. Instead, you should delete your existing item, then add a new one. You may not have permission to use all the available methods - see `Auth Roles`_ below Authentication =============== Before you can use the API, you will need an authentication token. This is done using the ``APAuth`` authentication server. .. code-block:: console $ curl -X POST https://authserver/oauth2/token \ > -H 'content-type: application/x-www-form-urlencoded' \ > -d 'username=&password=&grant_type=password' { "access_token":"", "token_type":"Bearer", "expires_in":86400, } .. tip:: Depending on your local setup, you might need to include the ``-k/--insecure`` flag to work with self-signed certificates, and the ``-L/--location`` flag to make curl follow redirects, such as from nginx reverse proxy. The standard PixStor installation uses, and thus requires, both of these. For a detailed discussion of the auth server, see the APAuth documentation. You must provide your access token with all future requests. When working in the console, it is usually convenient to store the token in an environment variable - ``$TOKEN``. This ``$TOKEN`` can then be provided to curl requests with the ``-u/--user`` flag .. code-block:: console $ curl https://mypixstorserver -u $TOKEN: .. note:: Notice the trailing colon - the token is used as the 'username' and the 'password' field (after the colon) is intentionally left blank REST Server ============ Once you have your access token, the PixStor Management REST server can be reached locally at ``http://localhost:5101/`` or externally at ``https://mypixstorserver/``, where ``mypixstorserver`` is the external hostname of your pixstor. (This applies to standard Pixstor installations. Other installations may provide an alternative endpoint) The base, or 'billboard', url with return a list of available resource types .. code-block:: console $ curl http://localhost:5101/ -u $TOKEN: {"collection": {"links": [ {"render": "link", "href": "/datastores/", "prompt": "Datastores", "name": "datastores", "rel": "resource"}, {"render": "link", "href": "/ilmsteps/", "prompt": "Ilmsteps", "name": "ilmsteps", "rel": "resource"}, {"render": "link", "href": "/profiles/", "prompt": "Profiles", "name": "profiles", "rel": "resource"}, {"render": "link", "href": "/spaces/", "prompt": "Spaces", "name": "spaces", "rel": "resource"}, {"render": "link", "href": "/templates/", "prompt": "Templates", "name": "templates", "rel": "resource"}, {"render": "link", "href": "/exposers/", "prompt": "Exposers", "name": "exposers", "rel": "resource"}, {"render": "link", "href": "/snapshots/", "prompt": "Snapshots", "name": "snapshots", "rel": "resource"} ] }} .. note:: This billboard endpoint can't be reached via the ``https`` url due to nginx proxying The default response format is `Collection+JSON `_. Collection+JSON (C+J) was chosen because of it's 'explorability' and usefulness to software clients. .. warning:: Familiarity with the semantics of Collection+JSON is advisable before attempting to use the API further. Accessing Collections ====================== Sending a GET request to a collection endpoint will return all items of that resource type. For example, to get a listing of all Spaces .. code-block:: console $ curl https://mypixstorserver/spaces/ -u $TOKEN: {"collection": { "href": "/spaces/", "items": [ { "data": [ { "name": "name", "prompt": "Name", "value": "root" }, ... ], "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf", ... }, { "data": [ { "name": "name", "prompt": "Name", "value": "new_space" }, ... ], "href": "/spaces/f4254930-faa5-4914-8f8f-dc5106e35e10", ... }, ... ], "links": [...], "queries": [...], "template": { "data": [...] }, "version": "1.0" }} (Some of the response has been omitted for clarity). Projections, Filtering, and Sorting ------------------------------------ It is possible to filter which entities are returned, as well as which attributes of those entities are returned, by appending query strings to the requested URLs. These are in line with the Python EVE query language. For example: * return only the space whose name is bob: ``https://mypixstorserver/spaces?where={"name":"bob"}`` * return any spaces that have a particular template installed: ``https://mypixstorserver/spaces?where={"templates":"3a78810a-a734-4dc2-b8d9-71998b84c131"}`` * return the spaces with a GPFSNativeExposer (filesystem) called 'mmfs1': ``https://mypixstorserver/spaces?where={"exposers.name":"mmfs1","exposers.type":"gpfsnative"}`` * return only the name attribute for all spaces: ``https://mypixstorserver/spaces?projection={"name":1}`` * return spaces sorted by name: ``https://mypixstorserver/spaces?sort=name`` * return spaces sorted by profile name: ``https://mypixstorserver/spaces?sort=profile.name`` NB. It is rarely worth using the projection system - the protocol overheads are orders of magnitude greater than the amount of data saved by the restriction, since all data is directly retrieved from the database. However, in the case where some "full-text" fields have been included, excluding those fields can help. Queries _______ The C+J includes a ``queries`` section, which lists a selection of available filters .. code-block:: console $ curl https://mypixstorserver/spaces/ -u $TOKEN: {"collection": { "href": "/spaces/", ... "queries": [ { "data": [ { "prompt": "Name", "name": "name", "value": "" } ], "href": "/spaces?where={\"name\":\"{name}\"}", "prompt": "Search by Name", "rel": "search", "encoding": "uri-template" }, ... "version": "1.0" }} The ``data`` section describes a particular field, and the ``href`` provides a template for constructing a query (filter) on that field. So to query the ``name`` field, we would visit the ``href`` with the variable ``{name}`` replaced with the specific name we want to search ``https://mypixstorserver/spaces?where={"name":"bob"}`` Pagination ----------- The result set can be (and is by default) paginated. Whilst it is best to simply allow the REST Server to paginate automatically - the server will be tuned to an appropriate page size based on the complexity of retrieving an item - you can request any particular page size you like. This is done by supplying a ``max_results`` parameter, as per the EVE standard. For example - to return the second page of spaces with 5 results per page: ``https://mypixstorserver/spaces?max_results=5&page=2`` Links are supplied in the collection+JSON for ``next``, ``previous``, and ``last`` in line with IANA standards. .. code-block:: console $ curl https://mypixstorserver/spaces/ -u $TOKEN: {"collection": { "href": "/spaces/", ... "links": [ { "render": "link", "href": "/spaces/", "prompt": "Spaces", "name": "self", "rel": "self" }, { "render": "link", "href": "/spaces?page=25", "prompt": "Last", "name": "last", "rel": "last" }, { "render": "link", "href": "/spaces?page=2", "prompt": "Next", "name": "next", "rel": "next" } ], "version": "1.0" }} Properties ---------- The response may also include a ``properties`` section. This provides some metadata about the response itself. .. code-block:: console $ curl https://mypixstorserver/spaces/ -u $TOKEN: {"collection": { "href": "/spaces/", ... "properties": [ { "name": "total", "value": 9 }, { "name": "page", "value": 1 }, { "name": "max_results", "value": 25 } ] }} Typical properties are: total: The total number or results in the collection, or matched by the given query. This value can also be looked up via the ``X-Total-Count`` header page: For paginated results, this is the current page of results being returned max_results: For paginated results, the maximum number of results being returned per page Accessing Items ================ If you know an individual item's id, you can access it directly .. code-block:: console $ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN: {"collection": { "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf", "items": [ { "data": [ { "name": "name", "prompt": "Name", "value": "root" }, ... ], "href": "/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf", "links": [...] } ], "links": [...], "queries": [...], "template": { "data": [...] }, "version": "1.0" }} URLs for individual items are also provided in each item's 'href' field when the collection is queried (see example above) Standard properties -------------------- Descriptions of the properties available for each object and relevant associations are given on the :doc:`orm` page, and are as on the Object Model diagram. However, there are several properties which all items have: id: This is a unique identifier for this item _etag: This is used to verify that an object has not been modified between a GET and PATCH/DELETE operation. It must be provided with all PATCH and DELETE requests (see later) _created: Date/Time when the object was created in the database - this is likely to differ from when the corresponding filesystem object was created, due to filesystem operations being performed asynchronously _modified: Date/Time when the object was modified in the database. status: This is an enumerated field which currently supports the following values: * ``NEW``: The item was accepted into the database. This is the default state. * ``PENDING``: The item has changed in the database, but the associated action hasn't started yet * ``ACTIVE``: The item is in a quiescent state - all associated jobs have completed successfully - and it's ready for use * ``INPROGRESS``: A modification is being made to the item - a job is active. Access to the item may be interrupted, and the precise state of the item cannot be determined * ``ERRORED``: The most recent change to the item failed - typically the result of an error during job execution (other values are possible, and should not be treated as an error condition comment: Some arbitrary comment attached to the object. extras: Additional settings or metadata. Related Items -------------- Some items have related items - for example, a Space will have associated Exposers. The collection+JSON for a Space will include a link to the relevant Exposer(s) in it's 'links' section .. code-block:: console :emphasize-lines: 8 $ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN: {"collection": "items": [{ ... "links": [ ... { "href": "/exposers?where={\"id\": \"d2a89b81-f348-4bfa-aabd-a09b6327ac48\"}", "name": "exposers", "prompt": "Exposers", "rel": "exposers collection", "render": "link" } ] }] } Checking Job Status ____________________ The progress of asynchronous jobs can be monitored via the job engine REST server. Jobs can be looked up individually at ``jobs/``. Alternatively, a link to all jobs associated with an item is included in the item's 'links' section .. code-block:: console :emphasize-lines: 8 $ curl https://mypixstorserver/spaces/62afc25f-fa84-48ea-a738-6296cb4deebf -u $TOKEN: {"collection": "items": [{ ... "links": [ ... { "href": "/jobs?where={\"resource_type\": \"spaces\", \"resource_id\": \"62afc25f-fa84-48ea-a738-6296cb4deebf\"}", "name": "jobs", "prompt": "Jobs", "rel": "jobs collection", "render": "link" } ] }] } Jobs are returned in id order, so to get the latest jobs you'll need to modify the url to add e.g. ``&sort=-_created`` or ``&sort=-start_time`` .. note:: job ``status`` values differ from those of other ORM objects. Adding a new item ================== When you perform a GET on a collection endpoint, included in the collection+JSON response should be a ``template`` section. .. code-block:: console $ curl https://mypixstorserver/spaces/ -u $TOKEN: {"collection": ... "template": { "data": [ { "name": "name", "prompt": "Space name", "value": "" }, ... ] }} This template provides a listing of what fields should be provided when creating a new item. .. code-block:: sh $ curl -X POST https://mypixstorserver/spaces/ -u $TOKEN: \ -H "Content-Type: application/vnd.collection+json" \ -d '{"template": { "data": [ { "name": "name", "value": "foo" }, ... ]}}' .. important:: You must include the ``Content-Type: application/vnd.collection+json`` header, as shown above, so that the REST server knows what JSON format is being sent .. important:: The trailing slash on the end of the url ``/spaces/`` is **required**. Without it, the POST request will fail. If the POST is successful, the newly added item will be returned as the response, with status code ``202 (Accepted)``. This implies that the object has been submitted for processing, but that it is not yet available to the user. Since every operation in the API is executed asynchronously via a job engine, it is very unlikely that you will receive a ``200 (OK)`` response, although this is possible. To determine when the process is complete, you should poll the ``href`` given in the ``Location`` header, and watch the ``status`` property. That property will transition from NEW to PENDING to INPROGRESS to ACTIVE. Once ACTIVE, the process is complete. The POSTed data will be validated by the REST server. If the data fails validation - for example, if a value is of the wrong type - the server will return ``422 (Unprocessable Entity)`` If an error occurs during the job execution phase, the object status will be set to ERRORED. The logs from the associated job can be retrieved to determine the cause of the failure. Polymorphic Objects ------------------- For polymorphic objects - objects for which there are different types - there won't be a ``template`` section. Typically, polymorphic objects won't share all the same fields, so a singular template doesn't makes sense. Instead, there will be a ``templates`` section, which contains individualised templates for each type of the object. .. code-block:: sh $ curl https://mypixstorserver/exposers/ -u $TOKEN: {"collection": ... "templates": { { "data": [ { "prompt": "exposer name", "name": "name", "value": "" }, { "prompt": "specific type of the exposer", "name": "type", "value": "nfs" }, ... ], "name": "nfs" }, { "data": [ { "prompt": "exposer name", "name": "name", "value": "" }, { "prompt": "specific type of the exposer", "name": "type", "value": "cifs" }, ... ], "name": "cifs" } As with the singular template, to create a new instance of an object, you just provide values for the fields in the type-specific template .. code-block:: sh $ curl -X POST https://mypixstorserver/exposers/ -u $TOKEN: \ -H "Content-Type: application/vnd.collection+json" \ -d '{"template": { "data": [ { "name": "type", "value": "cifs", { "name": "name", "value": "bar" }, ... ]}}' .. hint:: If there is neither a ``template`` or ``templates`` section in the C+J, that means the endpoint doesn't support creating new instances. This can be confirmed by performing an ``OPTIONS`` request and checking for ``POST`` in the ``Allow`` header. See `Auth Roles`_ below for more information Specifying Related Items ------------------------- When specifying a related item - for example, a profile to be used with a space - values can be given as either an id, or a URL to the item. .. code-block:: javascript {"template": { "data": [ { "name": "profile", "value": "profiles/e07a1fd3-b518-4f15-afa6-57fe51e223c7" }, ... ]}} To specify multiple items - for example, exposers to be used with a space - values should be a comma-separated list of ids or URLs .. code-block:: javascript {"template": { "data": [ { "name": "exposers", "value": "d2a89b81-f348-4bfa-aabd-a09b6327ac48,ba62e6a6-84f5-4706-9cb1-e41132152ddd" }, ... ]}} Template Validations -------------------- Template items may include extra fields which provide validation hints .. code-block:: sh $ curl -X POST https://mypixstorserver/spaces/ -u $TOKEN: \ -H "Content-Type: application/vnd.collection+json" \ -d '{"template": { "data": [ { "name": "name", "value": "foo", "required": true, "regex": "^[\\w\\-]+\\Z" }, ... ]}}' In this example, we see that the name field is required, and that it must match the given regex - which indicates the name must contain one or more alphanumeric characters, underscores, or dashes These validations are applied in the REST server. If they're not satisfied, the request will be rejected and the server will return code ``422 (Unprocessable Entity)`` The validation hints allow you to pre-check your data before sending it to the REST server. This may be useful when developing a client or UI for the REST API .. note:: The REST server may perform additional validation checks, so data could still be invalid, even if it satisfies the provided validation hints. Updating an item ================= As with adding a new item, to update an item you should use the ``template`` returned by the GET request. In this case, you only need to provide the fields you wish to modify. The update template, as returned by an individual item, may differ from the create template, as returned by the collection endpoint. This happens when there are fields which can't be updated once set. For example, you must provide a name when creating a space. But changing the name of a space is not currently supported. So, the ``name`` field will appear in the space create template, but won't appear in the space update template. If you try to change one of these 'immutable' fields, you will get back a ``422 (Unprocessable Entity)`` error. If there's no ``template`` section at all in the C+J, that means the endpoint doesn't support updating the item in any way. This can be confirmed by performing an ``OPTIONS`` request and checking for ``PATCH`` in the ``Allow`` header. See `Auth Roles`_ below for more information Etags ----- In order to prevent a race condition - where an item has been modified between GET-ing the current state and PATCH-ing an update - you must supply the item's etag with the PATCH request. The object's etag can be retrieved from a GET request on the item, either in the response header or data. .. code-block:: sh :emphasize-lines: 6,17 $ curl -i https://mypixstorserver/spaces/6 -u $TOKEN: HTTP/1.0 200 OK Content-Type: application/vnd.collection+json Content-Length: 6385 ETag: "8788611043145" Last-Modified: Fri, 02 Nov 2018 13:30:03 GMT Server: Eve/0.7.2 Werkzeug/0.10.4 HTCondor/8.6.12 Python/3.6 Date: Fri, 22 Feb 2019 10:43:38 GMT {"collection": ... "items": [{ "data": [{ "name": "_etag", "prompt": "_etag", "value": "8788611043145" }, ... ] }] "template": { "data": [ { "name": "size", "prompt": "hard limit on the size of the space in blocks", "value": "" }, ... ] }} The etag is then included as an ``If-Match`` header in the PATCH request .. code-block:: sh $ curl -X PATCH https://mypixstorserver/spaces/4a40b050-0bae-45d8-a554-6198af95e09c -u $TOKEN: \ -H "Content-Type: application/vnd.collection+json" \ -H "If-Match: 8788611043145" \ -d '{"template": { "data": [{ "name": "size", "value": 1024}] }}' Again, if the PATCH request is successful, the updated item will be returned as the response, with status code ``202 (Accepted)``. If you omit the etag, or if the etag doesn't match the one on the server, the request will be rejected with status code ``412 (Precondition Failed)``. As with POST, if the PATCHed data fails validation, ``422 (Unprocessable Entity)`` will be returned Deleting an item ================= As with PATCH, you need to send the item etag along with a DELETE request. This ensures that operations will not be lost. .. code-block:: sh $ curl -X DELETE https://mypixstorserver/spaces/4a40b050-0bae-45d8-a554-6198af95e09c -u $TOKEN: -H "If-Match: 574575684686" If the request is successful, you will get back an empty response with status code ``204 (No Content)`` .. note:: Items can only be deleted individually. It is not possible to perform a bulk-delete against a collection endpoint Auth Roles ========== Your auth token will have associated with it a collection of auth roles. These auth roles determine whether you will have permission to perform certain operations on a given endpoint. You can check your auth roles using the auth server's ``/tokeninfo`` endpoint. Alternatively, you can perform an ``OPTIONS`` request against an endpoint and check the ``Allow`` header .. code-block:: console $ curl -X OPTIONS -i https://mypixstorserver/spaces/ -u $TOKEN: HTTP/1.1 200 OK Server: nginx/1.12.2 Date: Thu, 08 Nov 2018 12:26:44 GMT Content-Type: text/html; charset=utf-8 Content-Length: 0 Connection: keep-alive Allow: HEAD, GET, POST, OPTIONS In this case, we see that our token grants us permission to view (``GET``) and create (``POST``) spaces. If our token didn't grant us write access, to the ``/spaces`` endpoint we wouldn't see ``POST`` included. If you performed an OPTIONS request against a specific item, you would also see ``PATCH`` and ``DELETE`` if you have write permission. The allowed methods may vary for different endpoints - e.g. you could have permission to create spaces, but not profiles. The ``Allow`` header will also tell you if the resource or items supports a particular method **at all**, regardless of your auth roles. For example, ``datastores`` can't be created, updated, or deleted. So their ``Allow`` headers will **never** contain ``POST``, ``PATCH``, or ``DELETE``, even if you have an auth role which would otherwise allow you to perform such operations.