Using the edgeryders.eu APIs

This topic is a linked part of a larger work: “Discourse Admin Manual

Content

1. Overview

2. API access

3. Custom API endpoints

4. API client registry


1. Overview

1.1. API Overview

All relevant information in the edgeryders.eu database is accessible via APIs. We support the following APIs:

  • Public Discourse API. This API gives access to everything that can be viewed as non-logged-in visitor on edgeryders.eu. No API key or user account is required. For the API documentation, see docs.discourse.org.

  • Protected Discourse API. All Discourse content that can only be viewed with a user account requires an API key to access it. See below on how to get an API key for your user account. It will give you access to what your user has access to. For example, access to some protected categories, while a moderator user’s API key is required for example to get access to user e-mail addresses etc… For the API documentation, again see docs.discourse.org.

  • Protected custom API. This API is custom-made for edgeryders.eu and gives access to “secondary content” (the codes and annotations of Open Ethnographer), and to the data collected with our “ethical consent funnel” form. Access requires a user with access to this data, and a Discourse Admin API key for that user. For details, see section 2. API access.

1.2. Tips and tricks

API rate limiting. API access is throttled to avoid server overload. The limits are set in config/discourse.conf and are currently:

  • max_user_api_reqs_per_minute = 20 (default)
  • max_user_api_reqs_per_day = 2880 (default)
  • max_admin_api_reqs_per_key_per_minute = 300 (default was 60)

After adapting any of these limits, you have to restart the relevant Discourse process with, for example, sudo monit restart puma_discourse_production.

Filtering by category with the API. In Edgeryders, we often want to look at content in a specific category. Discourse supports two levels of categories: top-level and sub-level categories. When retrieving content in a category, the API will by default also return return all topics in all of its subcategories. If you want to exclude some subcategories (for example, you might want “everything in the category except what is in the Workspace subcategory”), you need to do as follows:

  1. In a browser, visit the subcategory you want to exclude, and click on any of its topics. For example, visit https://edgeryders.eu/c/ioh/workspace, then https://edgeryders.eu/t/10334.

  2. Add .json to the topic’s URL in your browser. You will see the API response for that topic. It helps to have a JSON-prettifying add-on in your browser.

  3. Locate the category_id key in the JSON, and take note of its value (an integer).

  4. Adapt your code accordingly. For example, if you are trying to count the tops in a certain category, except those in its workspace, you can do something like this:

    # 328 is the category ID of /ioh/workspace
    if topic['category–id'] != 328:
      number_of_topics += 1
    

As an alternative to excluding categories you don’t want, you can also add up topics from all sub-categories you want and then add those from the top-level categories you want excluding the topics of their sub-categories. This is possible by appending /none to the URL of a top-level category, for example https://edgeryders.eu/c/ioh/none.

1.3. Python library

We wrote a small Python module containing a collection of functions to fetch Edgeryders content,consent data and Open Ethnographer codes and annotations via API:

2. API access

For access to the edgeryders.eu APIs, you need the following:

  • Public Discourse API. You do not need an API key for the public Discourse API, see above.

  • Protected Discourse API. You need either a Discourse User API key (which you can generate yourself) or a Discourse Admin API key, and then can access with that API key everything that the associated user can access.

  • Protected Custom API: ethical consent data. You need a Discourse Admin API key of a Discourse moderator or admin user.

  • Protected Custom API: Open Ethnographer data. You need a Discourse Admin API key of any user who has access to Open Ethnographer.

If you do need a Discourse Admin API key, ask @alberto or @matthias (or another edgeryders.eu Discourse admin) to create one for you. You can not create it yourself, and you should save it somewhere because you can not look it up in your Discourse account later.

For Discourse admins, this is how you create Discourse Admin API keys:

  1. Go to “Admin → Users” and select the target user.
  2. Scroll down to find “Permissions” and under it “API key”.
  3. Click the Generate Key button.
  4. Tell the user their API key via a secure channel (for example, an encrypted Matrix chat).

3. Custom API endpoints

3.1. Ethnographic codes

The Open Ethnographer codes API endpoint is accessible at https://edgeryders.eu/annotator/codes.json.

A standard response looks like this:

{
    "id": 13,
    "description": null,
    "creator_id": 3323,
    "created_at": "2017-09-05T16:09:52.870Z",
    "updated_at": "2017-09-05T16:09:52.870Z",
    "ancestry": null,
    "names": [
      { "name": "accessible laboratories", "locale": "en" },
      { "name": "zugängliche Laboratorien", "locale": "de" }
    ]
}

Open Ethnographer supports code hierarchies. The ancestry field returns the parent code of the code at hand.

Codes can be filtered by creator like this:

https://edgeryders.eu/annotator/codes.json?creator_id=3323

You can request info about a single code with its ID like this:

https://edgeryders.eu/annotator/codes/13.json

3.2. Ethnographic annotations

The annotations endpoint is accessible at https://edgeryders.eu/annotator/annotations.json.

A standard response looks like this:

[
  {
    "id": 5579,
    "version": "v1.0",
    "text": null,
    "quote": "comfortable for the patient.",
    "uri": "/post/33751",
    "created_at": "2017-05-22T19:13:10.000Z",
    "updated_at": "2017-05-22T19:13:10.000Z",
    "code_id": 342,
    "post_id": 33751,
    "creator_id": 3323
  },
 ...
]

Fields have been named in such a way that their meaning is self-evident. The one exception is text: this is a comment field that ethnographer can use to explain their thinking behind the annotation.

The following GET parameters can be used to filter annotations in the response:

  • topic_id: Return annotations which belong to the given topic.
  • post_id: Return annotations which belong to the given post.
  • creator_id: Return annotations which were created by the given user.
  • discourse_tag: Return annotations which are tagged with the given Discourse tag. The tag’s name, rather than the tag’s ID, needs to be passed in. A list of available discourse tags can be found here: https://edgeryders.eu/tags
  • code_id: Return annotations which are tagged with the given Open Ethnographer code.

Filter parameters can be combined as needed. For example, to return all annotations that belong to a certain topic and were created by a specific user:

https://edgeryders.eu/annotator/annotations.json?topic_id=111&creator_id=222

The output is paginated. By default, one page contains at most 100 annotations. This can be changed with the per_page GET parameter:

https://edgeryders.eu/annotator/annotations.json?per_page=200

If there are more annotations available than the given per_page limit they can be accessed on subsequent pages by using the page GET parameter:

https://edgeryders.eu/annotator/annotations.json?page=2

If no further annotations are available, an empty array is returned for that page.

3.3. Ethical consent data

The Edgeryders platform has a feature called the ethical consent funnel. It is accessible by API as described below, and (after fixing #194) on the user admin pages as field edgeryders_consent.

When a user tries to post for the first time, the ethical consent funnel is served as a popup form: it asks users to answer some questions before they are able to post in certain categories. They can only proceed past the form when they have answered its questions correctly. When a user answers the questions correctly, the platform updates the value of a field called edgeryders_consent. We interpret this sequence of events as having given informed consent to participating in a research project with Edgeryders, and having understood the nature of their part in the exercise.

Question definitions. The wording of the consent funnel questions and answers is contained in consent.hbs.

Data access by API. The data of field edgeryders_consent is accessible by JSON API at https://edgeryders.eu/admin/consent.json in conjunction with a suitable API key, which you can supply as a api_key GET parameter. In addition, currently you have to supply a request header Accept: application/json as a workaround for #212. Also note that this API endpoint does not support pagination – it provides the consent data for all users in one response.

Basic values. The interpretation of the edgeryders_consent field values as seen by JSON API access is as follows:

  • "edgeryders_consent": "1": User has given consent. Includes valid consent given on the Drupal platform that was later imported (with the consent timestamp reflecting the import time).

  • "edgeryders_consent": null: User has not gone through the consent funnel yet. This is true for many of the earlier users of Edgeryders, as the consent funnel was only fully implemented in July 2017.

    (In the Discourse database, the value is not indeed NULL, but a logical equivalent: the record in table user_custom_field will be missing for this user and name = edgeryders_consent.)

Additional values. In the Fall 2017 we attempted to elicit consensus from 191 users that had contributed to OpenCare before 2017 (but never after). For this, we created two new values for the edgeryders_consent field:

  • "edgeryders_consent": "0": User was re-contacted after contributing to Edgeryders before July 2017, and has denied consent.

  • "edgeryders_consent": "no answer": User user was re-contacted after contributing to Edgeryders pre-July 2017, but failed to answer after repeated attempts.

  • "edgeryders_consent": "unreachable": User user was re-contacted after contributing to Edgeryders before July 2017, but the e-mail they used to create the Edgeryders account is no longer active.

4. API client registry

Our custom APIs frequently change due to improvements and refactorings. To avoid the complexity of legacy APIs or API endpoint versioning, our method of change management is this API client registry. When you register your client application here, you will be notified when an API endpoint it uses changes. Registration is optional, but you’ll have to track API changes here in the manual if you don’t register. To register, simply edit this wiki. API client means an instance of a software; one application could be run in several instances.

Has Likes