Planning the new edgeryders.eu platform based on Discourse

Workspaces Software (General)
,
1

Here’s a rough draft how I plan to replace the whole Drupal based implementation of edgeryders.eu within the tech budget limits of the World Bank project. Most of this does not belong into the Technical Proposal, but will be helpful for us as background reference when writing it. The following already includes @Nadia 's proposal of splitting into a content collection platform (“challenge and responses” as now) and a Matrix based realtime communication platform for all company and project team members.

The good thing is, we know how we want the platform to be: the functionality should be like the current Drupal platform: challenges and responses format, forum with posts and comments, @mentions, daily digest e-mails; and in addition we want a realtime collaboration software like Matrix. After all the years of experimentation (and Drupal was quite flexible for that), it’s time to simplify and get compact, proper software do the job. Now it’s 2017, and unlike 2012, there is a lot of modern and compact open source software around now.

So I propose to split the current Edgeryders platform into several parts, as follows:

  1. edgeryders.eu: A custom Ruby software with a single login, providing multiple pieces of custom functionality:
    1. Static pages. Some static pages about the company and its projects (including Reef). Only editable for admin users. Technically part of the custom Ruby software. Using a Rails engine for blogs, which allows full HTML. Using Bootstrap markup and layout, incl. a quick reference on the screen when editing a post.
    2. Forum ("Challenges and responses"): the community space where all the content collection and public-facing content discussion takes place. A custom, highly compact (!) forum-type software application that is highly optimized for low bandwidth / site loading speed, and access from developing regions. That includes very good foreign script support (Devanagari, Arabic etc.).
    3. Reef: A separate part of the custom Ruby application, with features for space management, event announcements and other custom tasks for The Reef. There could be other, similar pieces of software on that same domain for other spaces, like the one that will belong to the MENA youth platform.
  2. PayCoupons: People's PayCoupons username, offers, hourly rates and GTCs would be listed both on their Matrix profile and edgeryders.eu custom software profile, in a dedicated linked field. Also, the forums software would have PayCoupons links after every mention of a username, and ways to tip people for good / helpful posts, using small amounts of PayCoupons.
  3. Github repository: all software related tasks of the custom forums software are handled here.
  4. riot.edgeryders.eu: Riot web client for the Matrix protocol, see https://riot.im/ . It is one of multiple clients that people can use to collaborate via Matrix, so there is no need to include it on the main domain. On the main domain, there will simply be a site explaining what Matrix is, how to join, and linnking to several options including riot.edgeryders.eu. Used for all Edgeryders collaboration needs around projects. Everyone who wants to be part of the Edgeryders codebase would take part here. Includes all collaboration with team members, community managers etc.. No login integration with the Ruby software needed, as Matrix is an instant messaging protocol, so the idea is that everyone has their own username, independent of any particular website. The custom forum application would link to that username for instant messaging etc.. We would do no (!) custom changes to the Riot or Synapse codebases.
  5. synapse.edgeryders.eu: Matrix (http://matrix.org/) server installation, using the Synapse reference server ( https://github.com/matrix-org/synapse ). Not linked anywhere, as it is just the server, providing the "rooms", while people always connect via clients.
  6. edgesense.edgeryders.eu: Stand-alone Edgesense installation, using the data of the forum software. No login integration with the forum software, as it has only very few users and these would simply stay logged in permanently.
  7. ethnography.edgeryders.eu (optional / later): A stand-alone software for tagging the edgeryders.eu content. Can work on the same database, and its data may be used by the edgeryders.eu forum software installation for showing related content etc.. This software can be based on Annotator.js, with the extensions developed for it for the Open Ethnographer Drupal software.

Implementation proposals for the custom software application:

  • Essentially: it will be a custom application in Ruby, developed by my brother Daniel and administered and managed by me. Daniel has in total has 8 years experience in Ruby and is very fast at implementing features with that framework. He only does Ruby – not like me who does a bit of everything (and is correspondingly slow). I don't know another developer whom I'd trust to replicate ederyders.eu until the launch phase in October, but together with Daniel I can get it done. We have worked together extensively on PayCoupons, so we have a lot of routine working together …
  • I am sick and tired of Drupal-style overly complex and "super configurable" software where everything takes about 4x longer than planned. In the future we'll have super compact code instead. There will not be the current flexibility for admins to change things around by configuring them, but our desired feature set is quite well-known now. We can have Ruby courses though, so everyone interested will be able to make small changes to the website in code. Content changes are all still possible in the admin interface, of course.
  • The forum base software would be either Thredded (10,500 lines Ruby code only, MIT licence) or Discourse (full fledged forum, 140,000 lines Ruby code).
  • Boards of the forum will be called "Challenges", messages will be called "Stories" (as in "Add my story").
  • The editor should be ProseMirror (MIT licence). It outputs natively to Markdown, but allows rich text editing as well. When editing in source, there should be a live preview using a second editor widget, just as on Stack Exchange. It is basically as capable as our current editor, while allowing faster editing for markdown skilled users (esp. on mobile devices), and avoiding all the crazy security considerations around "Full HTML" in editors. More editors that can output in markdown format: see here.
  • Another custom component will export all (!) content into a public JSON file, for an external Edgesense installation to consume.
  • A simple text box with a hiearchical link list in Markdown will do to define the contents of the main menu.
  • All (!) existing public content of edgeryders.eu will be migrated to challenges and responses in the new custom forum application. That may happen only after the end of the OpenCare project, however.

Edit: The current project journalling and task functionality of Drupal will be lost. If you want a realtime message style collaboration solution like Matrix, that’s what you get :slight_smile: I proposed (task based) Taiga.io before but it got turned down – that’s fine, as for software issues I’ll simply use the Github issue tracker then. You’ll have to find your own ways to manage tasks, as there will not be structured support for it. For example, a simple Google Spreadsheet could do for team internal task management.

The existing journal and task contents would be migrated to Google Docs and Spreadsheets documents, as that is where all our other project knowledge resides. Wiki documents would be partially migrated to static pages on edgeryders.eu (in case of the website manual etc.), and otherwise also to Google Docs.

1 Like
2

!

Matthias, you are one of a kind. :slight_smile:

It looks convincing – better than convincing, actually – though I do not understand everything. Questions:

  • One signup will give us access to both the chat (Matrix) and the forum functionality. True of false?
  • We still need APIs for batch data analysis, the equivalent of Drupal JSON views. For example, I like to induce project-specific interaction networks using Tulip. You mention a custom component that outputs the whole thing as JSON, basically a dump of the whole database. I am not sure this will work – I can see timeout problems, for example. It simply shifts the API issue somewhere else – now you need APIs to query a monster JSON. Is it not better to bite the bullet and have APIs on our own DB? 
  • We will still have the equivalent of groups or projects. Maybe this will be "boards" in Thredded, not sure in Discourse? This is important both at the Ux level and at the data/project evaluation level. I have explained this point in some detail here. True or false?
  • Edgesense will work via APIs - Catalyst protocol. The existing code base is completely reused, we only need to"feed it" JSON files as extracted from the APIs. True or False?
  • We lose Open Ethnographer. We'll have to re-implement it, because it is an important part of our methodology. RECODE, is completely based on it (but it does have a money and time budget for software. OE needs plenty of work on user interface and utility, but it works quite well at the data level. What are your thoughts around that? 
  • Related to the last point: right now, OpenCare stuff feeds into a piece of software called GraphRyder (not displaying properly at the time of writing), one of the main artifacts of OpenCare. Again, this is not integrated with edgeryders.eu, but it does rely on APIs for both content (like Edgesense) and also OpenEthno annotations. This is something we'd like to keep ideally. It will be further developed in the course of OpenCare and (if we get it) RECODE. You do have a plan for all of this (Annotator.js), but I am not familiar enough with Open Ethnographer: is it like Edgesense, independent from edgeryders.eu installation? Something that you only need to feed an API call? 
3

Trying to answer …

Can be true, but I think it’s not necessarily recommendable to make the effort. Matrix is not a single platform, but a messaging protocol with multiple client software applications, for example for Android, iOS etc… We don’t force people to have a certain e-mail address or Skype address when signing up for Edgeryders. We could, of course (similar to how the early Facebook Messenger would automatically give you a XMPP / Jabber handle so you could chat with anyone using that open protocol).

I think it depends on what we want to do with Matrix. If it’s for company and team internal communications only (20-30 ppl), then account integration is not needed. Here, integration would rather mean having a field in your edgeryders.eu forum software profile that mentions your Matrix handle, and links to a chat with you on the riot.edgeryders.eu Javascript Matrix client. If Matrix is for connecting with community members as well, then of course we do integration. In that case, we can even think about using the Matrix Javascript SDK to integrate messaging right on the site, rather than using a Riot Matrix client on a subdomain. In that case, login integration is for free, but the interface may be more limited than with Riot.

The full JSON dump is an example. With the same ease, we can add a few more export options, including some that include parameters (“ederyders.eu/export/json?group=123&fullcontent=y”). 4-5 different, parameterized types of exports would feel reasonable, and should enable all work in external software. We just have to think in advance what export options make sense.

Can be true. Thredded’s message boards would however be the equivalent of “challenges”, and called that way. In general of course, we can introduce any concept we want to the datamodel, as it’s custom code in the future. In this case, I’d propose a lightweight solution as follows. We keep “Challenges” as the one and only ordering concept that a frontend user is exposed to (similar to how OpenIDEO does it, feels nice and simple). Plus, we would implement an equivalent of the current “Challenge Chains”, which logically groups together any set of challenges and their responses. They’d have a page that acts as the start page of a project like OpenCare.

Then, for data analysis teh challenge chain ID can be used as parameter in JSON exports etc… While for web analytics, we can sum up visit counts to multiple challenges using a custom Google Analytics view, or if that is not possible, set a custom parameter inside the tracking code to the challenge chain ID and base tracking on that. There are quite some sophisticated things inside Analytics that we have not yet tapped into. (Note: Currently in Drupal, Challenge Chains belong to groups, so with SQL subqueries it would be possible to get all challenge responses that belong to a group transitively via a challenge chain.)

That’s right.

Integrating OE closely with Drupal was done b/c we expected ordinary users to try their hand at ethnographic coding. That “massively parallel” coding did not materialize, so for re-implementing it I recommend making it a sweet, small stand-alone Ruby application on a subdomain. Like oe.edgeryders.eu. It would be connected to the main site with an API that feeds it updates with new pieces of content (posts and comments). Each piece of content will only be sent once (ignoring future edits), but a few days after the original post date ass the chance of future edits is then small enough to ignore it. We can re-use Annotator.js, our changes to the Annotator.js UI and re-implement the whole data model of how annotations are saved externally. Since we also do not have to deal with Drupal, implementation would take 30% the effort of the original version.

Until that happens, we may choose to run Drupal in parallel just for researchers to use Open Ethnographer, but I doubt that it’s worth it.

Open Ethnographer comes in two parts: data storage of codes and annotations is tightly integrated with Drupal, everything Annotator.js related (incl. the AJAX interface for tagging) is modular enough to be re-used easily as it’s all JavaScript. Why you ask that, do you want an OE-style annotation interface implemented inside GraphRyder?

I suggest we make it a principle that all our analysis tools are little stand-alone software pieces, reside on one edgeryders.eu subdomain each, and use only public APIs to get their data (no authentication needed, anonymization done by the API server software as needed). In that case, GraphRyder would reside at gr.edgeryders.eu, pull content from edgeryders.eu, and annotations from oe.edgeryders.eu.

4

Ok…

Ok. So APIs, with API documentation. Good.

It could work. The challenge chain thing is more useful to humans: for data purposes, what matters is that the database can easily parse the content (“give me the content in boards 36, 214 and 21”). For web analytics, what matters is that the different boards have different URLs (https://edgeryders.eu/my-challenge/hello-world, and not https://edgeryders.eu/hello-world, which is how challenge responses work now. As it is, I cannot answer the question “how many visits does the OpenCare content gets?” This has been awkward during the project. For humans, we’ll see. I admit I am not fully sure of how it should work.

Works for me.

No way! I would like the new edgeryders.eu to to what the old one does, i.e. serve data to GraphRyders via API. Documentation here: opencare-data-documentation/apiary.apib at master · opencarecc/opencare-data-documentation · GitHub

This is a great idea. Except: I am not sure about anonymisation. It is really useful to click on Edgesense nodes and find out who they are. In practice, this has not been a problem: we deploy analytics tools that are not threatening, so they can be not-anonymous. BUT, if the APIs are also public, anyone could make a non-anonymised database dump. Are we good with that?

5

Pseudonymous by default?

I think so, if we push people to not use real names on signup, except if they are very sure that they are ok with that and understand the consequences. In the more distant future, community managers and everyone else can have local de-anonymization. This works by annotating any user accounts with their real names to keep track of who’s who … but real names would only be stored in offline browser data, never sent to the server, and never appear in backups. It can be backed up as a lcoally saved JSON file, and re-imported as needed. When working in politically more difficult environments now, we’ll need something like that.

6

Unrealistic

Here we depart, Matt.

On the one side, we don’t care about real names. We know people on the platform by their handles, so we want to know if that highly connected node corresponds to trythis, not what trythis’s real name is. On the other hand, experience tells us people never bother with security with any degree of reliability. So, there is no way we can be very sure that they are ok with using real names and understand the consequences. We’ll have to make a hard call.

7

real names or just an ID

It has been my experience that just an ID works fine as long as the person/ID is consistent.  When it changes all the time then you never really know who you are talking to and it becomes very easy to play head games.

8

From the user perspective - real names

As I mentioned before this is an issue that needs careful consideration. When it comes to MENA and what I understand about the politics, laws and practices of the four countries involved, we deal with people who are likely to get a lot more government intrusion into what they say and do than we are used to. And even though our server and system would be in an EU country and not strictly subject to demans of a particular MENA government (at least I think), it still can be problematic for the users in a given country at a given time.

So whereas real names as a default helps people collaborate quicker in some ways, especially when matched up with private conversation spaces, we have to decide if we want to host such spaces knowing the volatile political climate. Maybe we want consistent user names and chats, forums, documents, etc. to all be public as the default.

9

From the user perspective - challenges, groups, or…

What you call a “challenge” is really a word that signifies an area of context. You could say “domain” or “group” or something else entirely. But I think what you are picturing here is not unlike our current Groups page as a default, but that any one of those Challenges would lead directly into a page that is either Thredded or Discourse. Am I correct?

If so, we would still need a way for a user to see documentation, blogs, wikis and other prepared material. How would that work?

10

Time for a chat about ux and information architecture?

During Matt’s time in Brussels we discussed the next iteration of the edgeryders platform.

What benefits are there to be drawn from chatting on our platform as opposed to say Facebook or a random forum/thread?

Matthias made a this short “sales” presentation as a result of the Power Pitch weekend (longer discussion here).

The concept we have been developing for what participants on the platform “are building” is a kind of p2p resource for thriving in the robot/post-job economy.

  1. We produce the challenges as calls to action: Users produce content in response to challenges. We have a map of things needed for a good life. Each new project/call to action aims to cover that part of our map

Connecting the dots

  1. We present the (editorialised) community contributed content as “recipes”/ solutions to needs, stories of how others have tackled the issues/secured what they need, and opportunities to build/learn what you need to be able to do it from your peers. Think a mix between make magazine and whole earth catalogue with discussion forum and high quality editorial content (Stories) + events (workshops etc)

Assets_OpenVillage_Edgeryders_Map.jpg2269×1375 1.66 MB

Assets_OpenVillage_Edgeryders.jpg1492×992 1.46 MB

11

Matrix integration

We’ll have to discuss how we want Matrix integrated (when I am in Brussels).

For exmaple: since Matrix provides SDKs, it is possible to have basic chat in multiple rooms integrated below a “Talk & Collaborate” menu item, which would be the other big one next to “Challenges”. It would also be possible to automatically have public chatlogs of what is said there, providing a searchable “everything is on the platform” experience. Not sure how recommendable that is though, as chatlogs are a notorious mess and hard to read.

2 Likes
12

Two arguments in favour of Matrix integration

[still does not mean I am decided one way or the other]

  1. In the MENA YP project, integration looks more like a platform in the sense that a client understands.
  2. In general, anyone that chats with us gains an edgeryders account, which lowers the barrier to her using the main platform (and generating ethno data, etc.)
13

Looks like we should create an integrated own Matrix chat client

Means, people sign up for edgeryders.eu and with that account and right on edgeryders.eu, they would automatically be able to use Matrix for instant messaging. This should be relatively simple to implement, as there are Matrix SDKs for Javascript and we really only need to implement a client. The Matrix server (called Synapse) would live on matrix.edgeryders.eu or similar, and no end user would ever need to know or visit that URL.

We would not get the full functionality of the Riot Matrix client (which is their most advanced one). In particular, it may be too difficult to implement WebRTC (voice and video calls). Even so, anyone intended to use voice or video calls can use the (externally hosted) Riot client, an Android or iOS app for Matrix, or another client.

My main reason for this turnaround is however that it avoids the need for login integration between the Riot Matrix client on (say) riot.edgeryders.eu and edgeryders.eu. Login integration is a notorious mess, as it either involved a third-party single sign-on service that breaks often (Persona, OpenID, …) or messing around with hashing methods, API calls or multiple database connections, all of which reduces the reusability of our code for others. So, no to login integrations, as a principle. Every normal user will only need a single edgeryders.eu account to participate in all of Edgeryders. Every company or project team member will need multiple accounts (Edgesense, Open Ethnographer, OpenProject, whatnot) but all auxiliary accounts would be set to “never expire the login”, so would not cause inconvenience after the initial sign-up.

14

From the user perspective - Matrix

I think we all agree that we want the closest and most seamless integration between asynchronous and realtime activity on the platform.

And in order to allow a user the quickest path to power use, we want seamless as possible integration - and home page/sidebar/nav access - to a number of other elements as well.

There are elements of the online experience that have not fundamentally changed since the 90s that continue to be core engines of interactivity, illumination, sharing, relationship building - all of it.

We all know this, but just to be clear, those core elements include:

all kinds of prepared material (written, edited, charted, listed, image, audio, etc) that formats well

aschronous conversation both public and private that can easily reference prepared material and/or attach directly to it (or not)

knowing who is a member

knowing who is online at the same time

being able to find out who they are and how to contact them (if they volunteer it)

quickly attaining a mental model of the “terrain” and finding things by browsing

having the system remember what you have and have not seen (and do it better than just html color change)

searching for content on the site

This current platform does some but not all of those things and some of them it does better than others.  Actually, I know of no system that does all of them really well in a way that is simple to discern if you are new and allows you to self-graduate to power use as you see fit.

But I think in reading Matt’s plan for the new plaform, all of these things are in play.  I hope they are because I think we need them all.

And we need them to be so easy to understand and use that being on this site is more enjoable and rewarding than being on other sites, unless those sites are used in service of this site.  In order for the MENA project, and by extension pretty much anything else we do in the future, to spread out to all who would benefit from participating, it has to be so.

I think I am stating something obvious, but the quick, seamless movement between all of those tools listed above, provides energy, amplification, and motivation for people staying and using your platform.

Having Matrix integrate right on the platform home or menu or nav is a crucial component of the toolkit.

You said, “It would also be possible to automatically have public chatlogs of what is said there, providing a searchable ‘everything is on the platform’ experience.”

Does this mean there would be no private conversations?  I can see that private would encourage some people to get to know each other better and see ablout collaborating, but I wonder if some of these MENA governments would want to make us couch up such records.  If they were all out in the open they could view them like anyone else.  At any rate, I think this point has big implications that we have to consider.

What about seeing who is online as you in real time?  Does Riot do that in the form you think we would use?

1 Like
15

And one more thing

Threaded comments only, please! :slight_smile:

16

Where do the images go?

So we have a godawful mess where all images posted on the platform go off into some black hole never to be seen again :slight_smile:

Also, we have gazoodles of custom made images and design files all over the place, including visualisations flickr accounts. Over the years this adds up to a lot of assets that we then have a hard time tracking down.

Is there an alternative to flickr and e.g. Vimeo we can self host for our own materials?

17

Image mgm proposal

I think this problem has two parts:

  1. User-contributed images in challenge responses etc., which are best managed in black holes like now, only available within the context of the original post they were uploaded to. For the (rather rare) case where anyone else wants to re-publish such an image under Creative Commons terms, they can save the image first from the web page it appears on, and re-upload it to a new post. No need for special software to support that.

  2. Communications materials created by the team for the various projects. This is important to keep around orderly for re-use, but not just as images, also in the source format for making edits and derivatives (.ai, .xcf, .svg or whatever people use). I propose to keep this within Google Drive, sharing project (sub-)folders as needed with more team people and making it a policy that everyone uploads the original artwork there. We have all our project files there already, so this would be the consequential solution. Even videos can go there, in their master file format. Migrating our file repo to a self-hosted equivalent like OwnCloud is possible, but I think only meaningful once a proper open source replacement for Google Docs and Spreadsheets becomes available as well. Until then, we can migrate away from the rest of Google stuff already, of course (esp. from GMail).

18

How to manage events?

So far our attempts to diy event sign up, collaborative program building etc has not been smooth. My best experience with this as far as software goes was reboot in Denmark .any thoughts?

19

I don’t know yet :smiley: Here are our options for event mgm.

What has become clear is that cobbling together some Drupal modules for this makes for a confusing UX not to be repeated. (The main reason for this is that Drupal is made mostly for publishing, it’s hard to do interaction with it.) The new basis, which is Ruby on Rails, is much better for developing interactive web applications, so in principle we can have a custom one at some point. However, we need to know exactly how it should be, and creating it will not be in the budget for this initial new platform (it won’t add more than 4000-5000 EUR either, however).

For now, @Nadia decided to stick with Eventbrite for ticket sales, and Matrix instant messaging for collaboration around the event. The main new edgeryders.eu platform will have static pages where admins can post anything about the upcoming event and tell people how to get their tickets on Eventbrite and collaborate on Matrix. There will not be publicly editable wikis, though.

Our events are quite complex projects, and I think any old-fashioned business would try a project management software here. That’s probably not for us though, as this kind of software will feel formal, not fun. However, I have to install OpenProject on our server today anyway as the tool of choice for making our MENA YP Gantt charts. You may then have a look if you like it as backend solution for the core team to keep track of the event preparations. It’s basically the best open source project management suite available, and also made in Ruby. So if people really really like it, we can do some integrations with our new Ruby-based platform. Such as a public task list where people can take over tasks for event preparation. But please, that would be for later (a year from now at least).

1 Like
20

Hmm there are two aspects to this: before and during

Before: Discourse + Matrix + Eventbrite + Social Media

During: Documentation is king. These guys do a solid job of making event documentation easy and fun http://smartlab.ftf.ludiconline.uk/login/internal (login:elimam, psswrd:ftf)