IT development plan for the H2020 projects


#1

Content

1. Introduction

2. Software architecture and budgets

3. Team plan

4. Time plan


1. Introduction

This will be my only “big picture” planning document for software development and maintenance in Edgeryders for the next three years (2019-2021). Basically all software we’ll make will be used in at least two projects (which is good), so it makes little sense to have separate planning documents for each. It’s all in here: for POPREBEL, NGI Forward, Trust in Play, Tiramisu, Biofabforum (relaunch as federated platform), Blivande and any other Discourse based platform we’ll create in that period.

Downstream from this document, there will be software design documents for specific software tasks in Dynalist and Github issues. The associated Dynalist tasks also contain the intermediate, internal deadlines / milestones, if any.

For the list of requirements implemented by this plan, see “Tech dev specifications for POPREBEL and NGI Forward”.

2. Software architecture and budgets

Below is a list of the major features and software changes we’ll do during these projects, separated per software tool that we have or will have. Some of that was inspired by or decided in the discussion below this post, and had been added here lateron.

Budget for project-specific IT tasks. Excluding the 20% margin of Edgeryders OÜ. In the order of project execution.

  • Trust in Play: 2,800 EUR IT budget

  • Timisoara: 3,040 EUR (4,800 EUR contract value minus 20% Edgeryders margin minus 400 EUR each for project management by Noemi and Natalia)

  • POPREBEL: 25,200 EUR (24,000 EUR according to revised version)

    (flexible use for all IT tasks, fully flexible distribution over the project runtime)

  • NGI Forward: 25,200 EUR (21,600 EUR according to revised version)

    (flexible use for all IT tasks, fully flexible distribution over the project runtime)

  • Biofabforum: 2,000 - 3,000 EUR (may include non-IT tasks for improvement)

    (currently excluded, as the project did not move forward as of 2019-05)

  • Total: 51,440 EUR

Budget for indirect costs. Some of the tasks in the list below (esp. the reporting and project management tools) will fall under “indirect costs” and cannot be reimbursed directly in the POPREBEL and NGI Forward projects. This means they have to be funded from Edgeryders OÜ’s “profits”, equivalent to the “25% of direct costs” that are allocated for indirect costs in Horizon 2020 projects.

Total. Taking the project IT budgets and the necessary budget for indirect costs together, the total can be around 55,000 EUR.

2.1. Discourse

  • :ballot_box_with_check: Login integration for federated platforms. Our new solution for providing whitelabel Discourse platforms will look very similar to the Stack Exchange network of sites: one installation, different databases, but synced user accounts and login sharing between them. All of this is already possible using the Discourse multisite feature and existing SSO provider / client plugins, so we’ll “only” have to deploy all this properly. There will be no content syncing / embedding between sites, such as via special categories on the frontpage etc., as that creates problems with syncing when running out of category levels.

  • :ballot_box_with_check: Site switch menu. Sites in our future network of federated Discourse sites will be coupled via a site-switching menu in the top right. This may be placed to the side of the hamburger and user menu, or in a menu bar above that, whatever is better for UX and from a tech perspective. It will simply list the federated sites.

  • :ballot_box_with_check: Discourse-to-Discourse migration script. To archive and safeguard the content of federated Discourse platforms after their decommissioning, we’d have a script that imports all that content into edgeryders.eu. Due to the single-sign-on feature, all users of the original platform will be able to log in and contribute under their original usernames, still. The import script would also have to update links in post content, and forward links from old URLs to new ones (such as whitelabel.net/t/123edgeryders.eu/t/19345). The latter would be managed via automated creation of Discourse permalinks, as we did successfully for our Drupal-to-Discourse migration. With such a script, our whitelabel offering would be complete, covering the full life-cycle of a platform.

  • Discourse façade website toolkit. A re-usable framework for a single-page or few-page website that “looks pretty”, presents a project that is active on a Discourse platform and displays static textual content, featured content and information / statistics about the Discourse content, all taken from that Discourse platform, and, where relevant, linking to it.

    Note that taking the site’s static content from Discourse means that Discourse users with the necessary permissions can edit the façade website right away, without needing another account etc… For an example of this mechanism, see the topic about the Blivande website. Ideally, that static content would be accessed from an access-protected Discourse category using a suitable Discourse API key. To know which content is edited in which Discourse topic, there should be a “secret” mechanism that will display overlay links for editable areas after appending a certain “secret” GET parameter to the site.

    For reasons of security and server load, no server-side scripting should be involved in page rendering. A client-side web application could query the Discourse API using JavaScript, but that puts unnecessary load on Discourse. So the best alternative seems to be to use a static page generator and let that run regularly (1-2 times per hour?) on the server to re-generate the static façade websites. If that page generator would use Handlebars as its templating engine (the same used inside Discourse) it would be perfect to build a small companion software for Discourse that can be released as an open source application.

  • Improved signup and login processes. Due to the “Edgeryders Communities” mechanism, the signup and login processes are not as smooth as they could be. We’ll improve that as follows:

  • Table of Contents plugin. A small plugin that can generate a table of contents (in the format following our conventions) when pressing a button in the post editor. Not essential, but we need it quite a bit and it would save us time when writing documentation, and would enable users to contribute more structured, better navigable content. Seems a good thing to have for NGI Forward, as discussions will be quite technical at times.

  • Cross-platform notifications in the site-switch menu. (optional) Implementing this depends on having another paying client for a federated Discourse platform after TeaM2021. Technically, it will be an improved version of the existing site-switch menu (see above), now also showing the notifications from the federated sites. A number bubble for the menu entry of each site would sum up the normal (blue) and message (green) notifications that the user has on that site, and perhaps there would be another number bubble (or a white square this this time?) with the number the user would see in the “Unread” section of that platform. The dropdown menu label itself – perhaps “Communities” – would sum up the notifications on the menu entries it contains. The current platform would be listed in the menu but without notifications, to not confuse people.

    Later, in an improved version, we could allow the menu entries to fold out, revealing a list of the actual notifications as shown on the federated Discourse sites, with direct links to posts and messages there. That may or may not be good UX, though. Probably it would be good because it’s also done this way on Stack Exchange.

    Finally, there should be a setting in the user profile so the user can choose if links from these menus to federated sites are opened in the same or a new tab.

  • Content cross-promotion plugin for federated platforms. (optional) A Discourse plugin that cross-promotes posts between multiple federated Discourse sites. Comparable in function to the “Hot Network Questions” feature in the Stack Exchange network sites sidebar. Implementation proposals:

    • Create a new entry “Discover” that is displayed in the “Categories | Unread | New | Latest | Bookmarks” menu and shows interesting content from across the Edgeryders Communities sites.
    • Show topic proposals from across the Edgeryders Communities sites at the bottom of the page, below a topic.
  • Federated Discourse platform software to download and install. (optional) Since this is a “nice” software product that others may want to re-use and even contribute to, we will invest some effort into proper packaging. Ideally as a Discourse plugin that would simply have to be installed in all federated sites to provide the notification integration (while login integration is provided by the existing SSO plugins).

  • :hourglass_flowing_sand: Improved cross-platform user interaction. (later) Currently, @mentions work only between users of the same Edgeryders Communities platform, sometimes causing issues for community builders of confusion to users. We can improve on that in several ways, and we still have to decide which ones we want. Examples:

    • True cross-platform @mentions. They are not easy to implement though. And the implementation should keep it clear which users are part of the own platform and which are part of external platforms. For example, by mentioning the platform URL(s) behind username proposals that are not part of the current platform.
    • Enabling the user directory and messaging on communities.edgeryders.eu, at least for staff members. That’s a simple fix and could already help/
    • Noting on user profiles on which Edgeryders Communities platforms a user has an active account. This information would be collected and distributed by a script looking into the databases of all the Discourse platforms.
  • :x: System for automated translation of post content. (discarded) This would provide buttons below each post that allow to view a machine translation of that post into the user’s own language. Machine translations would be cached to make the system faster and to lower the cost of API usage.

    However, in the team call 2019-02-04 it was decided that we do not want this feature for the POPREBEL or NGI Forward platforms. For POPREBEL it is not needed as the plan is to provide one sub-forum per language and focus community management on one community per language. For NGI Forward, the concern is rather to better not use Google, Microsoft etc. technology, which includes to not use their machine translation technology.

Budget: [TODO]

2.2. Open Ethnographer

  • :ballot_box_with_check: Code internationalization. Called “multilingual coding”, but that’s rather a misnomer; the idea is not to allow coding foreign language text, which is already possible, but to allow ethnographers to translate codes, esp. for re-using those of other researchers with names in ones own language.

    There were two proposals to implement this, and @matthias decided that the implementation should be to translate code names. Rather than creating new codes in a different language and using them as child code of some parent code (as @alberto proposes). Because the latter is already possible with ones own codes anyway, and we’d like to avoid implementing multiple parallel hierarchies for codes as that is a messy data structure.

  • Multimedia coding. Relates to image, video and audio annotations of content embedded in Discourse posts. Requirements to be decided in detail together with @amelia and @mette. The current, rough implementation proposal by @matthias:

    For images, we would integrate an existing image annotation solution that uses Annotator.js 1.2. Ethnographers would be able to select any rectangular area of an image with the mouse, add a code to it, and repeat as often as needed for an image.

    For video and audio, we would keep the usual Discourse system in place where the actual data is stored on an external platform and embedded in Discourse via Onebox embeds. Currently, people can use YouTube or Vimeo for video and Soundcloud for audio. This is how posting video and audio content works on a Discourse platform, and changing that user experience is not a good idea. (Also, while there is the discourse-brightcove plugin which can directly upload from Discourse to Brightcove, the pricing of Brightcover is just completely over the top for the low-throughput needs we have here.)

    For annotating videos, we will use the Open Video Annotation software, which is based on Annotator.js 1.2 just as Open Ethnographer right now. Its licence is GPL, just as that of Discourse and Open Ethnographer, so there are no issues with distributing both in the same repo. Annotations for videos created that way would have a start and stop time, but not additionally an associated part of the video area. This way, they work much like subtitles.

    Using Open Video Annotation requires us to have the video files accessible on our server, at least in the coding view. Because for example the YouTube Terms and Conditions forbid to show their videos in third-party player software directly (see). This is somewhat enabled by Discourse already because uploading a .mp4 file will show it in a HTML5 <video> element, embedding it as a video to be played with the browser’s integrated player (example). So the only thing to do is to load the video-js player (required by Open Video Annotation) in the Open Ethnographer coding view, in such a way that it takes over rendering the <video> elements from the browser’s internal player.

    Now there is the problem to distinguish between videos created by the post author (that should be coded) and embedded third-party videos (that should be ignored). We don’t want irrelevant videos and sound file to eat up space on the server. Software can’t recognize which is which, so a manual action is needed. In addition, Discourse is not a video hosting platform so cannot deal with transcoding video and audio obtained from third-party sites. So the following process is proposed:

    1. When ethnographic coding starts, the ethnographer or tech support create local versions of all embedded video and audio files to be coded. This would be a staff-only action by marking .mp4 and .mp3 as staff-only extensions (and giving staff users higher upload file size limits). Audio files would be uploaded as .mp4 videos showing a still image, thus allowing to be coded with Open Video Annotation as well. These videos are embedded in a [details="Locally hosted version"] … [/details] element below the original embedded video or audio element, by editing the post. Manual conversion and uploading also allows to minimize the file size, by choosing resolution and compression according to the type of content. If this step should be automated later, it can be performed by a bot user when triggered by a staff user performing a custom action in the admin menu of a post (“convert embedded media to local”).

      As an alternative that does not need manual intervention, users can upload their .mp4 video files to a file hosting service like Imgur that allows direct access to these files. Posting links to such files as Onebox links in Discourse will also embed the browser’s default video player (example) resp. in Open Ethnographer use the video-js player as done for all <video> elements. If we want to choose this, we can give instructions to users in the upload / embed dialog of the Discourse editor, by configuring the UI strings used there. Not sure if this solution is better, as users would have to create yet another account and they would need to know a bit more about video tech for useful results. Staff users could always replace too large embedded video files with a reference to a smaller version, though.

      Simpler and perhaps even better, we could allow users to upload video files directly to our Discourse installation, up to maybe 300-500 MiB. There is currently no way to make this limit depend on a user’s trust level (we’d have to create a plugin for that), but in our case the probability of abuse is low enough. Again, admins could replace too-large files with smaller versions, and by enabling “clean up uploads”, the too-large versions would be automatically deleted.

    2. Ethnographers will then see the locally hosted versions of audio and video files in the Open Ethnographer coding view inside the video-js player with Open Video Annotation enabled, and code them there.

    3. Archiving the Open Ethnographer content will simply include the local video and video files as well.

    All other features like annotating audio in audio files and uploading audio directly from the Discourse editor would be something for later versions.

  • UI rework / improvements. Both according to input from ethnographers and according to own judgment by the developer / development team. Especially, this includes to deal with the open issues of Open Ethnographer.

Budget: [TODO]

2.3. Graphryder

Among others, this implements the “interactive dashboard” (version 1 and 2) as called for in the POPREBEL and NGI Forward tech specifications. It also includes features that let us use Graphryder as a community management tool, replacing Edgesense.

  • Installation process documentation and small fixes. Currently undertaken by @hugi, the task being “setting up Graphryder for OpenVillage data, and documenting the process to be reused for further corpora”.

    Budget: Initially 1600 EUR (40 h × 40 EUR/h), paid from the H2020 project budgets; since that did not suffice for the work, we assigned “some more”, if necessary paid from Edgeryders Core budget.

  • Multisite ability. Equivalent to how we manage the multi-site setup in Discourse and Open Ethnographer: one installed codebase, multiple configurations / databases. Each client for whom we provide a “whitelabel platform” should be able to analyze their own data – and just their own – in their own Graphryder setup. The fact that it shares the same installed codebase is not visible to them, but helps us to have a cleaner / lower maintenance server setup.

  • Data selection (incl. by category, code, tag). A way to select the content (as a subset of the associated Discourse platform’s content) that Graphryder will work on. Needed, among other things, for taking over the current use cases of Edgesense (i.e., looking at the whole platform).

    Proposal, to be discussed in detail: a few form elements at the top of the main window that lets the user filter content to include by tag, category and ethnographic codes (again selected by their hierarchy and / or author and / or perhaps their code language). And perhaps by the time span of post creation. Then a semantic social network will be visualized for that. How exactly codes are selected by language depends on how multi-language codes will be implemented in Open Ethnographer, which is not yet decided.

  • Improved data downloads. Better support for downloading the graph and treating it with tools outside Graphryder, esp. in Tulip for one-off analyses. To be implemented as download option for “what is visualized on the screen” in .tlp format or some other Tulip-compatible format. When not choosing any data filters (see above), this download would be equivalent to the full / main graph.

  • Evaluating post relation by quotation. (perhaps) Given the way Discourse works, creating a post as a reply is not the only way to interact with other posts. Users can also quote other posts. In fact, Discourse encourages them to quote multiple others in one new post rather than posting multiple replies to different previous posts. In turn, Graphryder should be extended to also create connections between users based on “who quoted whom”.

  • API extensions. (perhaps)

  • Installable packages. (perhaps) Another improvement for better installability. Undecided yet – it would help others use the software but is not essential, so we have to wait and see if we have resources left for this.

  • [more, to be decided]

Budget: [TODO]

2.4. Dynalist integrations

  • Google calendar integration. Mostly finished, but we’re still waiting for Dynalist staff to fix one bug for us.

  • Notifications. Mostly finished, see @anu’s dynalist_companion software. But this can still be polished and improved. Our issue tracker on Github contains all relevant tasks for that.

  • Gantt chart integration. (useful but optional) Let Dynalist Companion show Gantt charts for project management and personal task management. Such charts would help greatly to keep an overview of ones personal tasks in Dynalist, which are usually scattered among multiple projects and due to this, not ordered by date either. Elements in the Gantt chart would then link back to Dynalist.

    The charts would be generated live from Dynalist documents according to the node hierarchy and constraints added to nodes via some form of custom syntax – as an inspiration, see the Gantt chart syntax of mermaid.

    Then, we need a library that can generate Gantt charts as HTML and not just as static images (as that would not allow folding / unfolding, links to Dynalist and other interactions). So far, we know of the following open source software components that can do this: (1) mermaid as mentioned above; (2) dhtmlxGantt, source code being here; and (3) jQueryGantt, the component used in Twproject.

  • :x: Freemind integration. (discarded until demanded explicitly) This is an idea developed for Anique, who likes to have a more visual way interact with a list management application like Dynalist. However for now, it seems not urgent enough and we want demand for this from others before going forward with this plan.

    Technically, we would let Dynalist Companion provide a remote Freemind server and a Dynalist list editing client. It would dynamically translate between the two so that in effect one can view and edit Dynalist documents live with Freemind. Edits would ideally be posted to the Dynalist server when leaving node edit mode in Freemind, if technically possible. This would minimize conflicts happening due to concurrent edits, and if this is rare enough it can be managed with by letting the last edit win (as is current practice for concurrent edits by two Dynalist users). Not sure if this whole proposal can be technically implemented as a clean solution, though.

Budget: [TODO]

2.5. Project management and reporting

  • Financial reporting utility. This is a Python based command line utility as proposed in “Financial reporting for a Horizon 2020 project”.

  • Kimai v2 extensions. (optional, long-term) This would take the Kimai v2 time-tracking software for teams and extend it with an approval process and an auditable log of changes, as discussed in the thread at “Reporting for a H2020 project”. Given our current process there is no strict need for this, but it would make our time tracking less overhead and more suitable for audits.

  • tom / tom-ui export to Kimai v2. (optional, long-term) To make time tracking even more comfortable in our company, we would extend an open source cross-platform time tracker such as tom-ui to directly exports its tracked time to Kimai v2, the platform that would be used to aggregate, store, and approve tracked time. For more details, see the discussion at “Reporting for a H2020 project”.

  • :x: Timetracker integration with FreeAgent. (discarded) This was a speculative idea and later replaced by a tool-agnostic process instead of requiring everyone to use FreeAgent for time tracking (see: “Reporting for a H2020 project”). And for a longer-term, comfortable and technologically proper solution, it was replaced with the “Kimai v2 extensions” proposal above.

    The idea here was that if users experience time entry into FreeAgent as too cumbersome, we can automate that away with a time tracker application that records worktime right into FreeAgent, either directly or synced from local storage.

    From a short analysis, the most promising technical options are as follows. For OS X, there is FreeAgent Widget (open source) and Slips (commercial), both listed in the FreeAgent extensions directory. For other operating systems there is nothing yet. There are integration platforms like Zapier, but they only sync contacts and tasks and not tracked time (sources: 1, 2, 3, 4). Also, they only connect expensive commercial, web-based time tracking services to FreeAgent – which makes little sense, as FreeAgent itself comes with a (simple) web-based time tracker inside now. What we want is a desktop widget. So we’d have to extend a small open source time tracker with the desired functionality ourselves. Good candidates are porting the open source FreeAgent Widget, or perhaps watson with crick or arbtt. The latter are both cross-platform, written in Python, but unfortunately rather geeky command-line tools. So we’d also have to add a GUI widget.

Budget: [TODO]

2.6. Server admin, support system, development tools

  • Staging website. We have a Discourse staging website, but will have to complete that system with a script that can take over data from the live site and a way to have staging websites for each of the federated Discourse platforms (also, to be able to test the login integration between Discourse sites).

  • E-mail server migration. As part of the security optimization / hardening, Edgeryders’ e-mail server will be migrated to run on the same system as the Discourse platforms. The old e-mail server will be switched off.

  • Better backup system. Daily pull-based backups to two online locations with server hosting inside the European Union. Fail-safe alters in case a backup fails.

  • Server setup improvements. Some of these will have to be paid from Edgeryders money as they are “indirect costs”. The detailed list of tasks for this will be developed directly in Dynalist.

  • Support process. (perhaps) A category in Discourse will be sufficient, but we should probably make it more known and document better how to get support.

Budget: [TODO]

3. Team plan

In alphabetic order:

  • @anu: junior programmer for Python based project management tools. This means Dynalist Companion and (if Anu has time for it) also the financial reporting utility.

  • @daniel: All Ruby based programming. Means Discourse Core and Open Ethnographer.

  • @hugi: Software designer / project manager for the extensions we intend to do in Graphryder for the POPREBEL and NGI Forward projects. This is not that much worktime, but requires creative work / vision. It would include doing the software design (incl. a rough list of features), while others would do the actual programming and testing.

  • @luca_mearelli: When we need reinforcement, an additional developer for Graphryder (Python) and also, if needed, Discourse and Open Ethnographer (Ruby, JavaScript). He told us already that he’d be available.

  • @matthias: All project management of the development work. All final decisions. Server and software administration work. Python based programming for Graphryder and (partially) the financial reporting utility.

  • @owen: HTML and JavaScript web developer for the Discourse façade website toolkit.

4. Time plan

  • all-time tasks (2019-01-01 to 2021-12-31):

    • technical support and bugfixing for the end-user platforms

    • technical support and bugfixing for ethnographers for using Open Ethnographer and Graphryder

    • backups

    • server security updates

  • :ballot_box_with_check: 2018-12-03: Trust in Play platform ready. It should be the first federated Discourse platform: sharing an installed codebase with edgeryders.eu and using edgeryders.eu as a SSO provider for login sharing. The site switch menu and cross-platform notification features will still be absent and follow later.

  • :ballot_box_with_check: 2019-01-10: TeaM2021 platform ready. This would have the same feature set as the Trust in Play site (federated Discourse platform with login integration) plus the site switch menu. The cross-platform notification feature will follow if and when it can be implemented in a future project of a future federated Discourse platform for another client.

  • :ballot_box_with_check: 2019-01-29: Blivande migrated to a federated Discourse platform. (Internal target date, not a strict deadline.)

  • :ballot_box_with_check: 2019-02-28: Ethical consent funnel for POPREBEL finalized. This is a modification of the ethical consent funnel on edgeryders.eu, using different questions as agreed on with an ethics advisor. The translated version might follow a bit later. (Internal target date, not a strict deadline. Details discussed here.)

  • :ballot_box_with_check: 2019-02-28: Biofabforum migrated to a federated Discourse platform. (Internal target date, not a strict deadline.)

  • :ballot_box_with_check: 2019-02-28: Software upgrade of edgeryders.eu. Necessarily done after the Biofabforum split-off as we will remove our custom code from Discourse that we used for the first iteration of the Biofabforum. (Internal target date, not a strict deadline.)

  • :ballot_box_with_check: 2019-03-01: POPREBEL platform launch. (Was 2019-02-11 before.) This will simply be a top-level category on edgeryders.eu. A federated Discourse platform would separate POPREBEL too much from the existing Edgeryders user community.

  • :ballot_box_with_check: 2019-05-01: NGI Forward platform launch. Will simply be a top-level category on edgeryders.eu, and an own domain that works as a forwarder (or site of entry, or read-only version). A federated Discourse platform would separate NGI Forward too much from the existing Edgeryders user community.

  • :ballot_box_with_check: 2019-06-01 or later: Code internationalization feature. This is not included in any deliverable for the POPREBEL or NGI Forward project, so formally there is no deadline except the end of the project.

    This 2019-06-01 date would be a good one for researchers to start coding in conjunction with the multimedia coding feature, but it could easily be pushed back a few months (esp. when providing features that allow researchers to add code internationalization when the feature is ready, while they can already start coding with codes in English or in their own language right away).

  • :ballot_box_with_check: ≤2019-05-31: NGI Forward façade website launch. A pretty-looking entry website on its own domain that works as a forwarder or site of entry for the Discourse based NGI Forward communication platform.

    It was agreed that this can be done after launching the Discourse platform. Recently Nadia requested that work is started on this (and it should be finished within month 5 of the project, in line with the corresponding “platform launch” milestone of the project plan):

  • ≤2019-06-30: OpenEthnographer multimedia coding feature. The POPREBEL project plan requires this as Deliverable 2.1 at “month 6”, which means anytime in 2019-06. Delivery includes “code for multimedia coding deployed, committed to GitHub and documented”. See details.

    UCL sent a template for this deliverable, but probably we will not have to use it as this is of type “code”. See the grant agreement for this.

  • ≤2019-06-30: POPREBEL Data management plan. “A document detailing the governance of the data produced during and after the project.” For more details, see here from “Types of data to be deposited …”. The POPREBEL project plan requires this as Deliverable 2.3 at month 6, which means anytime in 2019-06. People involved: lead by Edgeryders OÜ, collaboratively by Research Director and tech team.

  • ≤2019-07-17: NGI Forward Ethical & Data Management Plan. The NGI Forward project plan requires this as “Deliverable 6.5” at month 7, so ≤2019-07-31. But the consortium leader wants to see deliverables two weeks in advance. People involved: the tech team leads Edgeryders’ contribution to Nesta’s lead.

  • >2019-06-01: POPREBEL façade website launch. A pretty-looking entry website on its own domain that works as a forwarder or site of entry for the Discourse based POPREBEL platform. It was agreed that this can follow at some time after the launch of the Discourse platform. Date to be decided by the POPREBEL community management / engagement team.

  • 2019-mm-dd: Software design for the future of Graphryder. Associated task here.

  • ≤2020-01-31: Graphryder interactive dashboard feature v1. “A software application to generate and refine on the fly visualizations of semantic social networks.” POPREBEL Deliverable 2.5 (month 13), also NGI Forward Deliverable 2.3 (month 13, means same date or later).

  • ≤2020-01-01: Open Ethnographer UI rework / improvements. This is not included in any deliverable for the POPREBEL or NGI Forward project, so formally there is no deadline except the end of the project. In practice, improvements can be done incrementally as time is available and based on feedback from ethnographers, and most of that work would probably be finished in early 2020.

  • ≤2021-10-31: Graphryder interactive dashboard feature v2. “A software application to generate and refine on the fly visualizations of semantic social networks.” POPREBEL Deliverable 2.7 (month 34), also NGI Forward Deliverable 2.6 (month 34, means same date or later).


#2

Matt, this is… Beautiful. Is there any reason why it is in a protected cat?

This made me swell with pride for our little company. :grin:

As for Graphryder, to a first approximation we will need to encode visually the new data structure and put it under the control of the user. We have two new elements:

  1. The hierarchy of codes.
  2. The different fora (cats) from where the posts live. This is not really new, but it has not been a problem so far.

With the hierarchy of codes you might want to generate visualizations of the data at the child level (“only the data with Polish/English/etc codes”) or at the parent level (“all data”). Child code level visualization poses a choice, as you can either visualize the whole network at once (all Polish codes AND all English codes AND…) or select for the codes in one of the languages. If you do the former, the co-occurrences network will resolve into n “islands”, where n is the number of languages you are doing your work in. Given vectorial visualization this might not be a problem, as the ethnographer can pan and zoom at will.

I have not yet thought about how we might fully take advantage of the fact that we have different fora. If we don’t account for it, that might be misleading. This is certainly true in the social network. That of OpenCare (one language) has the usual core-periphery structure that we see in this type of social interaction (Edgeryders itself is the same). But if you have a convo in different languages, you will get a highly modular network, with people talking in Polish over here and not interact much with those other guys there, who write in English. A few bilingual people will connect these well-defined community. In other word, the social behavior of the participants is influenced by how we have set up the conversation environment.

Finally, I would like better support for downloading the graph and treating it with data science techniques outside Graphryder. This is great because it takes pressure off developers: some feature requests that might be one-offs can be treated by me in Tulip, instead of by your team. I would like to download the main graph, probably a conversion of the Neo into .tlp or some other Tulip-compatible format. Right now I can only download what I see on the screen, with a substantial loss of information.


#3

Thank you for this clear and structured plan. I like it.

I think that it might be wise to think about developing the backend for Graphryder in such a way that a single installation could host many graphs, and develop the front end in such a way that many graphs could be explored by same instance. This would be a significant but not enormous amount of work, and we could probably pull it off with maybe 15-20k EUR of budget, over the course of a few months. It could be a worthwhile investment that would pay for itself after 5-6 projects using it. If we had the budget, I’d be happy to lead such a project.

This is perhaps some nitpicking, but there will certainly be more “big picture” software development plans in Edgeryders during this time. In fact, if Participio takes off, there will be one such plan every year.


#4

Alternative proposal: extend Graphryder with a multisite mechanism similar to the one Discourse possesses. Means having multiple parallel configurations available, and the graph shown by Graphryder depends on the configuration that was activated when accessing it.

This would fit in better with the system of federated Discourse sites that we’ll create to provide proper whitelabel platforms for POPREBEL, NGI Forward, Blivande (if you like) etc. – see here under “1. Solution: Federated Discourse site” for some details. Because for each client having their federated “own” Discourse platform, it will also appear that they have their own Graphryder instance to explore it.

It would also be less complex, as no complex user-and-roles feature has to be implemented. This way, it would need less than the 15-20k EUR (which would require investing from Edgeryders profits as it exceeds the budget share in POPREBEL / NGI Forward).

Fixed it for you in my original post :wink:

No, except for possibly a reason why this whole category is still protected? I think it’s intended to become public at project start (whatever counts as that). cc @anique.yael


#5

Thanks for the input re. Graphryder, Alberto. The amount of work is pretty clear from that (though we’ll have to discuss some features in more details later – especially, I still think that multilingual codes should be implemented as translations, not as child codes in the hierarchy).

To make things complete: what’s the plan for the Edgesense and Assembl tools? For example, can we use or extend Graphryder in a way that allows it to fulfill the same functions as Edgesense? The fewer custom tools we have around, the cheaper their maintenance will be …


#6

I kind of like this. There is something in sitting on one single database for SSNA, also for for future re-uses as a training dataset for algos etc. Before we make any hard-to-reverse technical choice, we would need to talk this point through in depth.

No one is clamoring for Assembl – and anyway this is the competition’s tool (Bluenove). We drop it. And yes, the social network pane in Graphryder is the same thing as the social network in Edgesense. The latter has more bells and whistles, and two substantial differences:

  • Extends to the whole of Edgeryders (no need for coding).
  • Keeps track of changes in time.

The first feature is fundamental: if Edgersense is a community management tool, it needs to read the whole Discourse interaction network. The second is negotiable. It’s nice to have when you go impress people with the time slider, but in your everyday uses you are going to look at the current network only.

All in all, I would be OK with a plan that scraps Edgesense and looks into developing Graphryder so that it takes its place.


#7

Ah, yes, but it does not quite achieve what I mean here. You will still want to be able to access multiple networks for the same plattform, as done today with tags. For example, you might run a plattform for a community over a number of years, and during that time you might want to run a number of SSNA projects for different subsets of posts. Take for example, the case of the Open Village conversation that I recently made a GR instance for. We might code other conversations on main ER plattform later, and want to dive into those as separate SSNA investigations. Similarly, after we retire a white-label platform and transfer the conversations to ER, we still want to keep them explorable through their SSNA graphs.

With a federated approach, to reach what I’d like to see, Graphryder would have to be both multisite as you describe and be able to handle multiple projects for each site.

How much is the budget for Graphryder work for POPREBEL and NGI Forward combined, approximately?

I also thinks this makes sense. I think a lot of the code to do this is already in place in GR. Furthermore, GR uses more heavy duty graph viz libraries than Edgesense, and I think it even runs on WebGL. It would probably be able to handle the large amounts of nodes and edges without choking like Edgesense does.

I read all of this again, and I have to say that the plan around the federated plattform seems super solid and good. Great work! One thing I’m curious about, when you write that the user accounts will be synced, do you mean that a user from one federated site could be tagged on another without having ever visited that site? In that case, the notification would then show up somehow in the new notification menu?


#8

That’s the part I still have to work out. But for reference, the tech budget of POPREBEL and NGI Forward combined is ~50,000 EUR and has to cover everything listed above except items “Dynalist integrations” and “H2020 project reporting toolkit”.

Ah, certainly. I misunderstood. That multi-project part is needed as well for taking over the current use cases of Edgesense (i.e., looking at the whole platform). For this part I imagine it in such a way that the first step in the (new) Graphryder interface is “content selection”, via a few form elements that let one filter by tag, category and ethnographic codes (again selected by their author and / or hierarchy). And perhaps by the time span of post creation. The a semantic social network will be created for that, and the associated data would also be provided for download.

The aspect that cannot be covered by a multi-project, multi-site Graphryder is analyzing the network across the federated Discourse sites. Technically it would be possible as the user accounts are shared between them, but the data will be under the control of various different parties / clients. So let’s hope no researcher gets on that idea … :smile:

I think for an @mention to work the user must have visited that other site at least once before. It will be based on the existing Discourse SSO (single sign-on) plugins, and I’d surprised if it works differently there (but have yet to check). Stack Exchange also works this way: on the first visit with login to a new federated site, the user account is created, always with the existing username and without a need to enter ones password again.


#9

Yep. As I wrote above, I see some value (potentially, even some financial value) in a single Neo with all the corpora ever coded. It becomes “digital ethnography central”, so that we make a claim to own that space. Plenty of network externalities in the RECODE vision, I am loathe to throw them away.

In practice, if a federated site goes dark, we still have all the data with a #ethno-something code in Neo (and perhaps in ER as well).


#10

Since it’s a bit buried in the document above, I should bring the attention to this myself:

My current plan for both the POPREBEL and NGI Forward online platforms is to launch them as federated Discourse websites. It’s the thing we called “whitelabel platforms” but with a new implementation. Means: each platform has its own domain, shows only categories related to its own project on its homepage, and is integrated with edgeryders.eu only via a “single log-in” function, a cross-platform notification menu and some cross-links in other places. It will also have its own data set (“corpus”) in Open Ethnographer and Graphryder. For more details, see here under “1. Solution: Federated Discourse sites”.

If that technical implementation would conflict with anything else in the project, now would be the time to tell me :slight_smile: @anique.yael @amelia @directors


#11

I don’t like it much, Matt. Because:

  1. I would like to involve the existing ER user base in the new projects.
  2. I would like to pull the participants of the new projects into ER.
  3. I would really like to keep the database (including OpenEthnographer stuff) into the main site.

This is both for technical reasons and for commercial pitching. The larger our community and our dataset, the stronger we are.

Can you reassure me on these points?

Wait, I would like to store the database on the main site when this ends.


#12

My main reason was that the NGI Forward project wants this:

“Interoperability” probably means just a link from ngi.eu, and choosing similar colors. But I’d consider it weird and not smooth if that link goes to (say) edgeryders.eu/c/ngi-forward and then users find a truckload of other, unrelated information on edgeryders.eu.

Still possible by cross-site @mentions and a content cross-propagation plugin displaying in the footer. But serendipitous content discovery by users would likely suffer …

Again possible by cross-site @mentions. But it forces users to understand how they can be referenced from a site they never visited. So the confusion would be moved from the point of arrival to the federated platform to this point of discovering the other parts of the federation …

When the project ends, yes, totally. We’ll import over all the content of “our” federated Discourse sites into the main site when they fold up. I even think we should even put that into a contract template that we’ll use for other whitelabel clients such as Trust in Play and Timisoara: that the public content has to come with a CC-BY licence and that we may archive it incl. all metadata on edgeryders.eu if and when they want to close down their whitelabel site.

 

So, hmm. A federation setup has its own technical complexities, and avoiding it for these two projects is certainly less work on the tech side. So if we don’t have to host the POPREBEL and NGI Forward content on their own domains, maybe it’s better not to? We could take our time to get the multisite setup “right” in smaller projects like Trust in Play, Biofabforum and (if Hugi wants to migrate) Blivande.

Anyway, still interested in other opinions. The advantage of a federated setup is “more freedom”, of course: everything we customized on edgeryders.eu can be customized differently.


#13

I do, I think it would makes sense to have Blivande be a federated site rather than a category on Edgeryders main. And much like Alberto, I’d like to be able to pull people from Blivande into other ER sites and vice versa.


#14

Thank you for this comprehensive plan @matthias!

I don’t understand some of it due to my level of technical knowledge but trust @alberto’s feedback has filled the gap there.

Some questions:

  • One of the selling points of our contributions to both POPREBEL and NGI Forward has been that we will leveraging an already existing online community. How will the activities of the federated sites feed back into the Edgeryders main site? For example, how will the “Content cross-promotion plugin for whilte label platforms” feedback into edgeryders.eu rather than just between federated sites?

  • Similarly, can I double check that the while Discourse-to-Discourse migration script mentions content import after decommissioning, will it also enable users from the new sites to continue to be active on edgeryders.eu?

  • On this note, having just come across how OPENCARE’s fora have been archived into one workspace, I’m wondering if there is a way to expand on this such that the sub-fora can be kept clear? The sub-fora made it really easy for people to go to their content of interest easily and considering the content will maintain relevance, as well as the projects as examples of our work, I’m wondering if there’s a way to create a kind of portfolio subsection that can maintain the past projects in their form, while still avoiding what you rightly so have called a “category graveyard” @matthias?

Some clarifications:

  • The “interoperability” with the existing NGI.eu site is important yes. But keep in mind that we are taking over “consultation” from that project in NGI Forward. As above, this is based on an already active community. If we have an empty site it will be an issue. How can we work with your suggestion of federated sites while showcasing the existing active online community?

  • Multimedia coding requirements in OpenEthnographer are indeed image, video and audio. I recommend you liaise with @amelia and @mette for further specifications.

  • Please note that if we win AgriAgent, the two stage proposal that we should hear on in December or January, there will be increased Graphryder integration requirements. We have 18K allocated to this in addition to 12K for tech development and maintenance (30K total). If you’d like further details at this point, I suggest you review Work Package 4: Design and Development of AMB/DL framework from p45 of the Agri Agent proposal here. Specifically we would be working on integration of Graphryder into a project wide interface as follows:

  • On Dynalist integrations, I agree with all your points here, and the Gantt chart integration is a fine idea. I haven’t had a chance to move on freemind integration and let’s see if it becomes a priority for others.

  • On financial reporting, I’ve shared my thoughts in the thread Financial reporting for a Horizon 2020 project

  • On timing, please note that the POPREBEL platform cannot go live until the Ethics and Impact Plan is approved by the Commission. This is due at in February (hopefully we can submit as early as possible), with the platform going live planned for March.

Thanks again and do let me know if there’s anything else you and your team need from me at this stage.


#15

This has been true across the board. If clean whitelabeling means separate databases, we keep it as a solution of last resort: some clients will ask for it, and might make sense for ongoing communities, but we will always try to steer them towards cats in edgeryders.eu for one-off projects, like all H2020 stuff. This is the way that most adds value to our user base and our data base, and this is where it is our best interest to go.

Maybe we can think about a solution made of:

  1. A cat in edgeryders.eu.

  2. A self-standing site that populates itself via queries to the edgeryders APIs. @owen showed me a framework that does just that, and was saying how easy it is to create specialized sites based on anything that has JSON APIs (cats, tags…) . He is using it to make the particip.io website. This is read-only, but it might be enough to placate the client’s comms gods. Meanwhile, the real work is done on the platform.


#16

That seems like a pretty good compromise. Generally, I think the way Open Care worked was pretty good. And, as Matt points out, because of tagging it is easy to call it all back up if you wish.

Seamless movement between communities of interest is a good goal and not that simple to solve.


#17

Ok, so I think we agree on a solution by now.

So: both the POPREBEL and NGI Forward projects will live in public top-level categories on edgeryders.eu. And, as Alberto proposed (and we usually do) we will have landing pages. These may pull featured content from the edgeryders.eu site and display it, and can be multilingual, but in total will “just” be read-only funnel websites, without a way to log in there.


#18

I think @owen has discovered an efficient way to make read-only sites that update dynamically from Discourse APIs. Maybe we could ask him to make these pages outside of the platform (give partners more aesthetic freedom etc.)


#19

Good idea. I’ll discuss with him about that. Hopefully we can make it in such a way that we can more easily re-use the basic structure and code of these funnel websites for future projects.


#20

Thanks @matthias, you can see some of the work here - https://participio.netlify.com

I’m currently working on integrating these API calls into individual components in Vue, so they can easily be used on any webpage and styled accordingly - this should make things a little easier for templating.