Tech dev specifications for POPREBEL and NGI Forward

Research Network Horizon
1

As discussed and to reference in our call @matthias, here are the specifications for POPREBEL, and now preliminary ones for NGI Forward. Please note that NGI Forward is further behind in the process and things may be shifting slightly.

You had asked for the following:

  • notes (internal and from the proposal) available about the tech part
  • future extended functionalities for Open Ethnographer etc.
  • the maintenance / adaptation of the software we have
  • ideas for features to add if we have some “excess capacity”

Content

1. Notes (internal and from the proposal) available about the tech part

2. Future extended functionalities for Open Ethnographer etc.

3. Maintenance / adaptation of the software we have

4. Ideas for features to add if we have some “excess capacity”

1. Notes (internal and from the proposal) available about the tech part

IMPORTANT: Before we finalise the Grant Agreements (around end September/October), please confirm that the timing associated to each external milestone and deliverable is reasonable.

1.1. POPREBEL

  • I have shared with you relevant background information on POPREBEL and our WP2 in this post.

  • The Task associated to your team is

    Task 2.5 - Provide technical support to the WP (Involved: EDGE. Months 1-36)

    • Provide functionality for coding multimedia as well as text.
    • Maintain and improve user experience of in-platform coding application.
    • Customise and maintain a POPREBEL platform space.

    Description of work and role of participants

    • We set up and configure the POPREBEL online space. This shares the codebase and database with the existing edgeryders.eu platform, but is accessible through its own domain (for example poprebel.eu) and has its own color schemes. This requires minor extensions to the codebase (Ruby on Rails), and we have experience with it.
    • We engage an Ethics Advisor and provide the space with an ethical consent funnel. See WP8 for description.
    • We extend Open Ethnographer to support multimedia coding. This happens by building on the annotator.js library, and integrating the functionality in the version of the Discourse platform maintained by [Edgeryders].
    • We support ethnographers in the coding activity. Though the key functionalities are already implemented, this might require small front-end utilities like coding queues, or drag-and-drop interfaces to re-organize hierarchies of codes. These are implemented in Javascript to stick to browser-based interfaces for simplicity of use.
    • We provide technical support and server admin to the whole project.

  • The external milestone associated to the Tech dev team is:

    M2 (ie. February 2019)

    • MS 2.1 : POPREBEL platform ready for posting. Public access to POPREBEL forums within the edgeryders.eu platform available and registered-user capacity to post comments.

  • I’d like access to your internal milestones for overall project planning and scaling.

  • The external deliverables associated to the Tech dev team are:

    • D 2.1 : Code for multimedia coding committed to GitHub and documented. M6 (ie. June 2019). Deployal and documentation of the technical adaptations and extensions to functionality of edgeryders.eu platform (on Discourse) to allow for ethnographic coding of multimedia specific to POPREBEL.
    • D 2.8 : Data management plan (in collaboration with @alberto?). M6 (ie. June 2019). A document detailing the governance of the data produced during POPREBEL, during and after the project.
    • D 2.4 : Interactive dashboard, first version (Graphryder). M13 (ie. January 2020) A software application to generate and refine on the fly visualizations of semantic social networks.
    • D 2.7 : Interactive dashboard, second version. M34 (ie. October 2021)

  • There are no current additional internal deliverables associated to the Tech dev team for POPREBEL specifically (but some with the scaling as per below), however this may change in process and with dialogue.

  • Rather than paste 1.5 pages here, please see pages 103 - 105 of the proposal on Ethics and Data protection.

1.2. NGI Forward

  • I have shared initial background info on NGI Forward with you in this post, and have just updated it with more detailed background on our Work Packages and Tasks for ease of reference.

  • Edgeryders is more interwoven through the Work Packages and our work more demanding as a consequence.

  • The Task associated to your team is

    T2.5:Data management and technical support [M1-M36] Partner(s): Edgeryders (Task leader), DELab

    • Motivation for the task: To undertake inclusive stakeholder debate effectively, and ensure robust ethical management of the platform, technical support customizes and maintains a NGI Forward platform space, ensures smooth interoperability with the existing NGI.eu website, and provides functionality for data from WP1 and WP2 to be incorporated into the platform.

    • Work plan: We ensure rigorous and compliant data and ethics management, and maintain and improve the user experience of in-platform coding applications, by building additional functionalities and capabilities. For example, as the project will generate several sophisticated data visualisations, we want to build in capabilities to let participants annotate these images. We ensure data consistency, integrity and long-term stewardship, and commit to publishing data sets as open data. We combine primary and secondary data into a semantic social network (SSN). We export the data in a widely used, intuitive format (JSON) and store it for long-term safekeeping.

    • Partners role: This task is led by Edgeryders, with support from DELab.

    • Relationship with other tasks: This task will strongly interlink with T6.3, which concerns itself with data management and the development of a data management plan. This task also supports the rest of project by providing technical support and server admin.

  • The external milestone associated to the Tech dev team is:

    MS2.1 Platform launched Month 5

    Please note that there may be others once we get moving after the initial proposal, based on the other tech heavy Work Packages and out feeding into them. I have already asked when Grant Agreement preparations will begin and have prioritised clarifying how this will work to inform your own planning.

  • I’d like access to your internal milestones for overall project planning and scaling

  • The external deliverables associated to the Tech dev team are:

    D2.3 Interactive Graphryder dashboard, first version [Month 13]

    D2.6 Interactive Graphryder dashboard, second version [Month 34]

  • NESTA will be leading Data Management and I flagged that we’d need to be involved. Please see page 25, page 44 and Task 6.3 Data, ethics & IPR management of the proposal

  • Rather than paste 4 pages here, please see pages 99 - 103 of the proposal on Ethics as it refers to our work.

2. Future extended functionalities for Open Ethnographer etc.

  • As per the deliverables on each project above, current extensions to OpenEthnographer include

    • multilingual coding
    • multimedia coding

  • We also allowed for adaptations to be made to Graphryder that may involve API extensions. @alberto may have more to say on this.

3. Maintenance / adaptation of the software we have

  • I suspect substantial maintenance and adaptation will be required for Discourse to allow for the multiple fora across languages and stakeholder groups - we are the lead on this and will be having 1000-2500 active participants on both POPREBEL and NGI Forward - ie. up to 5000 active participants (OpenCare was 300).

  • I will want a magic spreadsheet like tool that can be used by our teams and staff for Technical and Financial reporting. I’ve asked both POPREBEL and NGI Forward Coordinators for their templates to inform how we can create one to funnel / serve both. Let’s discuss more in our upcoming call/s.

  • As already flagged, we need to adapt our project planning tool/s. You’ve mentioned that you and @anu are already working on notifications for Dynalist which are a must. I’d like us to explore how we can incorporate

    • calendar functionality
    • mindmapping software (for us visual thinkers as while I can work with lists they overwhelm me and I need to be able to visualise the roadmap and shifting parts. Let’s discuss more in our upcoming call/s.)

4. Ideas for features to add if we have some “excess capacity”

  • @alberto and I have begun talking on this and @hugi has also shown interest on synergies with SenseStack. More soon / in discussion.
2

I propose a tech meeting at your earliest convenience, with @matthias and @hugi.

Possibly relevant for Matt: Luca Mearelli contacted me to let me know he has some availability for picking up these issues. Anique: Luca is a very senior and trusted developer that has done work for us in the recent past. Those OE issues were being discussed as a backdrop to the most recent contract he had for us, when Nadia and Hugi asked him to reskin GraphRyder. Luca is very experienced in general, and especially in Ruby on Rails. So, he also could be a part of our scaling mechanism.

3

Notes from call @matthias and @anique.yael

Your needs

  • Tech specs
  • Budget
  • Details on what features have to be provided in what fashion for us to agree on to deliver performance requirements

Timeline

  • POPREBEL - Discourse tech ready by end of this year
  • Not sure about second half of next year

Reporting Tools

  • Need details of the pieces in place at UCL and NESTA and what cannot change
  • See user needs/ how people would like to use it
  • In principle, anything can be converted into anything - sometimes harder/ easier
  • Specify what is being reported - eg. Financial time and money + Technical coverage and outreach

Planning tools

  • Dynalist list as common interface and the editor the user uses is personal choice. With conventions consistent across the APIs (eg. #anique.yael, some elements not transferred (eg. visual identifiers)
  • Calendar
  • Potential to integrate Gantt charts into Dynalist by using date syntax
  • Latest start date/ earliest start date etc. - do that for the main tasks with teams refining - then computer can create dynamic Gantt charts
  • Could drill down to unfold one
  • Integration between Dynalist and Google calendar - put something in Dynalist and then automatically goes to calendar (Upgrade to Pro account)
  • Mindmapping
  • On the roadmap for the Dynalist crew
  • In terms of data structure it is a list, but not in my mind. Way to let the computer translate between lists and what my eye wants to see.
  • Freemind is the best option. Easy to copy and paste data from Freemind to Dynalist. Can draw mindmap in Freemind and then transfer. For automatic updating we could develop it - not a big task - 3 days of development + 1 week nice to use.

Particip.io
@hugi can you give us a clearer idea on

  • the functionality of your platforms needing development and
  • where you see an overlap in software that could be used by you and us?
    Take a look at the above on project planning and reporting tools as @matthias thinks there could be synergy there too.
4

Agreed @alberto, I had proposed today but looks like Hugi’s still busy. It’s on the cards and all is moving.

5

Noted. Not urgent anyway.

6

Ah, a misunderstanding then. I was waiting for a ping on the Riot or hangouts for when to join the call, was hoping to be on it.

I’m having a hard time understanding the notes and how it all fits together.
Best thing would be to have a talk about this.

7

Sure thing. I suggest you and I start - to discuss how Particip.io / SenseStack + NGI Forward align in general - and then you can take the specifics further with @matthias. I am available today or Monday afternoon, with notice.

8

So great to talk through Particip.io, SenseStack and the H2020 projects tech dev just now @hugi.

As discussed it’s clear that we need to have a call with @matthias to get clear on how to engage in the tech dev interoperability and approach the development holistically.

I know you don’t like to schedule calls @matthias but @hugi and I have a busy schedule next week so we need to lock it in - are you free next Friday 31st at 14:00?

1 Like
9

Yes, a very good and productive call. We spoke about the need to define a reproducible and easily operated pipeline for how to deploy the SSNA stack, for example:

  • Collecting the different bits and pieces together and writing documentation for how they fit together
  • An automated workflow for getting posts and tags from Discourse into the Neo4j database
  • Writing documentation for how to deploy the entire stack – from the custom Discourse to the Graphyryder API – from scratch
  • Drawing up a list of improvements we would like to implement before using the stack for POPREBEL and NGI
  • Coming up with a roadmap for how we could make it feasible to deploy the SSNA stack for smaller projects

What’s your take on this @matthias? I’m constantly running into really cool opportunities for where we could deploy the stack that are not large scale research project. In the long run, we could use this stack for smaller paid projects – for example to visualise in a graph how what has been said at a conference looks like from an SSNA point of view, or experiment with what happens if we feed it data from a large number of meeting protocols. I see a lot of applications of this outside of academic research, and I’ve even been asked by people to quote what it would cost to set up.

That would indeed be ideal for me.

10

If you need me (which I think you do), no can do. I will be at Smart Stuff That Matters in the Netherlands, with @ton.

FYI, @melancon has obtained funding from his university to build the GraphRyder idea into a robust, scalable technical solution. Their timeline is incompatible with ours: they will be working on this for a couple of years, and we need to be able to carry out analysis now. In fact, @amelia let me know that the OpenVillage - World Bank corpus is coded and ready to be put on GraphRyder for SSNA purposes.

That is not necessarily a problem. In fact, we can help each other: more case studies and more expertise in using SSNA will make the technology more attractive. We should not wait for a full-fledged technical solution, but have viable prototypes and use them as the “permanent” solution is built. So I asked Guy:

What would it take for you to produce an instructable (“install this, install that, configure these parameter as follows…”)?

To which he replied

I guess the only answer is “time”.

But that was July, and holidays were coming, and all that. Guy, if you are reading me: we should be in touch. I will reach out next week, when holiday season should be over.

1 Like
11

Hi @alberto

I am indeed back from vacation (since today), get in touch with me next week, we’ll see how we can help.

Guy

1 Like
12

Gotcha and yes the meeting needs you @alberto , @matthias and @hugi at a minimum.

How is 15:30 this Thursday for you @matthias and @hugi ? @alberto and I are both free then.

1 Like
13

(Thu) 2018-08-30 15:30 CEST works for me. Reserved it until further notice.

2 Likes
14

Confirmed!

15

Great. Let’s meet on zoom - will send the link through riot before.

1 Like
16

@matthias As per riot, an addendum to the above although likely not impactful at all.

Considering the precedence of OPENCARE and other obvious reasons, I’ve encouraged both consortia to do project coordination on the edgeryders.eu platform. Both are enthusiastic. So just noting here that in addition to multilingual and topic specific sub-fora, we will be requiring a closed workspace for project coordination related issues that can’t be worked out loud.

Again, I know this doesn’t require any additional work but simply wanting to keep the overall architecture clear :blush:

1 Like
17

@anique.yael The milestone deliverable “MS 2.1 : POPREBEL platform ready for posting” has now moved to month 3 (March 2019), right? Maybe update the specifications post at the top or just confirm here and I update it … trying to keep this document up to date as my “master todo list”.