Modular web kit strategy

In the last year, we have started developing a new set of tools that use the API of Discourse to both display and post content through project-specific websites and forms. Development of dynamic project-specific websites based on Discourse data started as an experiment by Owen and me at the Participio hackathon in Medenine in 2018, was then explored further with the development of the Culture Squad website framework (paid for by Participio and the Culture Squad budgets) and then brought to its current state by Owen for POPREBEL, NGI Forward and the Festival, under Nadias guidance and funding from her budgets. The most recent addition is a piece of software that posts content to Discourse without requiring a user to first register on the platform.

The resulting tech stack is a good example of how small aligned bets and experiments across our projects can lead to a consolidated technology and method. Through this experimentation, we now have the following:

  • A defined stack of tested technologies and methods to build these websites, and the in-house knowledge of how to do it
  • Some boilerplate code that can be manually cut and pasted to build new sites with less overhead
  • The beginnings of a consistent visual style for our front facing websites and forms
  • A “minimum viable product” middleware software that caches and packages content from Discourse to serve on websites.
  • A front-end framework to create custom forms that cook the responses into a discourse post and post it under a new user
  • A backend service that can handle the submission of these forms and create new users

A way forward: Modularity explained

We are now at a point where Owen can create new such sites for new projects, given time and money. However, Owen is a scarce resource, which limits our capacity to launch new project websites as we scale. Luckily the technical framework Vue.js, which all of our sites are built on, excels at modularity. By design, much of the current code is already modular and ready to be standardized into a library of components that could then easily be called upon to construct new sites. A component would then only have to be told what data to use – for example, to render preview cards of posts with a certain tag, or a gallery of user-profiles from a certain user group.

Given a library of standard components, and a defined language for how to configure them, a completely new project website might be rendered by supplying a template. Some sites might need custom components, and those could just be defined in the scope of that site and called like any other component from the template.

To explain how this would work, the following pseudo-code template might render the NGI Exchange site.

Click to expand pseudocode.
{
  "brand": {
    "title": "NGI | Edgeryders",
    "logoSrc": "/favicon.ico",
    "imageSrc": "/logo.png",
    "colorPrimary": "dark-blue",
    "colorSecondary": "light-blue",
    "colorHighlight": "white"
  },
  "sections": [
  {
    "type": "heroBanner",
    "body": "What should the Internet - and all it encompasses, which means pretty much everything - really do for humanity? How should it work? How should it be governed?",
    "settings": {
      "image": "globe.png",
      "gradient": {
        "primary": "light-blue",
        "secondary": "white"
      },
    },
    "buttons": [
      {
        "title": "Latest events",
        "url": "https://someurl"
      }, {
        "title": "Join the conversation",
        "url": "https://otherurl"
      }  
    ],
  }, 
  {
    "type": "descriptionTwoColumns",
    "title": "Introduction",
    "image": "networks.png",
    "imagetext": "An Opte Project visualization of routing paths through a portion of the Internet...",
    "subtitle": "While early internet pioneers dreamed of an internet that would be open, free and decentralised, the story of the internet today is mostly a story of loss of control.",
    "body": "Many of us feel increasingly uneasy about these developments. We live in a world where new technologies happen to us; the average person has very little agency... ",
  }, 
  {
    "type": "howTo",
    "title": "How to participate",
    "body": "My Cool Project is a quest to understand how we the people, who contribute towards building the evolution of the Internet and our digital technologies, actively build a better future against a backdrop of massive social, economic, ecological and political challenges.",
    "steps": [
      {
        "index": 1,
        "body": "<a href='https://someurl'>Sign up here</a> to join our exchange."
      },
      {
        "index": 2,
        "body": "Join a Virtual Cafe community call – an online video gathering to meet other community members! Calls are hosted in English."
      }
    ]
  }, 
  {
    "type": "latestConversations",
    "title": "Latest conversations",
    "tag": "my-cool-project-conversations"
  }, 
  {
    "type": "people",
    "title": "Meet the organizers",
    "tag": "my-cool-project-conversations"
  }, 
  {
    "type": "projectFooter",
    "title": "NGI Forward, part of the NGI Initiative",
    "body": "The NGI Forward project and exchange platform are part of the Next Generation Internet (NGI) initiative, launched by the European Commission in the autumn of 2016.",
    "image": "ngi.png"
  }, 
  ]
}

This syntax is very similar to that of the form builder by @gdpelican, so some work is already done on how to implement this syntax. While it requires some technical aptitude to write these templates, it’s a lot easier than writing the Javascript and CSS. It’s also easy to document and teach, and creating a site with this template language can be done in a couple of hours instead of days. Further, the text is easier to update and new sections can be added without much overhead.

A self-evident drawback of this approach is that sites built purely with our standard components would look quite similar. A few tweaks would differentiate the sites a bit, like a per-site color scheme configuration that is applied to all components, and custom cover-images for the header banner. However, if done carefully, we use this standardization to our advantage – perhaps it’s hard to say what Edgeryders IS, but at least people will know how an Edgeryders project looks and feels.

Edgeryders landing page and associated sites

Once we have a framework like this, we can start building an Edgeryders landing page and an ecosystem of sites for different aspects of the company, community, and its projects. Owen has already built and sneakily released start.edgeryders.eu, which shows the general idea. Building the landing page with this framework would just be another templated page, however with a few custom components.

When launching a new site, we would not deploy it apart from the landing page. Rather, we would just add a new config file to our codebase, together with any necessary custom components.

Some components would always be visible on all sites:

  • A top navigation menu (similar to the top navigation on the platform). Projects would be in the navigation while relevant.
  • A standardized footer, with company information and contact details.
  • Eventually, once it works reliably, your profile picture if you are logged in, and perhaps with a dropdown showing the same notifications as on the forum.
    16

Suggested components

This is a list of some potential components that we could standardize, to give you an idea of the potential. All screenshots are of Owen’s excellent work, which would be the foundation on which configurable template components could easily be built.

Site cover header module


A header module, allowing you to pick a custom illustration, decide on color and style, define buttons, titles and text.

Language selector


A module which could change the language strings of a loaded site.

Stories modules


Show stories from the platform which are loaded through a chosen tag. We probably want a few of these to vary between.

People module


A gallery of people and their user profiles, which could be manually curated by inputting the user IDs to the template, or by letting it show some set of users through the API.

Events festival module


Any project could now decide to run a festival, by simply including the events module and following its conventions when creating correctly tagged event posts on the platform.

Conversation preview module


Interesting posts and replies can draw people into participating in the conversation. One twist on this would be to also somehow show the ethnographic codes of the posts or to even allow users to select an ethnographic code and see a selection of content with that code.

Social module


Another benefit of this modular system is that we could update components automatically on old sites when we make changes to them. For example, if Edgeryders gets an Instagram account and we add the icon to the Social module, it gets added automatically everywhere where the social module has been used.

Edgeryders description module


Again, as the description of Edgeryders gets updated, it would be useful to have an up to date module to describe the company/community in a section of project sites. Even if people stumble across an old project, the description of Edgeryders on that project page will be up to date.

Real-time chat feed module

Owen has started developing some very promising interactive social media feed components that could allow us to have realtime content from various sources feed directly to our sites. This can be developed into a component that could either be integrated on project sites or run on its own on a designated social feed site for a major event.

On-boarding form module

Our onboarding form is built on essentially the same tech stack as the mini-sites, and that framework could be turned into a module. Every time that module is called, you would only need to define in what category the resulting content should be posted.

In time, this onboarding form could be developed into becoming the default onboarding experience when first encountering Edgeryders. Instead of being thrown into the deep end of having to register on the forum and find a category to introduce yourself in, you would follow a step by step process introducing you to Edgeryders and asking you questions about yourself in the context of what project you are entering Edgeryders through. At the last step, it would send you a personalized boarding email and redirect you to the forum.

Ethnography & SSNA interactive modules

For our projects with ethnography and SSNA, we could build modules that display some of those results by querying the Graphryder API. A first step could be to simply make it possible to embed interactive versions of the user graphs and code graphs, configured in some way that makes them possible to understand. By starting to experiment with this, we would begin to write the beginnings of modules that we could then combine into a rewritten Graphryder dashboard that would then be much more maintainable than our current dashboard.

Expected outcomes

Better onboarding

I predict that it will become a lot easier to onboard people when we don’t throw them straight onto a forum. I have witnessed that it intimidates most people from engaging. Having a smoother journey that looks more like what people expect from the web these days, is likely to help us with engagement.

An even better offer from the ResNet

With these tools added to our arsenal, our offer to research consortia becomes even better. We can start making bids that leverage things like offering to build project presentation websites like that of the NGI Policy Lab. Adding an up-to-date public-facing presentation of the conversation, ethnography and collective intelligence also levels up our current offer.

New entry-level products to offer

One challenge we have is that we don’t have much to offer a corporate client on the entry-level. Building relationships with corporate clients often requires you to first start by getting them to commit to a small investment, and then secure larger and larger contracts. If our capacity for building slick real-time updated websites with social content, we could use that to sell seminars and one-off events to build new client relationships. I think that our explicit goal should be to bring the costs down so much that we can afford to create a site for any project worth more than 10k EUR.

Another way through which this could become profitable is to sell setup of a white label version of the framework to communities on Edgeryders Communities. They could get their own interactive and up to date website built on content from their platform. If they need custom modules, they pay us by the hour, and those modules then become open-source and possible for us to use. We could start by deploying it for the Blivande forum as a test case, and use that to sell it.

Timeframe

First, I would like to sanity check the assumptions and test the idea with you. Have I gotten something wrong? Does the template architecture make sense?

If we go ahead, we have a few upcoming projects in 2020 which we could leverage to get this framework built. I’ll return to this post once we have some clear milestones and timelines, but I’m expecting us to work on the first version of this January to March.

Ping @owen, @matthias, @alberto, @nadia, @noemi, @johncoate.

2 Likes

It does, though a non-developer depends on the developers for assessing the productivity gain.

What I like, predictably, is this:

Even though we then need to put time in imagining a viable 5-10K product. Workshops are a popular format, and with @nadia we have started this discussion many times. Maybe we will be able to make more progress in 2020…

Hi Hugi

Hi. So strategically i would focus in a somewhat different direction as this reads to me ( as an external a client) like no different from e.g wordpress or any other plug and play website generator. Rather, what interests me is looking at at how this can make the next editions of our participatory distributed events a smoother and more rewarding experience for local organisers, as well as for the people coordinating between all the different groups doing things in different places. Taking a view of the whole cycle, from announcement, to documentation to production of a follow up document. Then triage.

As an internal client my primary focus is in making the flow for our event organisation method smoother. E.g I got this feedback re the form functionality:

“The Event Registration Form wasn’t friendly to user to sign up, as the user experience didn’t help them and it didn’t open from different browsers as I got different messages about this. As the form moves to next page easily from mobile that causes that most of participants data wasn’t complete for most of them.”

Hmm, I think you might be getting caught on the how instead of the what. The template architecture is just a technical implementation detail to achieve exactly this reliably:

For this to be the case, we need to be able to quickly iterate based on feedback. We do that through improving in increments, and that’s best done through being modular. This strategy doesn’t contradict improving our participatory event flow, it’s rather going to get us there faster.

Nice summary @hugi, I agree that modularising the most common components is a no-brainer, given Vue’s component system is designed with modularity in mind. I think we are half way there already.

What needs to be concretely done:

  • Standardise the stylesheet across all components. The last few sites (start.edgeryders.eu, festival.edgeryders.eu) have used Tailwind CSS. The benefit of this is visual consistency, the same set of utility classes (no guessing of class names or ids for anyone who wants to update the layout), easy to configure as well as customise on a per site basis without affecting the core library.
  • Set a convention for post metadata. One of the shortcomings of the festival site is it has a fairly arbitrary way of parsing posts - one example: for an event to be successfully displayed on the site, the event has to be tagged webcontent-festival-event, event-date-01_12_2019 (this will be changed to read from the date extension @matthias has integrated into discourse), event-location-brussels (location), event-confirmed (to say it is happening vs proposed). In my view, relying too heavily on custom tags like these can lead to confusion and content breaking if something is missed. It’s not obvious to the person writing the post and it also interferes with the semantic markup that makes tags so useful (I’m not sure how the ethnographic tools work, but I imagine these tags have to be filtered out at some stage as they are not relevant to the discussions). Having extensions in Discourse to handle this, like the existing one for date/time, means we can have standard input fields for events, community calls, organiser bios… The options are clear and we don’t have to second guess how content may be parsed.
  • Standardise language options: this is a little more work if we go the template route, as working with multiple languages modifies not only content but navigation, routing, organiser profiles (in the case of Wellbeing in Europe), the login UI. If we use the template you proposed it would have to include a language schema with translations of every button/paragraph/link etc for the content.
  • (Optional but convenient) Modularise the registration form, so questions can be queried and answered within each site without redirecting to a forms page.
  • Create a standard account control panel that integrates with the navigation menu

I agree, I think what’s being proposed here is a part of that cycle… It allows us to create and deploy these sites faster and with less effort so we can put more of our time/energy into other stages of the process.

1 Like

I would need this to be explicitly worded, and the process for how this would happen also clearly defined with dates, deliverables and milestones.

  • Are we talking about doing a post-festival audit of how things have worked so far; would need to included in this?
  • Are we talking about co-design sprints ?
  • If yes who would do this, when and how?

The work:

  1. Creating the core JSON schema:
    • Site structure
    • Styling options (colors, icons, fonts, images)
    • Languages
    • Forms
  2. Creating the template components:
    • Header
    • Introduction
    • Events
    • Stories
    • People
    • Conversations
    • Social & Contact
    • Form
    • Navigation
    • Login/Register
    • T&C
  3. Create the CSS core
    • Document any custom classes
  4. Create the Vue template:
    • Generates the site from the JSON file
    • Generates a Tailwind config file for each site
    • One click deploy to Netlify through Github
  5. Data:
    • Standardise Discourse post structure for events, stories, bios, calls, campaigns
    • Decide on explicit convention for post languages
    • Create an API for storing and updating forms data

Having worked on the components, #2 is more than halfway done. The work on standardising the styles and documentation can be done in a week or 10 days.

#1 Writing the JSON is trivial, agreeing on it is more important as it determines how every site will be generated (measure twice and cut once). I have put @hugi’s pseudo code on GitHub and anyone can make amendments and suggestions to this.

Stages #3 and #4 will take two weeks, at the maximum. Once the base is on GitHub it can be improved on by anyone. The main goal here is to set the standard and deploy a working template.

Total time: 3-4 weeks.

Net gain

  • Time saved on creating and deploying sites for any future events and projects
  • Design consistency
  • Anyone with a basic understanding of JSON can create a site from scratch
  • Can be iterated on by any future developer who understands the system
1 Like

@gdpelican, when defining the schema for generating fonts, did you use some particular standard or did you come up with it on the fly?

I would add on top of this/first priority is an audit of the current workflow and experience of the different people who needs are to be met followed by a clear description of the experience that would be needed and specs for different steps of the flow. Only then look at the further development of the modules of this webkit so that they are coherent with the needs of the process and people involved in it.

There are three elements to this:

  1. conducting user research/ collect feedback from people involved in two projects this year: Edgeryders Festival and Science fiction economics lab
  2. visualising this as the “current” flow of steps and description of what did/did not work well
  3. proposing a different flow: with specs for how the modules would need to work in order to enable this improved flow, then collecting feedback from people and proposing a final version based on this.

Those that we would need to be involved in this process are:

  • coordinators responsible for the final outcomes
  • people who registered then participated in the events
  • community members who organised local events (edgeryders festival)
  • people who came on board to support the organisers of the local events
  • Community management lead of the platform (Maria, John, Noemi, Jasen, Zmorda, Sohayeb + +)
  • People who set up the different components using third party services (e.g Augusto and Owen)
  • Project coordinators (Marina and Ilaria respectively)
  • Research team leads (Alberto and Amelia)

So there would need to be some accounting for this in the timeline and budget

I don’t agree with that assessment, and I think we’ll have to work that out before we move forward. I think it makes sense to first put in the work suggested by Owen in his last post. None of those improvements affect the UX. These improvements do however make it much faster for us to become fast and agile at improving UX in the future. Before we have a good architecture, we will not be able to implement the outcomes of such an audit in a structured way, but rather introduce more mess and feature creep.

I would say we do:

  1. Modular architecture refactoring
  2. Audit
  3. Update of modules and building of new modules based on audit outcomes

In that order. Of course, 1 and 2 could be done in parallell, but I would prefer Owen to focus on the architecture first since he knows the codebase best.

The problem of designing by consensus at this stage is we run the risk of delaying and or obfuscating the basic structure we need… While the people who organise the events and have first hand experience can give valuable input, this feedback will serve on top of a basic foundation that includes:

  • A template schema
  • A modular architecture
  • Easy deployment

I don’t think those aspects will be negotiated in a design session, as they relate more to how the site is made than what it contains.

I would say we do:

  1. Modular architecture refactoring
  2. Audit
  3. Update of modules and building of new modules based on audit outcomes

Working this way gives us a foundation to then integrate those proposals and suggestions from the audit. It also lightens the work as it would break the development into two distinct stages - core and concept.

2 Likes

yes, but I would like to see a clear commitment to inclusion of UX design steps by ensuring they are explicitly included in the timeline and budget. With clear allocation of responsibilities for who will be driving and delivering this part of the work. Without this I see a risk of having nice pieces of code but missing the bigger picture because things fall through the cracks.

1 Like

Agree, it needs to be factored into the budget and timeframe. I think to underline @hugi’s point, this order of development makes sense to me once the above is made clear:

  • Architecture - (stages #1 and #4) Work on a structure for generating these sites. The components could essentially contain dummy text at this stage.
  • Audit - Collect and document feedback
  • Design - We reach a consensus on what components are needed and how they should function (UX) based on the audit.
  • Implementation - Development of components as agreed upon in the design stage. Development work is focused on UX and design.
2 Likes

Agreed. Once the work on architecture is done, it’s time to turn towards doing a survey before then moving on to redesign components or create new ones.

Ok but who will take responsibility for driving that/ ensuring it gets done? You?

If the survey and module improvement plan happens later in 2020, I might even be able to secure that funding for myself somehow to keep driving that. But it’s really no different from who takes responsibility for the other parts of the proposed strategy. Since this is an investment from the Core, it falls on the board collectively to follow up on that investment. All that we’re discussing is the sequence of events.

I think @nadia has in mind a design session in January, that would be an appropriate moment to do such an audit. In the meantime I’m happy to work with @hugi on any aspects that need developing before then.

1 Like

Thanks for the description.

Do I understand correctly, that the sites will be mostly rendered in-browser and live on the “live” API of the discourse instance? This would have the drawback, that you cannot easily save a page locally (I still live without internet connection sometimes, and from time to time save webpages for later reading, the very old style), and one maybe has to go the extra mile to make search engines able to crawl and index the page. You see already, I have no idea about Vue :slight_smile: I guess it is baked in, but that should be made sure.

On a quick note: If I would have implemented that, i would have tried to go with one of the many stable and well-working static site generators (e.g. GitHub - myles/awesome-static-generators: A curated list of static web site generators. , I guess Jekyll is still strong) that allow meta-data via front-matter; and then build some configurator on top (to play with the styling etc). Deployments of these pages is trivial, but I am not sure how easy it is to come up with maintainable complex site structures, and updating the page would not be instant (vs pulling all data live via discourse API).

1 Like

@felix.wolfsteller They are rendered in browser at the moment, and it’s a good point about archiving and search crawl. Google crawls sites rendered client side, I don’t know about other search engines.

A framework I have used in the past, Nuxt, allows static generation of sites and we could go down that road - but it would also require as you mentioned a rebuild whenever content is updated. This can be automated with webhooks or a cron job - depending on how frequently it needs to be updated.

This is why we went down this road in the first place. When we had static sites in the past, they quickly became outdated as our projects move fast and often shape-shift. Static sites representing a living community and all it does is also very unsatisfying. Hence this move towards having dynamic and up-to-date curated windows into the goings-on of the communities on Edgeryders.