This thread is for coordination around the new experimental Graphryder and coordinating with collaborators. To not confuse this with the old codebase, we will call this experimental version “RyderEx” for the time being. In brief, the architecture of RyderEx is as follows:
An import script connects directly to the Postgres database of any number of Discourse platforms with the Edgeryders OpenEthnographer extension and imports all conversational and ethnographic content from those platforms into a single Neo4j graph after scrubbing and redacting any protected, private or sensitive information from the data. This script, written by me, is more or less finished and works as expected. Importing new data and rebuilding the entire graph from scratch takes less than two minutes, and is intended to be done every night once it has been deployed live.
Data from edgeryders.eu, bbu.world and forum.blivande.com are now live on a staging Neo4j database, which can be reached here: neo4j@bolt://graphryder.edgeryders.eu:7496. There is a password that can be given on request, as this database only contains data that is already public on these platforms.
These nodes and primary data relationships that are available in the datamodel:
There are more relationships that are not shown here, but those are redundant relationships that are kept to either make the code less complicated or to make queries faster.
I have started working on a GraphQL API in the same GitHub repo to access the graph from Neo4j. This is still barely working, and most of the resolvers and connectors are missing. This is what I think we should use to send data to the various clients we will build. GraphQL and Apollo are not technologies I am very used to coding with myself, so I will be looking for support on finishing the GraphQL API more efficiently.
Dashboard with Sigma and Graphology
I would like to start experimenting with building a new dashboard with Sigma and Graphology. My preferred way of working would be to contract someone with expertise in these technologies to build the framework and some functional examples based on our wireframes, and we can then continue according to their template.
Since we don’t know yet exactly how the new Graphryder should look and what it should do, we will be working to create a basic and modular framework that can easily be adapted when we come up with new modules.
These wireframes outline features of the experimental RyderEx, which will be the foundational functional prototype and proof of concept for the new Graphryder.
We are going to build a simple dashboard using three components that share a datamodel.
Component 1: Select platform and corpus
Component 2: Show cooccurrence graph for selection
Component 3: Show user interaction graph for selection
Components share the same data and when one is manipulated that also update the other components. Functionally, we want to achieve the same thing as on the old Graphryder Detangler view. However, in the first iteration, we are not concerned with exploring the content by clicking on nodes and relations. I am confident that this will be reasonably trivial to add later.
We should build the RyderEx prototype in such a way that we could add other components that manipulate the same data in some other way. For example, a third component that dynamically shows a list of posts with the selected subset of ethnographic codes. We might also want to look at other graphs for the same subset of posts - for example a graph that shows the different topics and their connected posts. These things are all possible with the old Graphryder through server side rendering, and one main point of inquiry with RyderEx is to understand how we could do the graph manipulation client side with Graphology.