As per agreement with @alberto, I’ve been working on setting up a new instance of the Graphryder API to query the OpenVillage dataset. I’ve made some progress, but now I’m running into some issues that I’d like to check in with @melancon and @jason_vallet about.
What works
-
I’ve set up a new Neo4j database on AWS and installed the graphAware plugins.
-
Fixed a small bug resulting from changes in the neo4j python package
-
Got the API and dashboard up and running locally
-
Found the right configurations to import the correct dataset from ER Discourse. All of the needed variables are not in the example config file, so that should be updated and documented.
[importer_discourse] abs_path = https://edgeryders.eu/ users_rel_path = administration/annotator/users user_rel_path = u/ tag_rel_path = tags/ tag_focus = ethno-openvillage-mena topic_rel_path = t/ posts_rel_path = posts/ codes_rel_path = administration/annotator/codes annotations_rel_path = administration/annotator/annotations admin_api_key = MY_API_KEY admin_api_username = MY_USER_NAME -
Successfully ran the HardUpdateFromEdgeRydersDiscourse call to populate the new Neo4j database with ethno-openvillage-mena dataset. Just to demonstrate, here is a (slightly pointless) Neo4j cypher query showing the shortest path between me and the “coffee” tag:
-
Dashboard conversation view renders correctly
What does not work yet
After this, I run into problems. Next step should be to ask the API to generate the graphs by querying the Neo4j database and create graph data files using the Tulip library. This is done by calling api-url/generateGraphs. This call seems to be successful, and files are generated in data/tlp which look something like this:
(tlp "2.3"
(date "09-29-2018")
(comments "This file was generated by Tulip.")
(nb_nodes 1958)
;(nodes <node_id> <node_id> ...)
(nodes 0..1957)
(nb_edges 4198)
;(edge <edge_id> <source_id> <target_id>)
(edge 0 0 1951)
(edge 1 0 1935)
(edge 2 0 1914)
(edge 3 0 1897)
(edge 4 0 1896)
...
However, going to the Detangler View just renders empty views. On the backend, it looks like something has not loaded correctly, giving error traces ending in lines like this:
File "/Users/hugiasgeirsson/gitrepos/graph-ryder-api/routes/tulipr/tulip_layout.py", line 25, in get
private_gid = self.gid_stack[public_gid]
KeyError: 'usersToUsers'
There are actually three of these errors, ending in KeyErrors for usersToUsers, tagToTags and commentAndPost.
It looks like there are supposed to be keys in gid_stack that aren’t there.
Going to Code View or Code View Full simply ends up crashing python. Here’s all I get from the terminal before it crashes after going to Code View:
"GET /count/post/1443549843638/1538244243638 HTTP/1.1" 200 -
"GET /users HTTP/1.1" 200 -
"GET /count/comment/1443549843638/1538244243638 HTTP/1.1" 200 -
"GET /tags/0/1538244243655/10 HTTP/1.1" 200 -
"GET /tags HTTP/1.1" 200 -
Initializing
Export
127.0.0.1 - - [29/Sep/2018 20:04:05] "GET /generateTagDateGraph/2940/0/1538244243655 HTTP/1.1" 200 -
Right after that, python crashes.
Some background about my setup. I’m running the Neo4j database on AWS, having set it up with the template available on the AWS marketplace.
I’m running the API and dashboard locally, on OS X 10.13, running the API on Python 3.6.5.
So, @melancon or @jason_vallet, do you have any hunches as towards what might be wrong? I can’t seem to find any additional instructions in the readme.
The questions you ask are far too deeply rooted into the code for me, I won’t be able to help unless I can find the time to dig into it and track the problem. 
