IT Development Plan 2019-2021

Yep. Friday?

Yes. Friday early afternoon (ā‰¤14:00) and later in the evening (ā‰„19:00) should both be ok.

I had to take a good hard look at the Graphryder software while installing it for the NGI Forward dataset. Due to that, and considerig the available budget, the bits and pieces of the plan for the future of Graphryder have fallen into place. There is simply not much wiggle room ā€“ itā€™s pretty clear what we have to and can do with the Graphryder software within the H2020 projects. I have updated that in the plan for Graphryder above accordingly.

@hugi, this also means we donā€™t need you to make a dedicated software design for Graphryder anymore. But youā€™re welcome to contribute to the refactoring and development towards the ā€œGraphryder interactive dashboardā€ v1 and v2 deliverables. If you have time for that, have a look at the plan above and let me know what part would interest you to work on. (Also, please let me know how much the Graphryder installation & documentation work took you in the end, and if you still need to receive money from Edgeryders for that.)

No, that was all covered by core funding already. That also means that the 1600 EUR assigned to me for that work is still not used.

1 Like

it would be great if that could instead be redirected to cover some of the costs of the interface thing (Nataliaā€™ thing ).

Hmm no, thatā€™s not possible. 1500 EUR of these costs is covered by the IT budget (see) and more is not possible with an IT budget of 3% the project value, give or take the 1600 EUR. Itā€™s already a very, very squeezed budget for handling a bunch of complex software applications.

(I wanted to write ā€œNot in a thousand years @nadiaā€ but I remembered in time that this is a public forum and we should appear more professional :smile: But I think Iā€™m clear: itā€™s really not possible.)

huh? I had interpreted @hugi post above to mean that there is an unspent 1.6K there. But clearly not it seems.

Well, not anymore :slight_smile: It was available for a microsecond and got immediately re-allocated to plug gaping holes in other underfunded parts of the IT infrastructure. Like that 4800 EUR hole that appeared out of nowhere at the beginning of the project when somebody (that we do no longer work with) had cut the IT budget down.

IT budgets are indeed quite flexible as you can cut down on software maintenance and quality for quite some time and nothing is seen on the outside. But to prevent a final collapse, there must be a limit to that cutting, and I think weā€™re well past it. So whatever budget I can find Iā€™ll have to reinvest in software quality ā€¦

1 Like

Never ever a dull moment on planet edgeryders.

1 Like

That much is certain :smile:

And, in fact, each installation of Graphryder API needs its own Neo4j instance. The community edition of Neo4j canā€™t handle more than one graph at a time. This means that we need to run multiple instances of neo4j on the same server. This adds a lot of overhead, as neo4j is a pretty expensive piece of software to run, especially from a memory point of view.

In light of this, I think it is a priority to rewrite the Graphryder API to use the same Neo4j database for multiple instances. This shouldnā€™t be incredibly difficult. What we need to do is to somehow label every node and relationship as belonging to a certain sub-graph. Another, more ā€œgraphyā€ way of doing it that might be more canonical for neo4j would be to introduce the new node type ā€œprojectā€ and create a ā€œbelongs_toā€ relationship from that project to all its associated nodes. This is is a more memory-efficient way to do it in a graph database since the relationships are all direct memory pointers from the first object, the only search operation is to get a node from a very small set of indexed project nodes.

1 Like

Good point. Didnā€™t know about that ā€œbusiness modelā€ of the Neo4j community edition so far. I created a Github issue for your proposal, and added a few thoughts. Namely, when doing this we should probably go all the way and let one Graphryder API instance handle all the datasets.

1 Like

I would be up for working on Graphryder with you, especially if and when it comes to writing Cypher queries for Neo4j or working on the API in general. Iā€™m less interested in working on the dashboard.

1 Like

Great :slight_smile: Then your own proposal is a good fit for you. In the variant ā€œmaking Graphryder API multi-dataset capableā€. As it would be part of the ā€œinteractive dashboard v1ā€ deliverable, weā€™d need it until 2020-01-31 as a hard deadline. Paid by the hour under a normal H2020 project staff contract (no issue here at all as youā€™re already mentioned as staff in the Grant Agreement). But let me know should it become clear during the work that this would need more than 2500-3000 EUR in budget.

Iā€™d do the dataset switcher inside Graphryder Dashboard myself, using the modified API that results from your work. And weā€™d both fix the most annoying software rot alongside as it suits the work and as we have the time (with about 4000 EUR additional budget for that). Taken together, that will be ā€œinteractive dashboard v1ā€. As there will be all this some repair and maintenance needed alongside, the main part of the interactive dashboard would be in v2, also a deliverable but around the end of the project.

Let me know if thatā€™s acceptable and Iā€™ll send you the contract.

Yup, it is! Sounds good. What would the deadline be?

Since 2020-01-31 is our hard deadline for the deliverable (ā€œuploaded to Github and documentedā€), letā€™s say 10 days before that so we have time to put it together with my work and test it all.

Iā€™m ok with late deadlines like that if you show the progress alongside working ā€¦ so I can be reasonably sure weā€™ll be able to put together the deliverable in the end. Basically this means, make a little plan in the beginning (just edit the wiki above) and a feature branch, and update both as you make progress. Nothing special.

Gotcha, that sounds good! My November is packed full, but if I start in December there should still be plenty of time. Iā€™ll try to make a plan before then.

1 Like

Since we have at least three cases now where people found or provided some other funding for IT tasks they missed in the plan above, we have a bit of a mess now ā€¦ I donā€™t know right now how much is available for what, and under what conditions.

Can everyone please add that information in a comment or right in the plan above? @hugi @nadia @alberto

Hereā€™s a start:

  1. 5000 - 6000 EUR from core funding provided for server maintenance and improvements, incl. e-mail system and backup system.
  1. 18K total in 2020 and 2021 from NGI Forward, WPs 1 and 3, previously allocated to @hugiā€™s ā€œambassadorial role in the NGI spaceā€. Allocated preferentially to research software development, if thatā€™s secure to be reallocated by the tech team at will. Please not that @marina might need to inform the Commission of budget reallocation.
1 Like

Once this reallocation is confirmed just let me know and Iā€™ll inform the coordinator. This shouldnā€™t be an issue as we are reallocating from one WP to another without changing the cost category.

1 Like