Post POPREBEL data treatment

To be honest, I am not completely clear of how the new project functionality work. I do not see a “duplicate project” or a “merge project” functionality. Section 6 of the manual covers the creation of a new project, but I am not completely sure it’s up to date with the roll out of the projects functionality (@daniel , is it?).

To get access, researchers need to be members of the annotator group of Edgeryders users. I have now made @Wojt an owner of that group. That means that he can assign users if his choice to to that group (of course they need to create accounts first).

2 Likes

Yes, the manual will need to be updated, it does not yet consider the changes we made to the projects. @matthias will updated the manual.

To copy codes between projects is not yet possible with the backend. We already have an issue for it, but it’s not yet done. I will copy the codes as described by @Jan. I’ll implement the copy-codes feature in the next days or copy them over by means of a script. Either way should be done in a few days.

3 Likes

Many thanks @daniel ! I have written to @Wojt to get in touch with you - he understands best what we are trying to achieve. @Jirka_Kocian: please chime in if you have anything usful for Daniel. And many thanks @alberto (BTW: Ilaria and Richard had a meeting today and are working on the last issues raised by the Commission re our final report. Ilaria will be asking for some additonal information from Marina). All my best, everybody!

This is now done. All codes and annotations of the listed projects have been copied into the new ethno-rebelpop-dataset-B project.

We implemented a “Copy to project” feature which allows to copy large amounts of codes into another project. It is still hidden from the UI as we need to run it as a background process to prevent timeouts.

Until it is available in the UI we can still run the command on the console (which prevents timeouts). So if further codes need to be copied just let us know.

We will continue to work on this in a few days - so should be ready soon.

2 Likes

Thank you!

Implementation of the “Copy to project” feature is now completed.

1 Like

Dear All, particularly @daniel: we are very grateful for all your work. MANY THANKS! We are now ready to begin the next phase of our study. @alberto: I hope we will be able to catch up at some point. @Jirka_Kocian, @Wojt and I will (1) finish the final revisions of the codebook, (2) review all the coding done so far, (3) code the remaining interviews, and (4) start new interviews, code them, etc. If we have any questions or news to share, we will ;post them in this thread.

With warm regards,
Jan

1 Like

Dear @daniel or @owen

I hope you can help.

I have followed the above instructions and copied the codes from one of my projects (ethno-poprebel-sexuality-zavtra-extremism) to another (ethno-poprebel-sexuality-zavtra-propaganda-2022). However, in the destination project, each code is replicated nine times. See below:

Have I done something wrong? Also, is it possible to delete codes en masse or do they have to be deleted individually?

Best wishes

Richard

Hello @Richard, just encountered a similar behaviour by copying the codes of ethno-poprebel-sexuality-zavtra-extremism into a test project. It turns out that one of the codes can’t be copied and crashes the background job that executes the copy operation. Background jobs are automatically retried when an error is encountered until it succeeds or a retry limit is reached. This is the standard behaviour and useful when e.g. emails are sent but clearly not useful in this case.
I’ll change the background job that retries are not attempted which will prevent duplicates from being created.
I’ll also figure out why the copy operation fails. In my test it stopped after code 265 of 494 codes in total. I’ll let you know when I figured out which code it is as codes after this particular code were likely not copied into your new project. After the fix is deployed you can then select the remaining codes and copy them over as well…
Sorry, but no batch delete exists up to now.

Hi @daniel! Thanks for looking into this for me.

Hi @Richard! Copying codes of the ethno-poprebel-sexuality-zavtra-extremism project now works. I also made sure no duplicates are created in case it fails again in the future due to an error.
The error that caused the copy operation to fail was caused by this code https://edgeryders.eu/annotator/projects/75/codes/36255 which references a deleted topic. This is valid but something we did not consider.
Unfortunately the order in which codes of a project are processed by the background job does not match the order in which they are displayed in the user interface. So we can’t easily see which codes failed to be copied to your new project. If needed I can help to build a list of codes that failed to be copied. Just let me know.

1 Like

Hi @daniel !

Are you the best person to speak to about producing simple graphs?

I’ve transferred the codes to my various projects and done the coding. I then went to http://graphryder.edgeryders.eu/, selected the edgeryders Platform and ethno-poprebel-sexuality-zavtra-propaganda-2022 as the Project and clicked on Load Project but then I’m stuck. If I click on Codes network, nothing happens.

I just want to produce simple network and ego code graphs like these from a previous project …


… and perhaps combing multiple projects into one graph.

Can you help?

Best wishes

Richard

Hi @Richard , @matthias will take care of this. He had to disable other projects in Graphryder to repair Graphryder for the Treasure project. He will re-enable the Poprebel project today.

1 Like

Thanks, @daniel !

@matthias , can you let me know when the POPREBEL projects have been re-enabled?

Cheers, Richard

Hi @matthias

I was hoping to produce a few graphs for a conference I’ll be speaking at on Friday. Will Graphryder be up and running before then?

Thanks!

Richard

Yes, it will. Have obviously issues getting all my tasks done atm, but this I can promise to make possible.

Thank you so much!

Tried a lot of things but can’t get the Graphryder import script to successfully import from Discourse again. Will try with only your coding projects after sleep. Which ones are essential for your current work?

Hi Matthias

Thanks for looking into this. If it’s too complicated, don’t worry about it. I’ll present without the visualisations. But if you were able to produce a graph for ethno-poprebel-sexuality-zavtra-propaganda-2022 and ideally also ethno-poprebel-sexuality-zavtra-propaganda-2013, that would be great.

Best wishes

Richard

Hello Richard, I managed to get Graphryder to work for the two projects you mentioned. So you can go ahead and create your visualizations today.

Direct links:

The Graphryder software has various bugs / growing pains due to the increasing data amounts that we will have to fix soon (and of which I have taken note). But for your use case, it works reliably for me right now:

Not sure if it will all continue to work after the automatic daily data syncing, the next scheduled for 2024-04-12 4:00 CEST. Also, if you get an error “node … not found in graph”, try reloading the page in the browser, as this seems to be an occasional issue from an incomplete download of data.

1 Like