Hi, guys. We’re still working with the data on the platform and need help with two things. First, due to contractual obligations we still need to discuss POPREBEL data storage. That’s pretty urgent.
Second, we need help with copying some codes from two existing projects into a new one.
@matthias @alberto
1 Like
Dear @Wojt , @Richard , @alberto, @Jirka_Kocian , and @matthias,
I hope you are all OK.
I am writing, Alberto and Matthias, to ask for your assistance. Wojt, Jirka and I continue working on the Poprebel data base, getting ready to run the second wave of analysis. We have a problem however, as we cannot complete one internal transfer operation (to a new dataset on the platform) and find a way to transfer the corpus from the Edgeryders’ platform to a server at Jitka’s Prague center, where we will continue our work.
Moreover, we need to transfer the data base to the the UCL Research Data Repository, as this is formally promised in the Grant Agreement. I am afraid that the lack of this transfer may be one more obstacle in closing the project and finally getting the last tranche of funding (vitally important for Edgeryders). Our project officer has already inquired about this.
Please let us know how we can connect and resolve these issues. Wojt and Jirka are ready to work with you at your convenience.
All my best,
Jan
1 Like
Hi Jan and Wojt, I have been meaning to ask you for your take on the Polish elections – I guess you have interesting things to say!
To the points raised:
- I have zero capacity until mid-December.
- I am pretty sure that – according to the data management plan for WP2 – the forum data are to be kept on edgeryders.eu, where they are only accessible to people with API keys. The obligation to publish open research data is fulfilled by publication on Zenodo in pseudonymized form, if memory serves also a deliverable (hey, 2K downloads, congrats!).
- I would strongly recommend updating the dataset on Zenodo. Have a look at its structure (follows the Data Package standard). We have a script to export the data as four CSVs (https://github.com/edgeryders/export-ssna-data). But maybe you also added fields etc., in which case also the metadata file should be updated, and that needs to be done manually.
- You can also just use your API access to download the data in raw form and put them in your uni’s server… though, if the point is publishing open datasets, nothing beats Zenodo, really.
1 Like
Many thanks @alberto. Some technicalities you list are above my pay grade, so I will leave the details to @Wojt and @Jirka_Kocian. I just wanted to facilitate a conversation and I understand that they can count on your help no earlier than mid-December, right? The data storage explanation sounds great to me! We need to copy the data, without loosing information (ie codes) to Jirka’s server in Prague, so we can continue.
You may be interested in our plan. We will: (1) finish the codebook (make sure that even the “minor” codes are carefully reviewed), (2) double-check all codes in the existing corpus, (3) generate new visuals, (4) produce the next generation of interpretations, (5) collect more data (waiting for the results of another grant), and (6) repeat the whole process with new data. You will be interested in our research question: Is people’s thinking changed when right-wing populists are out of power? Is there any difference? If so, how can we characterize it?
The Polish elections: HUGE! Although that clown, the President, just announced that he will ask Kaczynski’s people to form the government. They will fail, but Kaczynski wants to use the delay to concoct something. God knows what.
Ciao,
J
1 Like
This is, indeed, a highly interesting question!
Correct!
Dear @alberto and dear @matthias ,
hope you are well. @Jirka_Kocian, @Wojt and I are getting ready to start working on the corpus again and need you help. Here is what we are kindly asking you to help us with:
We need to copy the codes and annotations from four projects (tags):
- ethno-rebelpop-polska-interviews
ethno-rebelpop-polska-ukraine - ethno-rebelpop-czech-ukraine-interviews
- ethno-rebelpop-czech-interviews
Into one new project: - we may call it - ethno-rebelpop-dataset-B
To do this we can either copy them from the backend side (much quicker) or do it manually.
We have a few people who could do it manually, but they’d need privileged access to the forum and they should be able to access the open ethnographer module.
Needless to say, we would very much prefer the former option - doing it from the backend side.
Hope you can help. Please let us know at your earliest convenience.
Yours,
Jan
To be honest, I am not completely clear of how the new project functionality work. I do not see a “duplicate project” or a “merge project” functionality. Section 6 of the manual covers the creation of a new project, but I am not completely sure it’s up to date with the roll out of the projects functionality (@daniel , is it?).
To get access, researchers need to be members of the annotator group of Edgeryders users. I have now made @Wojt an owner of that group. That means that he can assign users if his choice to to that group (of course they need to create accounts first).
2 Likes
Yes, the manual will need to be updated, it does not yet consider the changes we made to the projects. @matthias will updated the manual.
To copy codes between projects is not yet possible with the backend. We already have an issue for it, but it’s not yet done. I will copy the codes as described by @Jan. I’ll implement the copy-codes feature in the next days or copy them over by means of a script. Either way should be done in a few days.
3 Likes
Many thanks @daniel ! I have written to @Wojt to get in touch with you - he understands best what we are trying to achieve. @Jirka_Kocian: please chime in if you have anything usful for Daniel. And many thanks @alberto (BTW: Ilaria and Richard had a meeting today and are working on the last issues raised by the Commission re our final report. Ilaria will be asking for some additonal information from Marina). All my best, everybody!
This is now done. All codes and annotations of the listed projects have been copied into the new ethno-rebelpop-dataset-B project.
We implemented a “Copy to project” feature which allows to copy large amounts of codes into another project. It is still hidden from the UI as we need to run it as a background process to prevent timeouts.
Until it is available in the UI we can still run the command on the console (which prevents timeouts). So if further codes need to be copied just let us know.
We will continue to work on this in a few days - so should be ready soon.
2 Likes
Thank you!
1 Like
Dear All, particularly @daniel: we are very grateful for all your work. MANY THANKS! We are now ready to begin the next phase of our study. @alberto: I hope we will be able to catch up at some point. @Jirka_Kocian, @Wojt and I will (1) finish the final revisions of the codebook, (2) review all the coding done so far, (3) code the remaining interviews, and (4) start new interviews, code them, etc. If we have any questions or news to share, we will ;post them in this thread.
With warm regards,
Jan
1 Like
I hope you can help.
I have followed the above instructions and copied the codes from one of my projects (ethno-poprebel-sexuality-zavtra-extremism) to another (ethno-poprebel-sexuality-zavtra-propaganda-2022). However, in the destination project, each code is replicated nine times. See below:
Have I done something wrong? Also, is it possible to delete codes en masse or do they have to be deleted individually?
Best wishes
Richard
Hello @Richard, just encountered a similar behaviour by copying the codes of ethno-poprebel-sexuality-zavtra-extremism into a test project. It turns out that one of the codes can’t be copied and crashes the background job that executes the copy operation. Background jobs are automatically retried when an error is encountered until it succeeds or a retry limit is reached. This is the standard behaviour and useful when e.g. emails are sent but clearly not useful in this case.
I’ll change the background job that retries are not attempted which will prevent duplicates from being created.
I’ll also figure out why the copy operation fails. In my test it stopped after code 265 of 494 codes in total. I’ll let you know when I figured out which code it is as codes after this particular code were likely not copied into your new project. After the fix is deployed you can then select the remaining codes and copy them over as well…
Sorry, but no batch delete exists up to now.
Hi @Richard! Copying codes of the ethno-poprebel-sexuality-zavtra-extremism project now works. I also made sure no duplicates are created in case it fails again in the future due to an error.
The error that caused the copy operation to fail was caused by this code https://edgeryders.eu/annotator/projects/75/codes/36255 which references a deleted topic. This is valid but something we did not consider.
Unfortunately the order in which codes of a project are processed by the background job does not match the order in which they are displayed in the user interface. So we can’t easily see which codes failed to be copied to your new project. If needed I can help to build a list of codes that failed to be copied. Just let me know.
1 Like
Hi @daniel !
Are you the best person to speak to about producing simple graphs?
I’ve transferred the codes to my various projects and done the coding. I then went to http://graphryder.edgeryders.eu/, selected the edgeryders Platform and ethno-poprebel-sexuality-zavtra-propaganda-2022 as the Project and clicked on Load Project but then I’m stuck. If I click on Codes network, nothing happens.
I just want to produce simple network and ego code graphs like these from a previous project …
… and perhaps combing multiple projects into one graph.
Can you help?
Best wishes
Richard