MENA Open Village Coding: Lessons Learned

project-mena-youth-platform

#1

This post is a follow-up to a previous post I wrote here, so if you haven’t read it perhaps check that out first! I’m going to go through the points I made in that post more explicitly as they relate to the Open Village project and to future projects, as well as reflect on the co-occurrence network generated from the Open Village coding.

Let’s start with the co-occurrences network.

It seems to me that we need to implement something of what we have discussed with @melancon on multiple occasions – a way of creating code hierarchies. One of the reasons this graph is somewhat difficult to read is that there are a lot of higher-level codes that don’t carry that much specific meaning (e.g. “obstacles”; “improve community”; “supporting community”). These kinds of codes are definitely useful (think back to the “community-based care” code in Open Care that carried a lot of meaning in its co-occurrence with other codes). But because they are more generic, they are more likely to co-occur with other codes, as they apply to a wider range of concepts. You’ll find obstacles in places as disparate as recycling (to draw from this project) and domestic violence reporting (to think back on other projects). Also, I’m interested in how ‘obstacles’ is productively different than ‘problems’ which appears in the lower left hand part of the graph. (If there is no productive difference, they need to be merged. There is also ‘problems of education’ which is its own code despite there being ‘problems’ and ‘education’ codes).

This is why the ethnographer coding for SSNA has to be careful with these kinds of codes. Even more general codes have to have a precise meaning. So my concern with this particular ontology, for example, is that I can’t see a clear difference in meaning between “supporting community” and “improve community” based upon co-occurrences or in my own mind. As a result I think they need to be merged or clarified. “Community-based care”, for example, went through a few different merge/fork iterations, swallowing other codes that were being used interchangeably with it and forking from codes that were meaningfully different (such as ‘peer-to-peer’, for example, which community members were using in a specific context).

Similarly, how does ‘sharing knowledge’ differ meaningfully from “collaboration”? I can certainly imagine a few scenarios in which they’d be different (there are kinds of collaboration other than sharing knowledge, for example) but would expect more of a code network around to let us know.

I’d love to know how collaboration and obstacles connect, and am confused as to why there aren’t more codes surrounding those two. I can imagine if they co-occur so much that there should be codes co-occurring with the both of them that might illuminate WHAT those obstacles to collaboration are, or how collaboration can overcome obstacles! Those are the code co-occurrences that make SSNA so useful.

There are also codes that don’t have a clear meaning to me in isolation or in their network. What does “authorization” mean? Perhaps it is like our regulation - safety - existing system failure nexus in Open Care-- looking around it, it’s possible that lacking proper authorization (not sure from whom in this case) leads to obstacles, or vice versa?

Let’s crawl out of those weeds for a moment and look elsewhere in the larger picture. I’m interested in the quitting job - job dissatisfaction - passion triangle. I can imagine (and from reading many of the posts) I gather that a lot of people feel a lack of passion in their current employment situation. I also see unemployment, but am interested in why it is unconnected to that nexus.

Similarly, I’m interested in why ‘education’ is unconnected with ‘sharing knowledge’. But education - volunteering - community health could be interesting! How does education relate to community health? Are we talking about mental health or physical health? I’m also having difficulty clicking on nodes to see the conversations beneath to dig deeper, but I will give it another go soon.

As @alberto mentions in this post, the graph does illuminate solutions (and potentially the issues as well: job dissatisfaction, unemployment, potentially authorization, and funding problems (those are two different linked codes)). I definitely get a general sense of problems and solutions from the graph. I also feel a renewed sense of confidence in the method: even in suboptimal coding conditions, higher levels of meaning are still emerging in the social semantic network that allow us to get a big-picture view of a conversation as a whole.

Moving on from that brief discussion of the codes to other lessons learned.

  1. It’s clear to me that we need people with explicit experience in both ethnographic research/writing and qualitative coding/data analysis. A lot goes into ethnographic coding that can be invisiblised, but in order to have high quality outputs there have to be high quality inputs. This graph to me illustrates a lack of underlying rigour in coding practices — merging similar codes, forking codes that have distinct meanings, generating codes that have a clear meaning, and coding thoroughly enough to create maps of co-occurrences. You can view my other post for a discussion of why coding fidelity is fidelity to our community members and is therefore a very important task. In future projects I therefore think it is imperative to take the time to onboard people with explicit experience and who demonstrate a committed attention to iterating in coding practice.

  2. Flakiness test. This process was rife with flakiness issues — we lost a few people early on who had committed, and ended up re-onboarding a person who had originally disappeared. More periodic disappearances occurred after that, making the entire process frustrating. In future projects, we need people who commit to showing up for calls, since we are going to have to communicate frequently across language coding. Frequent and open communication is vital to make sure the coding starts high quality and remains that way throughout so that the emerging graph carries meaning.

  3. Open Codebook. I talked about this in the last post, but in order to ensure rigorous coding I think we need to keep a codebook which allow us to keep our coding decisions rigourously supported and also easily communicable to our teammates.

Added later: I am also realising as I do this that the graph is an excellent way of doing code cleanup in a supervisory capacity (e.g. as the one not doing the coding)! If I had this throughout while supervising someone I could much easier actively advise them on where to look to potentially improve or clarify their codes.


#2

As an input from a bystander, the quality issues with the coding work of newly added team members might be due to a lack of explicit documentation. I’m not saying it’s ok for them to bunk off the team calls, but this is asynchronous, decentralized collaboration after all, which always profits from explicit, written, detailed documentation of what and how to do stuff.

It’s the same thing I am pushing for (in the Collaboration category) re. our company processes.

If you want to start writing about your experience and best practices in the coding work, the Open Ethnographer Manual is a good place. Hopefully it’s still possible to bake this work into the upcoming paid contracts (that’s the realm of @alberto I guess).


#3

In this case, the training materials included the Ethnography for SSNA course which was pretty heavy in the documentation arena— I think this is discussed elsewhere (and should be discussed again) but it’s important that we find a way to get that content on-platform so it becomes part of the OE documentation.


#4

@alberto wanted the flakiness thing clearly documented and while I like to stay positive I have to agree on this one… I’m not sure how, but we have to find a way to get people who are consistently showing up for stuff (digitally speaking) and delivering on time. It’s even more important not to fall off the map exactly because we work remotely. I think for me that’s more important than having people with a background, but having the background would make life a lot easier. As I mention in my other post, above all it requires a commitment to learning new things and making things work as we go — which requires that commitment and reliability.


#5

I have a friend who is a heavy duty programmer. She wrote a lot of the deep code that lets bank ATMs interlink (which is the name of the code she wrote - Interlink. You still see it on ATMs sometimes). Once she told me that the difference between a 50K programmer and a 100K programmer has nothing to do with the quality of the programming. The difference is the 100K programmer delivers on time.

Do we pay decently for this work? And where do the recruits come from?

Back to the weeds:
In American health care authorization means whether or not you can have a procedure without paying retail for it yourself. So, over here it leads to obstacles all the time.

Sharing knowledge can be simple providing a link, no? I don’t see that as collaboration, which I think is an activity that leads to an outcome. Plus collaboration suggests a relationship that is formalised at least a tiny bit. Being part of a discussion in and of itself seems to me to not qualify as collaborating per se.


#6

The whole point of scaling this activity is that it needs to be doable by junior people. Nermine is a master student of ethnography from Egypt. I imagine the rate was adequate for her corner of the world and professional status.

Also, your friend seems quite the cynic. Work ethics is a premium service?


#7

This is a fantastic piece of info. It tells me that we could perhaps add to your course an exercise. That would work by exporting some subset of data from Graphryder, which becomes the dataset for the assignment. The student is invited to make her own observations. This after being exposed to these considerations.


#8

Wait. I see a difference here. “Obstacles” is just a category, not that different from “research question”. “Supporting community” is still a category, but a bit more specific: it indicates the need to be proactive in replenishing your community assets. “Collaboration” is fully semantic: a specific thing that is being talked about.

Or?


#9

The idea was more along the lines of Steve Jobs’ quote “real artists ship.” But I’m not sure how relevant it really is to this discussion. I was really just probing a bit into the flakiness aspect…


#10

I’m fairly certain that’s not what authorisation means in this context — but good evidence that this needs clarifying


#11

Right. These are all higher-level (less specific than, say, ‘recycling’). This is not to say they are on the same level or carry the same meaning!


#12

And I don’t think ‘obstacle’ is like ‘research question’ — RQ was a meta-code (indicating a kind of contribution) whereas obstacle carries semantic meaning.


#13

Do the people you hire have contracts of any kind? In other words, what structure helps mitigate the flakiness factor? When they walk away, I assume they walk away from being paid too.


#14

John! Everybody in Edgeryders always has contracts. Company rule.


#15

Ok, it has one-bit semantics over the “bad/good” space. :slight_smile:


#16

Thanks. Just making sure…