Qualitative research, knowledge and discovery: a discussion on the epistemological strengths and limits of ethnography and SSNA

How do we know what we know? When doing research (or anything, really) you need to be able to tell knowledge and facts from mental constructs that are not valid knowledge. Epistemology, a branch of philosophy, addresses this problem.

In some disciplines (economics/social science are notorious) the idea is this. You have a gold standard method, which is based on sample statistics. Given a known population, you draw a random sample of sufficient size from it, and then rely on mathematical results that ensure that the sample is representative of the population as a whole. What you know about the sample, you can (mostly) extend to the population.

In practice, doing this is super difficult and expensive for some problems: the population is not known with certainty; you extract a random sample, but not enough people in the sample return your questionnaire, and so on. So, people take shortcuts. For example, they use oxymoronic “non-representative samples”, where respondents to a questionnaire are not drawn randomly from a known population, but self-select to answer an online questionnaire. The Loving More Survey on ENM is one such example. Notice that this is basically the streetlight effect, and that its tradition in the social sciences – though always problematic – is at least 100 years old.

Every time this happens, scholars are “stretching” the method. Some of these shortcuts (example: non-representative samples, processed via nonparametric statistical methods) are deemed to be acceptable: papers are published, people learn.

Ethnography, too, has a gold standard. For an accessible explanation, see this presentation by @amelia. In Edgeryders, we do not operate according to its letter, though we do try to adhere to its spirit. In this discussion, we do a “stress test” of SSNA. Through it, we hope to get a better idea of how far we can stretch our conclusions, and how we can improve our methods so as to make them more robust. The discussion will inform the first draft of section 2 of the SSNA white paper.

Main discussants:

  • @amelia, Edgeryders’ lead ethnographer, affiliated with U Oxford.
  • @markomanka, affiliated with ScImpulse Foundation and CERN, who brings a rich perspective on epistemology spanning from clinical medicine to statistics.

The meeting happens on Zoom, at this link. No registration is necessary. Everyone welcome. Recommended for @annotator, and those in @company who feel invested in research activities.

8 Likes

This sounds like such a fascinating discussion. Unfortunately, I will be in dreamland at this time but would love to hear a recording and partake in future explorations.

2 Likes

Homework for me: track down a publication that follows up on David Lane’s work on ontological uncertainty to share, read and discuss between us.

I have been mostly unsuccessful so far in this task. Lane’s paper is quoted in about 250 other papers, including the authors’ own logical follow-up on the Journal of evolutionary economics, which, however, goes off in the direction of zooming in on innovation. There is clearly a Brian Arthur/Santa Fe Institute influence here.

I am going to go out on a limb here, and propose a structure for section 2 of the white paper, based in part on Foresight, Complexity and Strategy itself (and I do recommend reading it), and in part on this discussion. What strings them together is the idea of interpretation.

What is SSNA good for?

I propose that the main move of section 2 is to delimit the field in a positive, rather than negative, way. We do not claim that SSNA is a method of knowing in general: rather, we illustrate the types of inquiry where SSNA is valid.

It would look something like this:

SSNA is designed to yield insights about the intersubjective cognition of groups of humans, and about the structure of the relationships driving it.

Breaking it down:

  1. SSNA is a method for the social sciences, and has no claim to extend to the natural sciences.
  2. SSNA returns a set of interpretations, not of facts. A typical result would say things like “solarpunk activists think global warming of 2°C or more is very likely, and cities should be ruggedized to withstand it”. This has nothing to say on the objective likelihood of Earth warming by 2°C or more.
  3. Such interpretations are intersubjective, not individual. This is not only true in the trivial sense of aggregating interpretations, like surveys do (as in “43% of Swedes aged 18 to 25 think that planet warming by 2°C or more is ‘likely’ or ‘very likely’”), but also in the sense that people in a SSNA study are typically observed negotiating interpretation and meaning, trying to convince each other, etc.

This means that SSNA is well suited to give rich accounts of negotiated intersubjectivity. This is because intersubjective interpretation emerges from interactions, which SSNA keeps track of (in a social interaction network) and observes as they happen.

This point is best understood in the context of a study of innovation. In Foresight, Lane and Maxfield propose innovation happens in what they call generative relationships. A relationship has generative potential when when the people in it are similar enough that they can understand each other, but different enough that they can induce “cognitive shocks” in each other. These cognitive shocks will set them on a path that leads to changing one or more attributions of key agents and artifacts in the space they both share. In simpler words, their interaction changes the way they see the world. This change, in terms, will prompt a change in what they do: an innovation. In SSNA we can observe many relationships, some of which will be generative. The generative ones should stand out by the way they induce non-intuitive connections (via co-occurrence) between apparently unrelated concepts. Additionally, SSNA observes relationships not as stand-alone dyads, but in networks of social interaction. In the tradition of social network analysis, this carry additional information on the way information spreads, with individual informants boosting on dampening each others’ signals, acting (or not) as hubs and so on.

What world is SSNA good at describing?

It might make sense to include in section 2 some ontological considerations. How good a research method is depends on the assumptions you are willing to make on what the world is fundamentally like. For example, reductionism is good if you believe (1) nature is decomposable into some kind of fundamental unit (atoms, monads, quarks…) and (2) mechanics at the scale of the fundamental units can explain any phenomenon at any scale. Lane calls such a story about the fundamental properties of the world an account. My example above illustrates a reductionist account, or a clockwork universe account, applicable perhaps to 17th century physics.

An account that lends itself well to SSNA has the following characteristics:

  • The worlds that scientists study consist of a flux of energy, matter and information. The flux is generated by transformations, through which the patterns constructed from energy, matter and information change. These transformations result from interactions among these patterns (Lane et al 2009).
  • Information is not merely signaling (like in biological systems), but negotiation.
  • Actions taken by agents is informed by what they believe. A change in belief is likely to determine a change in action.
  • Belief itself forms in the context of relationships with other agents, where attributions about other agents and artifacts is negotiated. For example, many agents are currently negotiating the attributes of the artifact “blockchain”. Is it a vector of liberation from the oppression of centralized institutions? Is it a scam? An overhyped database architecture, perhaps?

This account was developed to capture innovation from a complex systems point of view. Arguably, SSNA is especially well suited to exploring innovation.

How does SSNA differ epistemologically from vanilla ethnography?

So, in essence, SSNA works by mixing interpretive processes and mathematical processes, according to the following sequence:

  1. Individual contributions are coded. In Lane’s language, when coding, the ethnographer makes an inference on the informant’s attributions (“she must mean X”).
  2. These inferences on attributions are aggregated in network form. The network is then manipulated by deploying mathematical techniques, such as those for network reduction.
  3. The resulting network is itself interpreted. In this case, interpretation is inference on the studied group’s collective attributions. The collective dimension of attributions is not shown as agreement or disagreement, but as negotiation between different interpretations, in the context of relationships.

In some cases, inference on collective attributions may lead the ethnographers to revise the coding of the individual contributions. This, of course, changes the data (not the contributions themselves, but their interpretation, expressed as coding), and therefore restarts the cycle.

1 is just normal ethnographic coding, with nothing to set SSNA apart from traditional ethnographic research. 2 is just the application of mathematical transformations, with no important epistemological implications. 3, on the other hand, is an interpretive move unique to SSNA. However, I would argue it does not add a new interpretive layer. Vanilla ethnography also implies a phase where individual contributions are brought together, and re-read the ones in the light of the others. Atlas.Ti quotation manager ("show me all snippets coded with global warming") does. The difference is that SSNA does this through a network form. Since the steps to build the network are formalized and repeatable, I would argue that interpretation in SSNA is, if anything, better accounted for than in vanilla ethnography. The researcher can make statements concerning the number of co-occurrences between two codes, their neighborhood, and so on, and all of these statements are verifiable.

2 Likes

@alberto As soon as I have a few days, I will engage with this thread. I love it. That is a conversation that will give us a great push. But for know I MUST finish our extension request (very tedious). I will try to be helpful before tomorrow in the other threat.

1 Like