@Jan! I finally got around to reading Schaffer’s paper as you recommended. I have the feeling I shall be quoting it for a long time. @amelia, if you did not know it already, please read it as soon as you can, it has immediate implications for what we do. @sander, you too please, as I would like this to be part of Skunkworks-1’s references.
Schaffer has a very solid case against surveys, even those with open-ended questions. He uses evidence from the Global Barometer Project and his own study to pinpoint the problems that surveys suffer from. He calls them compression, compartmentalization and homogenization (framing bias is not mentioned).
Compression has to do with long- vs. short form responses. It is compounded by problems with interviewers pasting standard phrases to summarize what the surveyed folks are saying. This is already solved by SSNA as it is, because (1) the conversational form in which data are generated has abundant space, and will record whatever key informants write. On top of that, (2) the semantic network visualization allows to immediately see how key concepts connect across the informants’ posts.
Most people that we interviewed juggled multiple standards [for what it is meant by “democracy”]. This woman seems to have two particular standards in mind – freedom of expression and low prices – and applies each of these standards at different moments in the interview (p.10).
I can practically see
freedom of expression connected to
low prices on Graphryder. Later in the paper (p.14):
The more generalized point is that the kind of multi-dimensional thinking evidenced by the fish vendor, shop keeper, and custodian is not likely to be captured in the short-answer, compressed format of the Global Barometer surveys. The informative messiness of how people actually understand demokrasya is replaced in the surveys by concise, single-idea, easy-to-write- down-in-a-few-word answers. Complexity, ambivalence, and interconnectedness are displaced in favor of the short, simple, and telegraphic.
Amen to that.
Compartmentalization arises when you try to mitigate the compression problem by giving people more than one answer to the same question (“What else?”). In that case, what gets lost is the link between the main concepts carried by the different answers. This is also supported in SSNA as it is. In fact, it seems as if this were the main problem that SSNA was designed to address. Schaffer’s complaint is spot on:
[Global Barometer researchers] code each of the individual’s responses as if they were free-floating and disconnected from each other
He goes on to claim that responses are interconnected, and they should be treated as one. SSNA’s conversational medium keeps track of what is said in what context, and so has native support for this kind of analysis.
Homogenization has to do with how you treat responses in different languages. Starting on page 23, the paper takes a deep dive into the nuances of the Tagalog word kalaayan, translated roughly as “freedom”. But the translation is simply too rough, as are those of the words that, in other Filipino languages, are understood by researchers to mean “freedom”. Money quote:
We see, then, that kalayaan has a distinct history that apparently reverberates to this day, a history not shared by its rough equivalents in either English or in other indigenous Philippine languages. Yet survey researchers have homogenized the meanings of all these words. Freedom and kalayaan are treated as twins, while any possible occurrences of kagawasan, wayawaya, kahilwayan, or katalingkasan or other similar terms were hidden by reporting all the results in English or Tagalog.
SSNA has not, so far had native support for this stuff. But with POPREBEL, it will acquire it. Implications for @amelia is to keep those open codebooks lively, and make sure to talk etymology across the team.
All in all, a very very inspiring read. Thanks!