Heard back from Field Methods— we got a revise and resubmit, which I think is great news. The comments are substantive and will be helpful for us in situating this work across fields. Seems like mostly we need to engage with lit on semantic network analysis and the lit on ethnography + network science in CSCW/HCI. I’ve gone through and bolded where I think our main focuses need to be, with comments added in italics. I’m going to go through and initial the points I think fall under my purview and I think it makes sense if everyone else does the same so we have a clear division of labour going forward. See reviewer comments below @alberto @melancon @jason_vallet @markomanka :
Reviewers found merit in your article, as did we, and we will accept your paper if you can make appropriate revisions. See reviewers’ comments, below, for guidance on what is needed. Remember that FIELD METHODS focuses on empirical tests of new methods for collecting, analyzing, and presenting data on human thought and human behavior and on new uses for existing methods. The data can be qualitative or quantitative as can the methods for analysis and presentation, but articles for FM should advance a method rather than simply report on the application of a method.
We send revisions and the author’s response to the reviews back to the original reviewers. Of course, you need not agree with all suggestions and criticisms of reviewers, but when you send in your revised paper, please also send a message addressed to each reviewer stating how you responded in your revised paper to each suggestion and criticism, or why you chose not to. This is a vital part of the review process.
The deadline for revised papers is 4 months. If we don’t receive your revised paper, and a reply to each of the reviewers, within 4 months, your paper will no longer be under consideration for publication in FIELD METHODS.
We look forward to seeing your revised paper.
With regards,
H. Russell Bernard, editor
Amber Wutich, associate editor
FIELD METHODS
fmx.sagepub.com
reviewer #1
Semantic Social Networks: A Mixed-Methods Approach to Digital Ethnography (FM6-25-18) proposes a mixed-methods approach to doing ethnographic research in a digital environment. It treats online conversations as human communities that ethnographers can engage with as they would in traditional fieldwork. Then, it represents those conversations as codes developed by researchers. Next, it applies the tools of network analysis, which it labels semantic social networks, and uses these methods to analyse these ethnographic data.
Overall, I believe that this manuscript is worth publishing in Field Methods. However, some revisions are required. I have two major criticisms and some minor suggestions. First, I strongly recommend that the authors review the social science literature on semantic network analysis, and place ethnographic research in that context. There are numerous publications by Barnett, Doerfel, Danowski, Carley, Diesner, Corman, Jiang, and many others. For a good entry into the literature see:
Jiang, K., Barnett, G.A., & Taylor, L. D., (2016). News framing in an international context: A semantic network analysis, International Journal of Communication, 10, 3710–3736. AH comment: I think we should all give this a read given that it comes up multiple times in the reviewer comments
GM This raises the second major criticism. How does the proposed method differ from the more widely employed semantic network analysis? AH comment: this is the main question it seems that the reviewers need us to answer clearly in a more robust literature review. Obviously, the ethnographer is conducting a network analysis of codes. But where do the codes come from? Are they derived from the keywords in the text, from social theory or the researcher’s intuition? Why not go directly from the words, i.e., do a network analysis of the keywords (those that are most frequent or that the research deems important)? And, what are the advantages of doing the network analysis from the codes rather than directly from the text itself? AH DONE
AH, with a check from AC Page 11. In the OpenCare corpus that was used for an example, the authors report 1,248 nodes. Are there 1,248 different codes? Or over 1,200 concepts that make up the codes? If it is the former, what is the utility of creating all these codes? Clearly, there is little parsimony in doing so, and it would seem to be a hindrance in describing the conversations in the text and the construction of elegant social theory.
AH comment: explain the first-pass coding and talk about how the codes collapsed over time w help from @alberto
AH DONE Page 12. The sentence, “The semantic social network gives both ethnographers who have spent extended time with the primary data and researchers without ability to sift through that data new insights”. Is unclear.
AH DONE Page 17. e ort is spelled wrong. Did you mean effort?
Page 18. The first implication of the method for ethnography is very important. I agree with you wholeheartedly.
AH DONE Page 18, pro-pose should be propose.
Page 19. Is the section on Open data and large-scale collaboration in ethnography really necessary? I’m not sure it fits into the focus of this manuscript, which is in the method of social semantic networks.
AC Page 20. In addition, I recommend that the authors discuss generalizability and the ability to perform cross-cultural comparisons of the networks of codes?
MM Page 21. Under future improvements, I would like more detail on plans to Weight the contributions (and consequently annotations) by a “reliability score” derived by applying social theory on the social network topology.
Overall, the manuscript makes a significant contribution to research methods in cultural anthropology by introducing semantic networks to the community of ethnographic researchers and should be published with the revisions suggested.
reviewer #2
This manuscript proposes semantic social networks as a promising avenue for conducting ethnographic research on online domains such as discussion forums. The topic itself is timely and of great importance considering the ever-increasing data produced online as well as the profound changes in the human and social interactions. While I appreciate your effort in taking an interdisciplinary approach based on network science and qualitative analysis of texts, I have some concerns which, I feel, would require a substantial amount of work.
Below are the major issues that I would encourage you to consider in further developing this draft.
My main concern with this paper involves its problematization of the existing methodological approaches concerning digital ethnography and semantic social networks and, consequently, its contribution. More specifically, the framing of the semantic social networks as a mixed-methods approach to advance digital ethnography is interesting, but it appears poorly grounded in a theoretical sense (see, for example, recent papers on digital ethnography by Beaulieu (2004) and Murthy (2008) and GM reviews on semantic social networks by Gloor and Diesner (2014) and Thovex, Trichet, and LeGrand (2014)). AH comment: These are existing literatures I am not very familiar with as they come from CSCW/HCI, so I’ll check them out and create a short lit review. @alberto if I focus on the digital ethnography literature (loosely defined, because anthropologists would not consider this ‘ethnography’, but I think the reviewers are right in that we need to review it or at least address it) can you take the semantic network literature mentioned above and below? AH DONE What I found promising in your paper is the potential of semantic social networks to extend ethnographic research by focusing on the collective level and exploring the interactions across participants which could be further emphasized.
I admire your effort in collecting such an impressive dataset in relation to the OpenCare project, but the current draft would require a more detailed description of the data that you have used for the purposes of this paper. In addition, I found the use of ‘primary’ and ‘secondary’ data quite confusing and, oftentimes, misleading. Based on my reading of your current draft, the primary and secondary data as used here refer to the different layers of the analysis. To be more precise, the primary data would refer to the data level of analysis and the secondary to the semantic or conceptual level of analysis (see also Thovex, Trichet, and LeGrand (2014)).
AH DONE I found that the section ‘An application: The OpenCare data’ could be improved a lot more. First, I would encourage you to provide more information with regards the research question and research design AH comment: I can frame a clear research question in ethnographic terms based upon the OC question ''what do communities do when existing systems have failed them," but I’m wondering if they want something more concrete with regards to the network science? I think this could come down to structural and language adjustment of the paper, since I feel our question is clear (and in ethnography, always emergent) which you report here for a better contextualization of the application.
Second, I would focus more on the structural features of the social network that you have developed which could support your claims about ‘clear core-periphery structure’. In your manuscript, you make use of the Louvain modularity but this should be explained and justified a bit more. I would also encourage you to explore other measures as well such as degree centrality and cluster co-efficient (see Carpenter and Jiang (2012) for a review). Finally, what I found as a gap in methodology is how the social network and semantic network are linked together in your approach.
The information regarding the semantic social network, i.e. the co-occurrence network needs improvement in presentation both in text and figures. More precisely, the visual information displayed in Figure 3 does not correspond to the textual one. In order to improve the clarity of the presentation as well as the robustness of your study, please review the following:
AC DONE It is not clear whether the co-occurrence network map is filtered for k>=6 as mentioned in Figure 3 on page 13 or for k>=5 as indicated on page 12 and page 11 (“with edges with number of co-occurrences k<5 filtered out”)
Which is the correct one?
AC DONE Relating to the previous point, it is not very clear why you have selected the specific number of co-occurrences. A more elaborate description of the generation of the semantic social network and the methodological choices at this stage would be fruitful.
AC DONE Is the network map a directed or undirected graph? While on page 10 you mention that it is undirected, the explanation on parenthesis indicates the opposite. Please report the right one.
AC DONE Please report the right colors of the respective nodes as displayed in the network map. For example, it is mentioned in the text that the node ‘legality’ is illustrated with the color blue while it is actually green in Figure 3. This might not be a major problem per se but adding the poor quality of the figure’s image creates certain ambiguities concerning the thoughtfulness of your research design.
AH + others I am curious, and suspect other readers will be, as well, as to what the nodes on Figure 3 (e.g. climate change, death, cultural difference) which are not connected with the main part of the network map represent. Please add an explanation of these nodes and what they mean in relation to the big picture of the co-occurrence network map.
AC and MM DONE but maybe @markomanka wants to have a look? While you provide some interesting implications of your study concerning policy-making, these seem to be more like interpretations that one could make just by looking at the co-occurrence network map. In relation to that, I would encourage you to focus more on the statistical results of the semantic social network analysis which would allow you to consider the most prominent codes as well as include some excerpts of them providing supporting evidence.
AH (+ others) The section on ‘ethnographic coding as compression’ seems like a promising avenue for further developing this methodological approach. My question, however, is how does it differ from the coding and meaning saturation as performed in a traditional thematic analysis?
Overall, I think that semantic social network analysis is an important and interesting approach which has the potential to contribute and advance the qualitative analysis of long texts and/or social media posts from a large number of informants. I hope that these comments will help you to improve this study. Best of luck!
References
Beaulieu, A. (2004). Mediating Ethnography: Objectivity and the Making of Ethnographies of the Internet. Social Epistemology, Vol. 18, No. 2–3, pp. 139–163.
Carpenter, M. A., Li, M., & Jiang, H. (2012). Social network research in organizational contexts: A systematic review of methodological issues and choices. Journal of Management, 38(4), 1328-1361.
Gloor P., Diesner J. (2014) Semantic Social Networks. In: Alhajj R., Rokne J. (eds) Encyclopedia of Social Network Analysis and Mining. Springer, New York, NY
Murthy, D. (2008). Digital Ethnography: An Examination of the Use of New Technologies for Social Research. Sociology, Vol. 42, Iss. 5, pp. 837-855.
Thovex C., Trichet F., LeGrand B. (2014) Semantic Social Networks Analysis. In: Alhajj R., Rokne J. (eds) Encyclopedia of Social Network Analysis and Mining. Springer, New York, NY
reviewer #3
The main contribution of the semantic social network approach described seems to be in providing an association map of key concepts, enhanced by a visualization technique. Though this paper’s described technique has potential, the contribution it occluded by the fact that the authors do not engage the research of a now substantial body of mixed methods ethnographers, who aim to connect qualitative ethnography and quantitative analysis, including some work that does so utilizing network perspectives (Gravlee, Johnson, McCarty) and is specifically aimed at collaborative digital ethnography (Dengah and Snodgrass).
Some questions:
Though network perspectives are used, the focus is on associations between edges, rather than on network structures, such as the many betweenness and centrality metrics. AC This seems like a missed opportunity, and makes me question the argued greater utility of a network approach compared to other association metrics and data reduction techniques. GM I suggest the authors consult work by mixed methods ethnographers such as Gravlee, Johnson, and McCarty for ideas on how to expand this network analysis to include structural metrics.
AH DONE The argument is made that this approach is superior to an iterative approach, where one moves, for example, from ethnography to survey. The authors suggest that this provides for more openness (p. 4) and less framing bias (p. 19). I’m not convinced by this. Other iterative approaches (such as work by Dressler) provide for flexibility. And the biases inherent in coding data such as this would seem to at least equal other kinds of survey response biases. But the bigger point is that without a serious engagement with this other work, it’s difficult to evaluate these claims. The engagement instead of work by purely qualitative ethnographers of life online such as Boellstorff and others seems ill-advised.
AH DONE Even closer to the point, mixed methods ethnographic, survey, and social network research by Dengah, Snodgrass, and their lab on virtual worlds directly addresses the issues grappled with in this ms. There are many examples, but two particularly relevant pieces are:
Dengah, Snodgrass, et al. 2018. “The Social Networks and Distinctive Experiences of Intensively Involved Online Gamers: A Novel Mixed Methods Approach.” Computers in Human Behavior.
Snodgrass, Dengah, et al. 2017. “Online Gaming Involvement and Its Positive and Negative Consequences: A Cognitive Anthropological ‘Cultural Consensus’ Approach to Psychiatric Measurement and Assessment.” Computers in Human Behavior.
AH DONE The approach seems a lot like the semantic network approach Bernard, Wutich, and Ryan describe in their book (Analyzing Qualitative Data), grounded theory too (also discussed there), neither of which are referenced. Indeed, I’d say that the techniques share a lot in common with semantic concept maps and visualizations, which can be readily produced in programs such as Maxqda. Again, for the authors’ contribution to come through, the field’s current perspectives and tools need to be engaged.
AH + AC DONE The closing pages of the manuscript (pp. 17-20) speculate on issues such as the open science movement. Many of the points seem to apply to most of the sources I cite above, rather than specifically to the technique that is the focus of the manuscript. I’d rather see there a more sustained discussion of this technique’s advantages compared to some of the other approaches I cite above, with perhaps special focus on the iterative ethnography > survey (> sometimes back to ethnography) techniques I reference.
reviewer #4
The authors present an applied and exploratory/ inductive research project where they constructed social networks in which nodes (participants in an online community) are linked through codes that ethnographers assigned to the conversation between pairs of agents.
The papers has several strengths:
The paper translates terms and procedures from social network analysis and text analysis into ethnographic terminology and processes. This work can help ethnographers to adopt research methods from network analysis and text analysis for work in their discipline, thus bringing gaps between disciplines and helping to promote interdisciplinary work.
The authors are doing an excellent job of demonstrating several meaningful applications of the constructed two-mode networks for research questions and scenarios common in ethnography. This work shows the usefulness of using semantic networks analysis and text analysis in ethnography.
The paper can be improved by considering the following comments:
Methods:
AH DONE The coding of conversations is interesting. I recommend providing more details on how that work was done. Did some prior theory about health related interactions or online support groups (see work by Moira Burke) guide the identification of codes? How long did it take? What do costs in terms of time and coders mean for scalability?
Background:
AH DONE There is a large body of literature on using ethnographic methods for studying online communities, and conducting qualitative coding of text data generated in online communities. A good starting point are the proceedings from CSCW (archive at https://dl.acm.org/event.cfm?id=RE169) and CHI (archive at CHI – ACM SIGCHI). The presented work is closely related to this literature.
See also “Analyzing Qualitative Data: Systematic Approaches" Second Edition, by H. Russell Bernard, Amber Y. Wutich and Gery W. Ryan that systematically introduced the methods used in this paper
Here are some additional references to basic work in semantic networks and a more recent application of using semantic networks to capture culture:
Shapiro, S. (1971). A net structure for semantic information storage, deduction and retrieval. Paper presented at the Second International Joint Conference on Artificial Intelligence.
Woods, W. (1975). What’s in a link: Foundations for semantic networks. In D. Bobrow & A. Collins (Eds.), Representation and Understanding: Studies in Cognitive Science (pp. 35-82). New York, NY: Academic Press.
Shapiro, S. (1977). Representing and locating deduction rules in a semantic network. ACM SIGART Bulletin, 63, 14-18.
Allen, J., & Frisch, A. (1982). What’s in a semantic network? Paper presented at the 20th annual meeting on Association for Computational Linguistics Toronto, Ontario, Canada.
T. Buzan. (1984). Make the Most of Your Mind. New York, NY: Simon and Schuster.
Sowa, J. (1992). Semantic Networks. In S. C. Shapiro (Ed.), Encyclopedia of Artificial Intelligence (2nd ed.). New York, NY, USA: Wiley and Sons.
Hartley, R., & Barnden, J. (1997). Semantic networks: visualizations of knowledge. Trends in Cognitive Sciences, 1(5), 169-175.
T. Van Holt, Johnson, J. C., Carley, K. M., Brinkley, J., & Diesner, J. (2013). Rapid ethnographic assessment for cultural mapping. Poetics, 41(4), 366-383.
I cannot find references to this related prior work in the presented paper.
Putting the paper in the context of this prior work, it is not clear to me what novel or unique contribution the presented paper makes.
Terminology:
AH DONE Define “social research methods” → methods from social science? Methods that study social phenomena?
AC DONE but I doubt the reviewer will like the answer. The answer is still correct, though! Define “open data”. I am uncertain as to whether the studied form is an instance of open data.
AC DONE Define open standards: are those standardized data formats for which documentation is publicly available?
AC HUH? We say “approaches”, not “methods” Pg 3: Methods for mining data → that’s not a method, but a data collection technique
AC Section 2 Define “Semantic social networks”, for example:
In R Alhajj, J Rokne (Eds.), Encyclopedia of social network analysis and mining Semantic Social Networks, 2018, you can find entries on “Semantic Social Networks”, “Semantic Social Networks Analysis”, and “Combining online social networks with text analysis”, which provide definitions and citations of related work
C. Roth, & Cointet, J. (2010). Social and semantic coevolution in knowledge networks. Social Networks, 32(1), 16-29.
Research ethics:
MM How were the data from the health forum collected? Did participants of the forum give informed consent? Were the terms of service of the platform considered?
Minor points:
MM and AC Precision: define term and provide evidence for statements
Example: pg 3: “a scale large enough for most applications” → what applications? AC added that
Logic: Digital media enable the transformation of codes into structured data. No, applying network analysis achieves that. MM can you help? This is false, since digital media is what makes the primary data digital to begin with. If people were not using blogs, fora and social networking sites this whole thing would be a nonstarter.
AH DONE Generalization: pg 12: “which community members are working on community art and community gardening”. A methodologically more truthful statement would be “which community members are associated with community art and community gardening”. Researchers who study online communities and digital social data sometimes take the footprint for the foot. What we are looking at are association maps that do not easily lend themselves to causal or inferential statements, and that’s totally ok as long as the authors do not overinterpret their data.
