On assessing impact, and what Edgeryders could do in that department

@nadia is just back from Techfestival, and she tells me she had an interesting conversation with an old acquaintance, @btincq (hello again, Benjamin!). Part of it revolved around indicators to measure impact.

I have been hoping that ethnography-at-scale via SSNA could integrate, if not replace, the indicator paradigm. Which, frankly, I do not believe in anymore, after being part of the intellectual effort to “port” cost-benefit analysis to environmental externalities in the 1990s. This was spearheaded by UCL, who created a whole research center around it (I used to hang around it as a grad student), the World Bank’s Global Environmental Facility and others.

Intellectually, experimental psychologists blew us right out of the water. We were trying to get people to assess their own willingness to pay for, say, avoiding the extinction of some type of frog in Madagascar, or lowering the PM10 content by 10%, and they showed us very convincing evidence that we could never trust our results. And anyway, market allocation only reflects social optima under very restrictive assumptions, that are never met in practice.

In terms of impact, CBA had precious little. Environmental taxes and caps-and-trade left the economy running (and polluting) much like before.

Later on, James Scott made a convincing case for why we are so fascinated by scalar indicators. According to him, they are propelled by the modernist ideology that was so successful in underpinning the coalescing of modern states. States discovered they became more powerful if they could superimpose administrative ordering to reality. This worked wonders when it came to “money and men”, levying taxes and conscripting soldiers. But:

  1. It worked wonders for the states. For the people… not so much.
  2. As soon as it touched more complex systems (with the invention of scientific forestry in the late 1800s, in Germany), the systems pushed back (managed forests collapsed due to lack of biodiversity, etc.).

Modern states have created a demand for scalar indicators (GDP, ROI, the modern central bankers’ short-term discount rate etc.), which correspond to scalar “control knobs” under their control (keynesian state deficit, money supply etc.). But, according to Scott, this has more to do with their in-built thirst for administrative ordering than with their drive to appreciate what is really going on in complex systems like the economy or the global environment. In the worst case scenario (Stalin’s collectivization of agriculture), administrative ordering has sought not to describe reality, but to change it, with violence, if needed.

Last year I finalized a rather bleak document, called the Black Briefing, in which I put together these elements into a critical appreciation of government, and the part played by indicators in it. Nadia was curious about it, so here it is.


1 Like

This Ethnography at Scale idea is pretty much what Dave Snowden, known for the cynevin framework has been using in a load of organisations:

I’m not aware of any open source alternatives yet, but I’d be curious if so, as that at least shows what one implementation/incarnation of the ideas looks like.

Yes, I have seen it. Not a fan of the triangle thing, because it (tacitly) rescales all measures. :frowning_face:

If you’re looking for ideas that explore the numerical approach more, then you might find the work done at Delft Univeristy in the Netherlands interesting.


Isabel Ordonez introduced the ideas to me, and while idea is used mainly as a design tool for businesses to understand the full costs of what they make.

The wikipedia is actually pretty good:

I don’t think it captures anywhere near the full impacts of economic activity, but it is at least useful as a design tool, when looking in terms of pollution, carbon emissions and things that already fairly easy to count.

But that of course, that’s a bit caveat.

I simply do not believe it makes any sense to put a dollar value to things like CO2 emissions, disappearance of glaciers, extinctions, or loss of life. So the top part of “costs” is bogus, AFAIC.

Hadn’t received an email from Edgeryders in a while, hello @alberto :slight_smile:

Thanks for sharing the reference and experience. I’ll forward it to Manu who’s a bit more knowledgeable than me about all this (she has an international development / policy background) — but in general I like the idea of SSNA used for impact management

We made a humble compilation of resources and methods currently used for impact management and what their limitations are in the context of a science/technology project (which is our main focus) in Chapter 5 of our report.

I’m a bit on a low-bandwidth at the moment but let’s keep this conversation alive :slight_smile:

1 Like

I never understood why so-called policy makers want everything dumbed down into a single metric that then says how well a policy measure performs. Such as GDP, or tons of CO2(eq) emissions etc…

Because with just a single metric as a target, what politicians do is not systems engineering but policy-making “work” in a feedback loop “until it is enough”. There’s no overall vision, direction, no integration of partial systems etc… The Energiewende (energy system transition) in Germany is a good example for the half-measures and internal contradictions that this leads to.

Politics has to be systems engineering, and if it does not work because too many people are involved and they demand compromises, then the unit of politics has to be broken down into smaller units until it works. No need for indicators then, because the system is designed right from the beginning to not have any negative impact on the environment etc…

Indicators are only needed when dealing with half-measures and compromises. The proper way in engineering is to include a safety margin – means, human civilization must be designed as a net benefit for nature to compensate for the accidents when it won’t be. (For life-critical applications the safety margin is usually 500% – for example, a lifting cable of a crane will have a breaking strength of five times the maximum allowed load on the crane. Dealing with the environment would qualify as a “life-critical application” – on the very long term only of course – but humans do not have any safety margin here …)

In practice, “politics as engineering” could look like my proposed autarky system for cities. In which the cities would be sovereign.

Side node: at second reading it becomes clear that you don’t believe in the indicator paradigm anymore. At first reading though, my initial understanding was that you don’t believe anymore “that ethnography-at-scale via SSNA could integrate […] the indicator paradigm”. I guess the wording could be disambiguated a bit :slight_smile:

1 Like

Making this a reality would be the most revolutionary thing humankind has ever done. And go ahead and call my friends and me ‘flower children’ for believing and wanting this way back when (you would not be wrong), but this is at the core of why so many of us “dropped out.” Because we could see no other way to get there.

1 Like