Wow - @J_Noga - it took me awhile to get through this set of remarks, this treatise - almost a manifesto. So many good points I can hardly keep track.
Let’s say Facebook releases a new algorithm on a billion people, we could introduce certain requirements. I think that’s kind of a basic step. You have some key criteria, you have some transparency, and documentation requirements about how the systems work, what it’s trying to maximize.
how the systems work, what it’s trying to maximize.
This is what they are not going to tell us no matter what. This to them is like the recipe for Coca Cola. But it is the thing we most need to see, because otherwise a more honest title for the business might be RatMaze. You can poke around here and there but ultimately you are going to go where they want you to go.
You spoke of the “self reinforcing loop” up there. I was thinking about this same thing through many of the paragraphs prior to you naming it. You also named part of the problem that contributes to this loop when you said that government doesn’t understand the technology, or words to that effect.
I think part of the reason why this is true is because government workers and tech people are usually two different kinds of people. Neither the government worker or the private sector, usually engineer, person is smarter than the other one. Rather, one - the private sector programmer or engineer tends to be more of a risk taker and the government person tends to be more stability seeking.
I know this might seem like an over simplification, and maybe it is. But what informs me in this comes not so much from a government perspective, but from years i spent managing a digital media project for a large privately owned newspaper and TV company in the San Francisco Bay Area during the height of the dotcom boom of the 90s. “Privately owned” meant that I could not offer stock options for my “startup” (which was a very cutting edge media project for that time), only good salaries and benefits (important in the US where there is no govt healthcare) and stability. So I had to seek out those kinds of people. Almost all the seriously inventive programmers and project people I knew - and I knew a good number of them from the years I spent in online community management - wanted the higher stakes poker tables of silicon valley where their work could bring them the chance of big paydays but also they could be on the street anytime. These are the makers. The stability seekers tend to be more comfortable mediating between the makers and the rest of society in one way or another. That keeps the public happy and wanting the stuff. So tech has a big self reinforcing loop going right there.
Same with cyber security where often the ones who commit the crimes wind up working for the government after they pay their debt to society so to speak and are thus watched all the time anyway. My point is, in addition to all the other complicating factors, the dynamic between these two types of people/workers is part of it too.