Spaghetti open data: a little thing that feels right

Knowledge Collection Open Government Data
, ,
1

In September 2010, after a happy hour in Rome, people started spontaneously to share links on Italian open data and tools to crunch them with. With a few others, I thought it would be nice to collect these links in one place, a sort of one stop shop for people interested in trasparency not just in theory, but in the practice of extracting information from public data. One thing led to another, and in less than two months Spaghetti Open Data was online. We aggregated 32 databases at launch; not bad when you consider that data.gov, with all the firepower of the Obama administration, started with 47.

It’s only a small thing, but it felt right for various reasons.

  • Firstly, it is a concrete achievement. I have had enough of complaining about the idle government, the backwardness of Italian culture, the financial crisis, bad luck. I have precious little time to spare, and I would like to invest it on projects that pay me back by yielding some kind of result. The Spaghetti Open Data group has put in some work, and in a few weeks it produced something which is actually there, and it works. If you want to build something with Italian open data you can, right now, without having to wait for structural change or a new generation in government. All it took is some voluntary work and 41 euro for hosting, which I paid up very willingly (the italian government eventually came around to putting online dati.gov.it. It took them another year and a lot of handholding, and even now it is far from a thorough job. I like to think seeing citizens seize the initiative put some pressure on the government to take action)
  • Secondly, it is intellectually rigorous. We had to ask ourselves the same questions that I imagine confronted the people in charge of data.gov and data.gov.uk. Are statistic data open data? (Apparently not) Does it make sense for statistical and open data to be collected in the same place? (Apparently it does, so that citizens can correlate the ones with the others) How to organize metadata? (We went for compatibility with CKAN, as in data.gov.uk) we have mapped a possible way for Italian open data, and future legitimate websites of open data have an all-Italian benchmark that they can consider, or even copy.
  • Finally, it is the expression of a small community of a mall number of bloggers, geeks and civil servants (about fifty at the start, about two hundred now) that worked together  towards a common goal, across considerable cultural differences, showing mutual respect along the way. I have also had enough of bashing bureaucrats as stupid or evil. Some are just that, others are wonderful people and great war buddies. Most are reasonably clever, well-meaning people who happen to be very different from me: collaborating requires investing a little time and effort to come to understand each other. It is almost always worth it.
Later, the life of the Spaghetti Open Data crowd revolved around a very well-loved mailing list. We used it to coordinate on several open data-related initiatives, helping and incouraging each other. Among those that I remember with the most fondness:
  • a couple of civil servants  joined forces with hackers from the Open Knowledge Fondation and the Guardian's datablog to release a fantastic dataset on public expenditure in a fully open and interactive form
  • we translated into Italian the OKN's Open Data Manual for free. In two weeks. And during the August holidays, when absolutely nothing gets done in the country.
In the future, I only want to do this sort of thing. I’m done with declarations, petitions and talk. Simply doing is too much fun, even for a daydreamer like me. :-)
2

Empowerment

If you want something done right, do it yourself. [proverb]

I am an avid follower of this proverb. But often, it plays tricks on me. And I regret it.

The first time I heard about Spaghetti open data, I was seduced by its name. This is probably the most original name of all open data bases in the world! All the better, if in this case, Spaghetti open data found its way in the Italian government.

I would not mind having a ‘Maple syrup open data’ in my area. But it takes geeky people to do this, which is not unfortunately really my case. But even if I were a virtuoso geeky extraordinaire, I would not do this, for the following reasons:

I often blow out the answers and provide information to policy makers (or to collaborators) in order to help them solve problems. But hacking in not in my lobbying bible. Why? Because I would be afraid to break the government’s… karma! (Yeah, please don’t laugh…!)

We cannot take the karma or destiny of someone else. No one can take the weight of another person. Therefore, I wouldn’t want to take the weight of the government (which is not a real entity, it is composed of a gathering of human beings).

If the government does not do its job well, or if it is not responsible enough, this should not be the burden of citizens. I consider, however, that this is part of my responsibility, as a citizen, to make sure that the community is well managed. I am responsible for my thoughts, for my life, for my health, for my family, for my neighborhood, for my government, and even for the planet. (Because we are all connected… the oneness concept.) I also consider that it should go without saying, for citizens to co-manage the affairs of the state, in collaboration with the government. This is why I firmly believe in the open government philosophy.

The only thing that we can do, in terms of karma, is to assist another person to overcome his/her own karma by making he/she evolve on a different plane of consciousness, out of a lot of love, selflessness and understanding.

For example, humanitarian aid works better (lasts longer) when, instead of throwing money on a problem, it is taught to local people how to cope by themselves.

I have learned over the years to bite my lips and say ‘no’. With my big heart, not so long ago, I too would have been very tempted to do it myself. But now, I prefer to try to see the degree of awareness of people, and help them by showing them how to generate individual, community and organizational empowerment.

3

Who is “the government” anyway?

Hmmm… I don’t completely follow your reasoning. What is “the government” anyway? Some people think it is simply an agency that We, the people, put together to carry out certain task on our behalf. And what the people giveth the people taketh: if we are better, faster or smarter at some of these tasks ourselves, we can and should relieve the government of those responsibilities.

With open data it seems quite clear that hacktivists and civil society organizations are just way better and faster than government agencies in performing some of the related operations. So… we do them ourselves. With Spaghetti Open Data we had a repository of Italian open datasets one year before the government came around to it, at no cost to the taxpayer. A conclusion might be: “the government” is not the only agency that can serve Us, the people. sometimes we can serve ourselves better than they would. So… let’s do it!

4

“Dades amb Pa amb Tomàquet”

Grazie Alberto, è una bella storia :slight_smile:

Lyne said: “If you want something done right, do it yourself”.

I prefer another proverb: “Better ask for forgiveness than permission,” I have used it several times …

There are some things in your story I want to comment:

1 .- You say that Spaghetti Open Data - yes, it is one of the best names of OpenData services :slight_smile: - is prior to the official italian government portal OpenData. Is the Italian Government did not consider that he had a OpenData portal in Italy was offering open data?, As we say in Spain, does the Italian Government reinvented the wheel?

2 .- I personally love that Spaghetti Open Data is created by civil society itself, is the best way this portal offers useful service to society. But also acknowledge that if the government gets involved much better, especially because it is who has more data and better quality, as well as financing for the portal.

3 .- I do not understand why statistical data can not be opened. In fact, I believe that statistical departments are a major source of data to supply OpenData portals. Another issue is to discuss whether these data are “raw data” or data developed. I personally believe that, in general, are not “raw data” although specific level, and depending on each reuse can be reached as “raw data”, so the difference is not clear… but, is it important?, the important is the data reuse, nothing else.

Congratulations for this initiative!

In Catalonia we should have created “Dades amb <a href=“Pa amb tomàquet - Wikipedia”>Pa amb Tomàquet</a>” portal :slight_smile:

MarcG

5

In the end, it has to be public

Hey Marc! Let me try some answers.

  1. as we launched (2010) we made it clear that we wished SOD to be a placeholder for a proper Italian government OD site, and not a competitor. The government was vaguely aware that SOD existed, but no one dreamed to doubt its right and duty to set up an honest-to-God dati.gov.it website. After the launch of the Italian official site we have been discussing redeploying the website, and some people have prepared a beta.
  2. agreed! SOD wanted to send a message: the Italian civil society wants this stuff so much that we are willing to do it ourselves, and pay for it too. So, dear government, you should consider makingit a part of your mission.
  3. statistical data can and should be open. Italy's ISTAT has adopted CC0 and free download all over the place in 2011. But they are something else: statistical data are collected to be released for informational purposes, that's their primary use. Open government data are collected for other purposes, and then re-used by being opened up and released. By implication, it is harder to get institutions to open up gov data than statistical data.
(I learn all of this stuff in the SOD mailing list, a really great watering hole)
6

What steps would you recommend others follow?

Alberto,

You hit upon a real need with SOD, one that was ahead of it’s time for the government of Italy, would you recommend that people in other countries replicate this until their countries release open data?

What process should they follow to get their hands on this data?

I have the perfect locatoin for them to store, and share, the data from, so the only challenge would be providing a roadmap and encouraging others to follow.

Thoughts?

7

It certainly worked for us!

I definitely recommend the “innovation without permission” approach in this case. Look, less than 18 months down the line the Italian OD landscape is unrecognizable:

  1. ISTAT has become a convert, and a very important player at the core of the Italian state. They launched a data warehouse with a lot of widget-embeddable stuff; licensed everything on CC0; waived fees for downloading datasets; organized nationwide OD events in very prestigious contexts, like Conferenza Statistica Nazionale; and are planning a Data Journalism School.
  2. Many regions and cities have now OD sites and policies in place
  3. The Italian government (the old one, not the new technocratic one) ended up following
  4. We built a super-active mailing list which welcomes civil servants and hackers in a mutually respectful atmosphere, with a healthy hacker ethics and the technically most competent people on semantic web stuff in the country. So we learn not only that OD are good, but how to map ontologies onto datasets using RDF etc. etc: the nuts and bolts of it. The ML has been discovered by Ton Zijlstra, the editor of the European Commission's Public Sector Information website, and this - as well as your own work at Govinthelab - has given the Italian movement some international exposure. We are only less than 200, but man, do we work hard!
None of this except the last is due only to the civil society's activism, but I think we can claim some of the credit for the change of status of OD in Italy, from obscure, fringe nerdy stuff in 2010 into political must-do in late 2011. Notice that the brand of activism we are talking about uses bold moves (raw data now) but very soft rhetoric (we only want to help our institutions, not disrupt them). So we don't waste time dealing with trolls and controversy, and just act. 

Where to find the data? Data are around - they are just often not open. We started by opening further up what was already there: for example converting PDFs off gov websites into machine-readable stuff; or Excel-and-no-open license datasets onto proper 5 stars material, indexed in CKAN and all, like in the case of the spending data I mentioned in the mission reports. Another line of work is to try and persuade your city or your region to release some datasets.

Example: right now a couple of hackers in SOD are engineering a scraping of a new Ministry of Education website that contains great data about schools. The Minister has promised to open them up, but they have not yet gotten down to it. So, we went out and tweeted: “hey, Minister, we are going to scrape your data and make the interfaces better! You don’t mind, do you”? You see the pattern. It’s collaboration, but kind of assertive. The subtext is “we are the people, we want to be your friends but we also think we have a right to accessing these data. By the way, we have the technical skills to get and re-release them. So we’ll do it now, you can sort out the legals later.”

You would think institutions might react badly, but so far they actually liked it a lot. I think what is happening is this: the civil society is emerging as an allied of the innovators within the public sector. They can go to their bosses and say “look, these guys are hellbent on this stuff. Either we move fast or they will move first and leave us looking like idiots. The good news is, we can ask them to help us, and they will! So we can appropriate some poltiical benefit releasing data, and everybody wins”.

Does this make sense from your global perspective?

8

Yes, Exactly!

Innovate without permission is critical!

This is a roadmap for citizen activism to open up government when they lack the vision, the resources, the desire, to do so themselves.

Governments release documents in all kinds of formats, we need to centrally store these and collectively, through citizen volunteers, make them OPEN.

I am going to launch something around this very soon and, with your permission, will use your response above as an example of how citizen activism can open a government evern when they are not yet ready.

An engaged citizenry can accomplish anything.

JohnFounder and CEOGovernment in THe Lab
9

Go ahead

You don’t need my permission for that.

10

Congratulations!

Alberto,

Congratulations!, your comment brings a way to do an Open Government from people, it’s direct, simple, powerful!

In fact, the key is “we are the people, [to Govs]  we want to be your friends but we also think we have a right to accessing these data”.

Don’t ask permission, just do it!

MarcG

11

Statiscal departments are one of the governments departments with more data (and about a lot of themes).

Yes, the main goal of these departaments is to inform people. But, I think is a good idea to have the Statiscal Department of your government as a partner of data openness initiative.

It would be as a example to another departments, so, you could convince another departments with this success.

MarcG

12

Not a department at all!

In Italy ISTAT is a separate body, like EUROSTAT in Europe. It has its own President, a political appointee who is not a member of the government. So actually, no: ISTAT’s commitment to open data cannot leak through to other departments. But ISTAT does play a role in the debate, and it does have influence. This influence, predictably, has increased under the present technocratic government, since a lot of ministers and the Prime Minister himself are economists and are used to data-backed discourse.

13

Technocratic Gov?

Ok,  thanks for information.

Perhaps a technocratic government is a better government in order to open data, isn’t it?

Regards.

14

Yes

Let’s put it bluntly: yes, much better. Because technocrats (1) tend to know what they are talking about and (2) they are politically weak, and radical transparency is one of the few sources of legitimacy they might have. Open gov/open data has made a lot of headway in Italy since we got this government.

15

The best statement I’ve read lately

Well said Alberto:

“With open data it seems quite clear that hacktivists and civil society organizations are just way better and faster than government agencies in performing some of the related operations”