Earlier this year we have submitted four Horizon 2020 proposals. In July we received the bad news – none of them is getting funded. What went wrong?
Horizon 2020 applications are evaluated by independent experts and the process includes three phases: individual evaluation, consensus group and panel review. More details about it can be found here. Each criterion (Excellence, Impact, Quality & Efficiency of the Implementation) can bring 5 points max., which makes the total of 15 points.
EPICS – Epistemological Insights on Citizen Science (total score 12)
Positive points:
- The project was evaluated as ambitious and progressive with clear research objectives, methodology and work plan, management structures and procedures, complementary partners and very well addressed resource attribution. Direct participation of citizen scientists in delivering the targeted actions is considered innovative as well as the application of participatory design and ethnographical research methods. Inclusion of grassroots initiatives, public and societal engagement are particularly strong. Exploitation goals, data management plan and communications plan are detailed. Outputs are aligned with impacts.
Shortcomings:
- Activities only partially address the topic, limited focus on incentivising career scientists - little is said of plans for the identification of barriers and enablers to their involvement. The targeted knowledge base of existing citizen science projects is not significant and the involvement of policy makers not evident. Gender dimension is only broadly considered (but not how it will be addressed in research terms). Dissemination is not fully detailed nor how results will be exploitable by wider stakeholder audiences. Some minor remarks include potential risks of combining citizens science and ethnography which are not fully considered; the rationale for the attribution of lead roles not in all cases fully described.
EVENTS – Understanding the Evolution of Society and Science through Citizen Science (total score: 7) This was a different type of call, evaluated on only two dimensions. Maximum score is 10, not 15.
Positive points:
- Approach and methodology are evaluated as innovative; objectives consistent with the exploitation strategy; the knowledge gap well investigated with objectives addressing it. The proposal is beyond the state of the art in relation to the ethnographic approach. The use of stakeholder knowledge is clear and the gender dimension precisely specified.
Shortcomings:
- The topic description in the work program is only partially discussed, the objectives may be achievable but not specific or easy to measure. The concepts of trust and accountability can be interpreted in many ways and need to be examined critically. Literature on the analysis and clarification of these concepts were not fully taken into account. The fellowship program lacks evidence; greater representation from ethical experts is needed.
- Little evidence is provided of how participants will yield insights into the reproducibility of citizen ethnography. Certain claims, such as to improve citizen science projects, are difficult to assess. Not enough details on how knowledge base will be significantly advanced beyond the state of the art. It should be more clear how the results will contribute to the expected impacts. Vague statements should be avoided.
Future calls to consider related to citizens science (all opening on 10 December 2019 with deadline 15 April 2020, more details available in the work programme):
SwafS-19-2020: Taking stock and re-examining the role of science communication
SwafS-23-2020: Grounding RRI in society with a focus on citizen science
SwafS-27-2020: Hands-on citizen science and frugal innovation (this one especially suitable to “re-use” the EPICS application)
SwafS-29-2020: The ethics of technologies with high socio-economic impact
SwafS-30-2020: Responsible Open Science: an ethics and integrity perspective
SwafS-31-2020: Bottom-up approach to build SwafS knowledge base
CYCLOPS – Cycle of Creativity: a Leading Open Intellectual Property System for the European Union (total score: 9.50)
Positive points:
- The overall scope and goals are very ambitious, the proposal is well structured and demonstrates awareness of the current issues. Some WPs are well thought through (WP5 and WP2 – “Giving all stakeholders a voice”, Edgeryders leading) – very likely to lead to new insights. The outputs are clearly defined and very well linked with the expected impacts and there is a possibility for the network to be created beyond the scope of the project. Critical risks for implementation and contingencies are addressed and management well described.
Shortcomings:
- Short duration for the proposed activities, the feasibility is not justified. Much of the work is unspecified and open, some research questions are unclear and vague. The results to be achieved are numerous and connected with each other in a rather complex way. Work packages focused on collecting the views of stakeholders are limited by a reduced set of languages and it is not clearly explained how the selection of languages, countries and issues is representative of the digital single market, its cultural and societal make-up. More detailed interfacing with senior lawyers would be needed in order to prevent mischaracterization and simplification of law for the purposes of empirical work.
- The interdisciplinary approach is not clearly expressed in all sections. The distribution of activities foresees very reduced cooperation among the partners while dissemination and communication activities lack details. The risks related to the project’s ambitious goals are not sufficiently considered.
AI SET – Arts Inspired Socio Economic Transformation (total score: 7)
Positive points:
- The concept is compelling and has imaginative components especially in terms of how a material object can collect and mediate experiences of mobility, social change, place and identity, as well as improve well-being among returning migrant populations. It is a bold, ambitious and multi-stranded project that combines data-driven enterprises with the everyday worlds of craft, objects and local, regional and global populations. The proposal is strong on stakeholder knowledge, the work plan clear and work packages appropriately structured. Consortium has a high degree of expertise, art specific approaches and competences in the field of digital economy are well covered. Particular strength: online element involving the ethnographic mapping – carefully defined and given adequate weight. (great point for us!)
Shortcomings:
- Connections between the existing field of research and the concrete case are not sufficiently described and links between socio-economic transformation and the core issues of the project not fully addressed. It is not clear how the social inclusion will be achieved or how the project will contribute to the further integration of the arts in policies and strategic goals of the EU. The objectives remain general.
- The use of social media and mobilising communities through online methods are not sufficiently explained (!), the development of new products and services not given sufficient attention. The project’s innovative character is also limited in terms of scope and impact. Academic knowledge and interdisciplinary approach are not considered well enough. Dissemination activities not concrete, different target groups not given enough attention and it is not clear how the partners would complement each other.
What is certainly clear from the evaluations is that our methodology is recognized to be innovative and our workpackage very relevant for the consortia. There is a need to stress more the interdisciplinary approach and emphasize the close collaboration between the partners. The dissemination part was often evaluated as weak.
What next? @alberto and I will be looking in more detail the upcoming calls to find the best way to use some of the proposals above and improve them based on this feedback.
Stay tuned!