Impact                                                        Some thoughts about keywords for FP9 (Part 1 of 3)

Download PDF

During the year 2017, the debate about the design of the next Framework Programme has evolved quite rapidly, notably following the presentation of the Interim Evaluation Report for Horizon 2020. At this relatively early phase of the discussion the focus is not yet on detailed contents and concrete figures, but rather on the broad concepts and key objectives for the European Research and Innovation Programme after 2020.
 
Against this background it seems timely and useful to shed some light at three keywords dominating the current debate. While this paper is having a closer look at the role of “impact” as a guiding principle for FP9, THINK Piece 4/2017 has a closer look at “missions” in research, and THINK Piece 5/2017 presents some reflections on “innovation” in this context.

0. Intro

The current statements from the European Commission promote very prominently the idea that the next Framework Programme should be designed to have greater impact, and go as far as stating that impact is even the key feature for the next FP.
 
At first sight, this announcement goes along with common sense, as it is indeed a reasonable expectation that a public intervention should have an impact – otherwise it would be meaningless and a waste of money. It is hence not surprising that the big chorus of FP beneficiaries gave a very warm welcome to these intentions and seem to fully go along with this new priority. As a result, there has been very little public debate on the role of impact as a yardstick for the design of FP9. This is somewhat surprising, as a closer look reveals that things might not be as obvious as the broad public consensus suggests.

1.  What is the impact of an FP?

Framework Programmes are multi-annual multi-billion multi-disciplinary multi-national endeavours, so the footprint of several thousand projects involving several ten thousand people is tremendous. The consequences of FP activities can be found in many different facets of our individual, societal, economic and political life.
 
A few examples might help to get a better feeling for this enormous complexity:

• Fusion research is aiming at new energy options for the next century
• MSCA open broader career path for thousands of researchers
• Medical research will lead to better diagnostics, new therapies, and at the end to additional years of healthy living for millions of people
• Space research will allow us to explore so far completely unknown areas in our galaxy
• Social sciences and humanities contribute to a better conceptual understanding of our societies and our different cultural contexts.
• And so on – these were just some very simple positive examples …

But there is of course also a negative impact of the FP, most visibly through the fact that money spent on the FP prevents public funding on other desirable issues …

Impact is thus a vast concept. This is positive, as it allows bringing together this wide range of effects under a common roof. At the same time, impact is per se not a very operational concept, and general notions like “more impact” or “better impact” are not particularly meaningful without further explicit interpretations.

2.  How to measure the impact of an FP?

Enormous efforts have been made over recent years to gain a better understanding of the many impacts of research, notably through more systematic and analytical accounting systems and the use of data mining and big data. This means that today we have a much better view on impacts of the FP than ever before.
 
Unfortunately, this progress is hampered by many profound problems which still prevent us from getting a complete and reliable picture. Two unsolved issues are mentioned here just to illustrate the difficulties:

• One can only count what is countable – such as publications, patents, number of young researchers involved, turnover of spin-off companies and so on. But what about impacts which are not countable, such as a new understanding of our universe and our history, new concepts for our societal life, entrepreneurial spirit or even such a broad feeling of “hope for the better”? These phenomena have undoubtedly an enormous impact on our life, yet they can’t be quantified for the time being and might remain for a long while outside any accountancy framework.

• Since the impacts under consideration are so diverse, one needs a common denominator to make these impacts comparable and to allow for some aggregation. The only such common denominator currently applied is money, which means one must put “price tags” at a wide range of effects. So, what is the monetary value of saving a ton of CO₂ emissions, reducing the pain for 1000 patients or discovering a new species in the rain forest?

We know today much more about many impacts of the FP than in the past, but we must acknowledge that a wide range of impacts still escapes from our radar – and that aggregating the diversity of impacts into something like a quantifiable impact figure looks still largely like a mission impossible.

3.   What are the time lines for assessing impact?
 
Not surprisingly, the different dimensions of the FP impact do not occur simultaneously, they rather spread over a long time. This can be illustrated by a closer look at the situation for Horizon 2020, the current FP running from 2014 to 2020:

• The first impacts probably occurred well before the official start of Horizon 2020, as the emerging debate over the priorities is likely to have triggered investment decisions by research organisations and individual professional career choices as from 2012 onward.

• Project funding will generate the most obvious immediate impact for the time span from 2014 up to 2025, when the last wave of projects is likely to finish.

• While the most classical impact such as publications and patents is likely to happen in the years immediately following the end of the projects, the wider and in a sense “real” impact on society and economy will only occur when the findings and innovations will be deployed in the real everyday life of citizens and enterprises. The speed of this process varies between branches and technologies, but to count for a good decade might be rough estimate across the board. This means that we might be able to get the full picture on Horizon 2020 impact by 2035 or so.

The sequence in the political decision-making process for the Framework Programmes is such that the interim evaluation of the previous FP becomes in fact the most influential assessment. Against the timeline described above it is by no means a criticism of the Lamy Report on the Interim Evaluation of Horizon 2020 to state that as of today we know in fact very little about the full impact of this programme.

4.   Which parts of previous FPs had the biggest impact?

Since the Framework Programmes cover such a large – and in fact ever more enlarging… - range of fields and activities, it would be very helpful, and in a sense crucial, for the design of a new FP based on “impact” as key factor to get a clear picture on how the different programme lines have scored so far in this respect.
 
Such information, however, is completely missing – not only for the FP, but also for all other large national or international research programmes.
 
The European Commission has put over recent years a lot of emphasis on “Better Regulation” and as part of this approach an ex-ante impact assessment has become mandatory for all activity lines. These assessments include a cost – benefit analysis which means that an activity should only be considered if the expected benefits (a proxy for impact…) would exceed the costs incurred. As these calculations often use different methodologies, numerous assumptions, weak data and sometimes large econometric modelling, they are useful as a first filter for a “go – no go” decision, but would not allow comparing and even rank different activities.
 
The straight answer to the basic question is thus: we simply don’t know.

5.   How to increase the impact of the next FP?

While increasing the impact of the next FP is a very laudable aim, the analysis in this paper so far does not suggest any obvious strategies to get there – as the evidence we have at our disposal is simply too limited and too weak to draw reliable conclusions.
 
Also, the European Commission has not yet come up with any clear statement on how to implement this in FP9. Reading between the lines of the most recent “Overarching Strategic Document” to the Horizon 2020 Work Programmes 2018 to 2020¹ hints at two possible avenues, mentioned in this specific context and possibly highlighted as elements of a future FP9 design:
 

• Impact Statements

 The new set of work programmes provides even more detailed indications on the expected impact of the planned activities, and applicants are expected to respond in full to these in their proposals.

While it is always positive to raise the awareness for this important issue, there is on the other hand the doubt whether the impacts imagined in the work programme are realistic ones – and whether there could be other, possibly even more important ones, which might be neglected.

“If we talk a lot about it, things will get better” is a frequent behaviour in everyday life, and we all know about instances where this proved to work – but also about many cases where it failed completely. From a more systemic point of view there is so far no evidence available that there is in fact a positive correlation between more detailed expected impacted impact statements in work programmes and proposals and a more substantive real impact from the subsequent projects.

• Big projects – big impact?

Another possible approach, which one could read between the lines of the Strategic Document, is a kind of renewed “big is beautiful” logic. This is a very relevant issue, as in Horizon 2020 there is a range in the funding per project from less than 5.000€ to almost 500 million€. It is obvious that large projects will produce more and most of the time more visible impacts than smaller ones. It is important, however, that small does not necessarily mean “low”, and that the great diversity of smaller activities might well generate in sum larger impacts than a single “big ticket”.
 
As long as there is a complete lack of any evidence on this issue, such as an indication of “impact per € FP funding”, it seems problematic to argue in favour of larger activities based on a “boosting impact” rhetoric. One might want to go this direction for political reasons, but this should be clearly communicated as such.

6.   Some preliminary conclusions

This paper argues that impact, one of the key objectives of the next FP, is an extremely huge and complex concept which largely lacks a coherent operational meaning. Available evidence is scarce, and even non-existent for several key questions. In other words, the current situation could be described as “We want to focus on impact, but we do not know much about it”.
 
In this context it is surprising that there is no public debate, and all major stakeholders seem to be so convinced of generating impact, that nobody dares to question the process.
 
There is, however, a big risk that the “focus on impact” argument could be used to promote other agendas, and to hide political priority settings behind the seemingly uncontroversial strive to “boost impact”.
 
For the emerging debate it might be useful to retain two simple messages:
 

• There is no evidence whatsoever that small activities have a lower impact (per € spent) than big action lines.
• And the choice of any specific dimension of impact as particular relevant (such as the short-term creation of new jobs) is a political one, again not supported through any empirical evidence.

Version 1.0 – 06.12.2017 - Feedback