“Success Rates” –
Some reflections on the current debate
Success rates have emerged in the current political debate on Horizon 2020 as one of the most frequently used concepts. As, unfortunately, the term might not always be used with a proper understanding of its real meaning, this paper aims at shedding some light on the concept and implications of this indicator.
0. Intro
Within the community interested in European Research Programmes, notably Horizon 2020, “success rates” are followed with great interest. The term “success rate” is defined as the ratio between the number of proposals funded and the total number of (eligible) proposals received. First statistics for Horizon 2020 indicate a significant drop in this success rate, from roughly 20% across all activities in FP7 to some 14% for the first year of Horizon 2020 calls. Over recent weeks this issue received a great deal of public attention – so it might be useful to recall some fundamentals, to bring the heated debate back on safe ground.
1. Success ... for whom?
It might be trivial, but it is important to state: A lower success rate in Horizon 2020 does by no means indicate that the programme is less successful.
This distinction is crucial, as the prime interest of the Horizon 2020 management has to remain an implementation which will allow reaching the broad economic and societal objectives of Horizon 2020. The extent to which these will be obtained by the projects funded will be the real litmus test for the “success” of the programme.
Success rates as discussed in the current debate refer to something completely different, namely the probability for an applicant to see his or her proposal receiving financial support from the European Commission. There can be no doubt that such an indicator is of some relevance to all those planning to apply, but its importance for a successful overall implementation of Horizon 2020 should not be overestimated.
Looking at it from the perspective of the real Horizon 2020 activity, as carried out in the thousands of projects funded, the “success rates” are an ex-ante indicator used to describe the selection process. What ultimately matters, however, are ex-post indicators to assess the quality and impact of the knowledge production process.
2. High success rates are not necessarily good
It is absolutely plausible that applicants to Horizon 2020 calls would prefer a higher success rate than the 14% observed in the first round. Would one actually ask them about their “ideal” success rate, they are likely to answer “100%”. A success rate of 100% is actually what normal people (with a minimum level of solvency) expect from an ATM machine. Once you “submit” (to use the Horizon 2020 terminology for once) your bank or credit card, you will receive the desired amount of money in return. A mechanism close to this does actually exist in Horizon 2020, where in some duly specified cases the work programmes include “calls for named beneficiaries”. This means that only very few predetermined organisations are allowed to submit a proposal – and sometimes just one single applicant. Unless the submitted proposal is of completely inadequate quality, this approach leads to a success rate of 100%.
Admittedly, not even the most radical and eloquent lobbyists of European research organisations have ever requested a more widespread use of this very special mechanism in Horizon 2020, as it is blatantly obvious that the price to pay for ensuring top success rates is too high – Horizon 2020 would end up as a completely sclerotic programme without any drive for innovation and change.
3. Low success rates are not necessarily bad
What brings momentum to Horizon 2020 is the fierce completion between different consortia for the best approach to tackle a given topic. Unfortunately for the applicants the selection logic in Horizon 2020 is such that for most topics all competitors but one end up being not successful. From a programme management perspective, a more intense competition is desirable in terms of boosting creativity, up to the point where low chances of success might actually deter valuable applicants from a submission. It is an open question whether the current development from a 1 in 5 chance to succeed towards a 1 in 7 chance means Horizon 2020 gets anywhere near the point where the benefits of an intensified competition are outweighed by the losses due to the frustration of key actors.
As a matter of fact, chances for success are much lower in other parts of our societal life, and yet individuals, organisations and firms nevertheless stay in competition, despite heavy investments to participate. The following list provides some very mixed anecdotal examples:
· There is a 1 in 40 chance for a given country to win the European Song Contest;
· There is a 1 in 20 (theoretical) chance for any Formula 1 pilot to win the next Grand Prix;
· There is a 1 in 50 chance for a German Peugeot dealer that someone buying a new car will buy a car of this brand.
Obviously none of these examples is by any means directly comparable to the efforts of preparing a Horizon 2020 proposal. But sometimes even explicitly comparing apples with pears might help to get some ideas about orders of magnitude...
4. Every improvement has its drawbacks
The lower success rates in Horizon 2020 as compared to FP7 have a variety of reasons, and exploring these will require a substantial analysis. At this stage, however, it might be useful to recall what has happened with the transition between both programmes: After many years of complaints about the unnecessarily complex bureaucratic procedures and lobbying for simpler and more rewarding funding mechanisms, Horizon 2020 was welcomed as a major achievement in terms of simplification. So far, so good.
Yet, if you keep essentially all other variables (such as the budget) more or less constant, and you make a funding scheme substantially easier to access – you end up with more applicants. This is exactly what has happened with the start of Horizon 2020, and this obviously explains the lower success rates.
In a positive perspective, one could qualify this as a kind of “collateral damage” of a successful simplification approach. Those who complain now about the slim chances to succeed, after having lobbied for years the case of simplification, might find some comfort in a traditional German saying which captures well their current dilemma “Die Bauern haben immer Grund zur Klage – entweder wegen der schlechten Ernte, oder wegen der schlechten Preise” (Farmers always have a reason to complain – either about the bad harvest, or about the low prices”)
5. Don’t expect miracles
The European Commission has announced its intention to tackle the issue. There seem to be - in theory at least – three options to change the situation:
· Bigger budgets – under the given circumstances, this cannot be regarded as a valid option, so stop dreaming;
· More restrictive access to funding (a kind of “De-Simplification”) – again, not a valid option for anyone who has gone through all the pain of reforming FP rules for almost a decade;
· Fine-tuning of procedures, like more two-stage calls – does not change anything about the fundamentals, but might help to make the reality less painful.
At the end, the best tactics might be to implement some small scale measures – and for the rest to simply wait and see. For as long as we do not know whether the current lower success rates are a persisting trend or just a temporary problem, hectic activism might be counter-productive.
Version 1.0 - 19.11.2015 - Thanks for your feedback
