New research agreement
“We want to move away from purely quantitative criteria”
The University of Bern has signed an international agreement to reform research assessment. In this interview, Andrew Chan explains the Agreement on Reforming Research Assessment (ARRA) and its significance for early-career researchers.
uniAKTUELL: Andrew Chan, what exactly is this agreement about?
Andrew Chan: With the agreement on the reform of research assessment, universities, research funding bodies and other institutions in the European scientific area have agreed on a common approach to research assessment. The assessment of quality in its various dimensions is to come to the fore, supplemented by a responsible use of quantitative parameters. This applies to projects as well as researchers and institutions. The agreement guarantees all participating universities and institutions a high degree of autonomy and freedom in the application of evaluation criteria.
So is the ARRA about moving away from purely quantitative evaluation?
Exactly: we want to move away from purely quantitative criteria. And that's nothing new. The University of Bern joined the San Francisco Declaration on Research Assessment (DORA) back in 2016, which calls for us to move away from purely quantitative metrics such as the journal impact factor towards an evaluation based on the actual scientific quality and content of the work. The ARRA is developing this approach further and prioritizing qualitative evaluation even more. What counts is what we actually achieve as researchers, not just how often we are cited or in which prestigious journals we appear.
What does that mean in concrete terms?
When the quality of research is assessed by peers, they also look at factors such as relevance, open science, transparency and reproducibility. Weight is also given to different types of scientific output, including collaborative and interdisciplinary work or outputs such as the provision of data platforms or scientific communication. It is not that quantitative metrics are no longer taken into account, but the signing of the ARRA is a reminder to use these criteria more mindfully.
ARRA (Agreement for Reforming Research Assessment) and DORA (San Francisco Declaration on Research Assessment)
The ARRA, developed in 2022 under the umbrella of the EU Commission, develops DORA further with ten voluntary commitments: the use of quantitative metrics is to be reduced and qualitative assessments strengthened. It also promotes the recognition of diverse scientific contributions – for example in data management, mentoring or science communication. Implementation is coordinated by the Coalition for Advancing Research Assessment CoARA, which was established by the signatories in 2022. It supports participating institutions by sharing experiences in the introduction of quality-oriented assessment procedures.
The idea for DORA was born in 2012 at the annual meeting of the American Society for Cell Biology in San Francisco. The initiative calls for a reform of the assessment of scientific research. Instead of the controversial journal impact factor, a variety of indicators should be used and scientific content should be prioritized over metrics. DORA is aimed at all stakeholders – from researchers to funding institutions and publishers – and is now established globally. The University of Bern became a DORA signatory in 2016.
What are the consequences of the agreement for researchers at the University of Bern?
The Vice-Rectorate for International Affairs and Academic Careers, together with the Vice-Rectorate for Research, will now draw up an action plan over the course of a year in which the general direction of our future strategy in this matter will be specified. In doing so, we will work closely with the faculties and centres as well as with academic staff at all levels. We will first take stock and then try to translate the goals of the ARRA into concrete measures for the individual faculties. We want to develop quality indicators, which may vary from faculty to faculty. What we are striving for is to get a process going and to take advantage of synergy effects in doing so.
ARRA is an international agreement, who is the driving force?
It must be clearly stated that the ARRA is mainly driven an EU initiative. By signing it, we have consciously aligned ourselves with values that are strongly represented by the EU; above all, transparency and academic freedom. These are values that are of central importance to the University of Bern. In a global situation that is causing uncertainty everywhere – also in science – we want to make a clear commitment to these values.
«By signing the agreement, we have consciously aligned ourselves with values that are strongly represented by the EU; above all, transparency and academic freedom.»
Andrew Chan
Young scientists in particular have high hopes for a paradigm shift away from purely quantitative evaluation criteria. Are they right to hope?
I have to do some expectation management here. This change does not mean that you can now simply choose assessment criteria according to your liking. No one can guarantee that the ARRA will make it easier for individual young researchers to achieve their career goals, because the ARRA did not bring about a rise in the number of professorships.
Conversely, there are also those who foresee negative consequences. People who fear that scientific quality will decline because of initiatives such as ARRA ...
Yes, I hear that a lot: we are losing excellence! I strongly disagree with that notion. On the contrary, our aim is to promote quality and highlight excellence. But we have to be open to the fact that scientific quality can also manifest itself in other ways than in the criteria that have dominated to date. Added to this is the fact that we are living in a time with new approaches and technologies, the effects of which we cannot yet foresee. With AI, additional quality dimensions will be added that also include components of scientific ethics. We will also have to include and value these aspects.
One area in which quantitative assessment criteria traditionally play a major role is appointments. How will the ARRA affect these procedures, for example, will the university management check which criteria have been included in the selection proposals?
Yes, we are already doing that, and we will certainly pursue this even more rigorously with regard to the ARRA. But I think we have to give the individual programmes and faculties their freedom and leeway. It is clear that the quality criteria in the various fields of study are very different. But we do need to think together about what are good quality indicators that we can agree on.
And what about the resistance to change, the tendency to cling to the old ways?
Of course, we are caught up in a system in which appointments have been made according to the old indicators and some of them still are. But I think we are largely in agreement that we want to change this. But like any change process, it will take time.
About the person:
Andrew Chan
is Professor of Outpatient Neurology, specifically Neuroimmunology and grew up in Germany. He completed his education with extended scientific stays in Canada, the USA and the UK. He has been working at the Faculty of Medicine at the University of Bern since 2016, initially as an associate professor and since 2019 as a professor. He is also Chief Physician at the Department of Neurology at Inselspital Bern.
Is the Executive Board of the University of Bern relying solely on advocacy work in this process, trying to win people over diplomatically, or is it also exerting pressure?
We will critically review the reports of the appointment committees, if we see that the ARRA's spirit has not been sufficiently addressed, that is clear. And of course we also have the option of rejecting reports. But the university Board cannot and will not presume to dictate what quality means for the individual fields of study.
You mentioned that the University of Bern has been using qualitative assessment criteria for some time. Has this already had an impact? Have you noticed any changes?
I see that more and more value is being placed on very different dimensions in appointment procedures. For management positions, for example, we no longer only discuss scientific excellence using traditional metrics, but also look at the characteristics and qualities that are necessary to lead larger institutes. This is about leadership culture. After all, groundbreaking scientific advances are being made less and less by individuals and more by groups. This requires different qualifications than a researcher quietly working alone at their desk in their little office.