‘You only assess what you care about’: a new report looks at how we assess research in Australia

November 19, 2023

The project, spanning researchers across science and the humanities, looks at how ‘research assessment’ affects research in Australia. What is research assessment? Our projectOur new project aimed to provide a better understanding of how research assessment affects research in Australia. Tima Miroshnichenko/Pexels, CC BY-SAWhat we foundWe found some difficulties with the current approach to research assessment. The way forwardIn our survey, we asked “What is one specific change you would recommend to improve current research assessment processes?”.

The project, spanning researchers across science and the humanities, looks at how ‘research assessment’ affects research in Australia.

Research plays a pivotal role in society. Through research, we gain new understandings, test theories and make discoveries.

It also has a huge economic value. In 2021, the CSIRO found every A$1 of research and development investment in Australia creates an average of $3.50 in economy-wide benefits.

But how do we know if individual research projects being conducted in Australia are good quality? How is research recognised? The key way this happens is through “research assessment”.

What is research assessment?

Research assessment is not a centralised or necessarily formal process. It can involve various processes and measures to evaluate the performance of individual researchers and research institutions. This includes assessing the quality, excellence and impact of various outputs.

Research assessment can be qualitative or quantitative. It can include publications in journals and the number of people who cite the research, gaining grants to do further research, commercialisation, media engagement and impact on decision-making or public policy, prizes and invitations to speak at conferences.

If research assessment is working fairly and effectively, it should achieve several things. This includes: helping to develop researchers’ careers, making sure innovative research does not get avoided in favour of short-term gains and helping funders and the community have confidence research is providing value for money and adding to the public good.

Our project

Our new project aimed to provide a better understanding of how research assessment affects research in Australia.

In a report released today, we surveyed more than 1,000 Australian researchers and more than 50 research organisations.

This included universities, research institutes, industry bodies, government and not-for-profit organisations. The majority of researchers (74%) were in academic roles. Across those research sectors, we also conducted 11 roundtables involving around 120 people and 25 intensive interviews to understand the issues.

This work was commissioned by Chief Scientist Cathy Foley and conducted by the Australian Council of Learned Academies (involving the academies of science, medical science, engineering and technological sciences, social sciences and humanities).

It also comes as the Universities Accord review examines how research is funded and approached within higher education.

Research assessment should help to develop researchers’ careers. Tima Miroshnichenko/Pexels, CC BY-SA

What we found

We found some difficulties with the current approach to research assessment.

We heard there is a tendency by some researchers to “play it safe” in terms of doing research they believe will score well. We also heard how the assessment process can unintentionally exclude or devalue particular forms of knowledge, particularly in the humanities and the social sciences, where outputs can be less easily quantified or less immediately seen.

As one interviewee said:

What is assessed and how it is assessed are an indication of what the organisation values. You only assess what you care about. Values and culture drive assessment.

Our roundtables told us senior staff and supervisors are often seen to reinforce the culture of “publish or perish”, with the number of articles being valued more highly than the quality.

We heard early and mid-career researchers and people from underrepresented backgrounds can have difficulties trying to “play the game” to advance their careers. For example, early-career researchers are often expected to produce work that benefits their larger team, at a cost to their own capacity for promotion.

As one interviewee noted:

Metrics are essential for defining value and comparative difference, but Australia requires a modern and fair framework for assessing our current and next generation of researchers.

Survey results

Our survey found a high level of dissatisfaction with the state of research assessment. This included:

  • 73% of respondents agreed assessment processes are not consistently or equitably applied across disciplines, in particular between the humanities and the sciences

  • 67% said there are not enough opportunities to provide input into research assessment practices

  • 70% said assessments took up unreasonable time and effort.

The way forward

In our survey, we asked “What is one specific change you would recommend to improve current research assessment processes?”.

Respondents wanted to see a shift towards quality over quantity. This means not just a focus on publishing as many papers as possible, but supporting research that may take longer for its value and benefits to emerge.

They wanted interdisciplinary research to be promoted and rewarded, because many of the complex problems of our world – from climate change to domestic violence to housing affordability – require multiple disciplines to be involved in finding solutions. In the same vein, they also wanted collaboration and team work to be rewarded more clearly and transparently.

They wanted less bias towards STEM (science, technology, engineering and maths) research and more promotion of diversity and of early-career researchers. This included better understanding of their personal and cultural situation, more focused career development and better managed teamwork.

To achieve all of this, and more, we will also need to understand that no single measure can assess all research or researchers. So, several tools will be needed, including quantitative indicators as well as qualitative measures and peer review.

Ana Deletic, Louisa Jorm, Duncan Ivison, Robyn Owens, Jill Blackmore, Adrian Barnett, Kate Thomann, Caroline Hughes, Andrew Peele, Guy Boggs and Raffaella Demichelis were all part of the expert working group supporting this work.

Kevin McConkey, Emeritus Professor , UNSW Sydney

This article is republished from The Conversation under a Creative Commons license. Read the original article.

The source of this news is from University of New South Wales

Popular in Research

1

May 28, 2024

Can exams be shorter and better?

2

May 29, 2024

HKGAI Debuts Cutting Edge AI Projects at InnoEx 2024 | The Hong Kong University of Science and Technology

3

May 28, 2024

Three NYU Faculty Awarded 2024 Guggenheim Fellowships

4

May 28, 2024

Campus garden initiatives can help grow the next generation of environmental change-makers

5

May 29, 2024

New molecule found to suppress bacterial antibiotic resistance evolution

Hagari: '99% of Iranian aerial threats were shot down'

6 days ago

Quiet before the storm: Is escalating Israel-Iran violence on the way? - analysis

6 days ago

Foreign holdings of US Treasuries hit record high; Japan holdings rise, data shows

6 days ago

Stormy Daniels' chilling warning to world on Donald Trump's bid to become US President

6 days ago

School suspensions and exclusions put vulnerable children at risk

6 days ago

Tackling cancer at the nanoscale

6 days ago