When people talk about research quality, they often jump straight to numbers: citations, impact factors, H-indices, and rankings. But, as Elizabeth Gadd explains a conversation on the ATG Podcast, research assessment is much bigger than metrics alone. To understand what makes research excellent, we have to look at the culture that produces it, the incentives that shape it, and the systems we use to judge it. Gadd discusses why bibliometrics can be useful in the right context, but also why they become problematic when used to rank people or institutions too simplistically, and how responsible research assessment aims to support a fairer, healthier academic ecosystem.
From English degree to research culture
Elizabeth Gadd’s path into research assessment didn’t begin with a grand plan. It started with an English degree, a love of libraries, and early work in public and college libraries. That experience drew her toward information work. A master’s degree at Loughborough University set the course for a career that has stayed closely tied to Loughborough for decades. Gadd says the university gave her room to learn, grow, and move into new areas—from librarianship, to research management and, eventually, research culture and assessment. Her current role as Head of Research and Innovation Culture and Assessment brings together two related priorities:
- Improving the culture in which research happens;
- Shaping how research is assessed, both within institutions and nationally
That connection matters. If institutions want better research, they can’t look only at outputs. They also need to consider whether the environment around those outputs is healthy, fair, and supportive.
Bibliometrics can inform—but they shouldn’t decide everything
Gadd became known for her work on bibliometrics, including the Bibliomagician blog. Her work in the area grew as academics asked for help understanding impact factors, H-indices, rankings, and citation data. As she looked more closely at these indicators, she saw how easily they could be used to suggest things they were never designed to measure. The key point is that bibliometrics are not a complete measure of research quality. They can provide useful information, but their value depends on the question being asked and the context in which they’re used. A research evaluation group, which she chairs, created the SCOPE framework for Responsible Research Assessment, which offers a way to think through that process:
- Scope:
What do you value about the thing being evaluated?
- Context:
In what setting, and for what purpose, are you evaluating it?
- Options:
What possible indicators or approaches might be suitable?
- Probe:
What could go wrong – what might the unintended consequences be?
- Evaluate:
Review the assessment approach—and the assessment process itself.
Hence, instead of asking “Which metric should we use?”, we should ask “What are we trying to value here?” Citation data can help show how work is being used. Altmetrics may reveal attention in policy, social media, or public discussion. But none of these indicators should automatically be treated as a direct measure of individual merit, without context. For example, an H-index can be influenced by career length. That can make it an unfair basis for comparing people at different career stages. The better approach is to use indicators in context, alongside qualitative evidence, and to be clear about what the data can—and cannot—answer.
Research culture and assessment are linked
What we measure is what we care about, but assessment systems can influence the behaviours they reward. If universities reward only certain outputs or career behaviours, they may encourage more of those behaviours. If systems favour short-term productivity over collaboration, wellbeing, or integrity, those priorities can become part of the culture. That is why the environment around research matters. Gadd argues that it’s difficult to claim research excellence if the conditions in which research is done are harmful or unsupportive. The issue also affects early-career researchers. Gadd challenges the assumption that because some researchers worked evenings and weekends and went on to succeed, that experience must be acceptable for the next generation. Surviving a system does not prove that the system is healthy
The REF, rankings, and the risk of measuring the wrong thing
Two metrics-related topics include the UK’s Research Excellence Framework (REF) and global university rankings. The 2029 REF uses the term Strategy, People and Research Environment (SPRE) rather than “research culture” in its title. But culture remains present in the guidance and, she says, is still recognised as an important part of research excellence. The assessment continues to use narrative submissions reviewed by expert panels, while the weighting of the research environment element has increased. Gadd would like to see improvements to the process, including:
- Publishing guidance before the assessment period begins
- Reducing the link between REF results and funding
- Making the process more formative and supportive
- Providing feedback that helps institutions learn, not only a final judgement
Why global rankings attract criticism
Gadd is more critical of global university rankings. The most-quoted rankings are produced by commercial agencies; they may use indicators that do not reliably measure the qualities they claim to represent. For instance, the number of Nobel Prize-winning alumni is not necessarily a measure of teaching quality. Global rankings tend to favour wealthy, research-intensive, English-speaking institutions in the global North. When rankings inform decisions about students, policymakers, or visas, those patterns can have consequences well beyond a league table. As one alternative to relying on rank alone, Gadd points to the More Than Our Rank initiative. It gives institutions a way to describe—in narrative form—what they contribute beyond what a ranking captures.
What better research assessment looks like
Elizabeth Gadd’s interview makes a powerful case for not relying simplistically on metrics. Research excellence also depends on fair assessment, healthy research culture, and systems that measure what truly matters.
Leave a Reply