What are research metrics?
Research metrics, sometimes known as indicators, are defined by Research England as “quantitative measurements designed to evaluate research outputs and their impacts.” Metrics include counts of publications and mentions of research outputs in academic publications (citations), social media and policy documents. They are available through a number of different databases and platforms which you can find out more about on our Research Metrics Tools page.
In order to use metrics appropriately and responsibly, following the University of Nottingham UK Guiding Principles for use of Metrics in Research Assessment, it is necessary to understand :
Considerations
For each metric, make sure you understand:
Traditional metrics, or bibliometrics, are based on counts of citations in academic literature i.e. the number of times a publication appears in the references or bibliographies of other publications. More recently “alternative metrics”, also known as “altmetrics”, have been developed to capture data such as page views, downloads, and mentions in social media, policy documents and patents.
Research outputs are cited and mentioned for different reasons, including to disagree with their conclusions, highlighting contrasting findings or acknowledge the existence of research on a particular topic, so citation-based metrics should be seen as a measure of attention or discussion and not necessarily impact or quality.
Some metrics relate to a specific period of time, for example Journal Impact Factor relates to a particular year. The platforms which provide metrics, such as SciVal, often have options to limit which time range is being considered to calculate metrics.
Metrics such as citation counts might apply to the number of citations received by an individual publication, all of the publications by a specific author, or all of the publications affiliated with a specific organisation. It is best practice to use metrics directly related to the entity you are interested in, rather than indirect metrics. For example, when assessing the attention received by an individual article, it is best to use direct citation metrics of that article rather metrics relating to the journal in which it is published, such as Source Normalised Impact per Paper (SNIP) or Journal Impact Factor (JIF). The University of Nottingham is a signatory of DORA which recommends against using journal-level metrics to evaluate individual research outputs or researchers.
Some systems for viewing metrics have options to filter what is included in a specific metric, for example by date, subject or institution. Make sure that you understand any limits that have been applied to any metrics you are viewing and communicate them when sharing metrics.
Metrics displayed on each platform are dependent on the underlying indexed data about research outputs (journals, books conference proceedings etc.). The platforms have different data sources and so produce different figures. It is important to understand the limitations of any data source, especially when working with data about an individual or group of individuals.
Scopus and Web of Science, for example, each have their own set of selection criteria and exclude some good quality scholarly documents and/or activity. The citations counted within these databases, and their associated analytical tools, enable analyses of an academic environment that is closely curated and managed under the ownership of commercial entities, Elsevier and Clarivate. Both have better coverage of science disciplines than arts and humanities and have significant gaps in coverage for books and book chapters.
Google Scholar follows a more inclusive but less supervised approach by crawling the open web for scholarly documents, and aggregating the citations into Google Scholar Profiles for individual authors. The results are more diverse in all aspects, but can lead to more errors, duplicates and fewer tools to normalize highly cited papers or discount self-citations.
Citation counts increase over time, and the norms of publication and citation vary between disciplines. Therefore, it may not be appropriate to compare raw figures without taking into account these factors. Some systems provide weighted or adjusted metrics which apply calculations to the data. In SciVal, for example, the Field Weighted Citation Index is the ratio between the number citations received by a publication and the average number of citations received by all other similar publications indexed in Scopus of the same publication type, publication year and discipline (defined by Scopus journal classification system).
It is important to remember that even with adjustments, metrics may still be impacted by disciplinary differences as there is difficulty in defining which publications belong to which subject (especially interdisciplinary research). When working with a small dataset, the average of measures like FWCI can be easily skewed by the presence of any highly cited outlier.
Some metrics, such as the Altmetric Attention Score, are composites of a number of measures of attention, weighting each of them differently. This can make them less transparent and more difficult to interpret.
At all stages of the research lifecycle, there are systemic biases which disadvantage people from specific groups, for example along the lines of gender, perceived ethnicity or race, geographic location, career stage, and language. Metrics therefore reflect many factors other than the quality of research. Any use of metrics in decision making must be carefully considered to avoid perpetuating biases and inequalities.There is evidence of such biases impacting rates of citation in the academic literature across a number of disciplines, including political science and social science, physics and neuroscience and in the sources alternative metrics such as mentions in news media. Inequities are exacerbated by the “Matthew effect of accumulated advantage”, whereby those with existing resources, including social reputation, are likely to accrue greater rewards for the same work. Harriet Zuckerman and Robert Merton, who coined the term, demonstrated that the research of more eminent scientists would receive more attention than that of less established researchers.
Publications are cited or mentioned for different reasons, including to disagree with their conclusions, highlight contrasting findings or acknowledge the existence of research on a particular topic. Citations in these circumstances signal attention or discussion rather than positive impact or quality. Likewise, quantitative alternative metrics indicate attention but do not show the nature of that attention. The Guidance on the responsible use of quantitative indicators in research assessment from the DORA Research Assessment Metrics Task Force provides useful guidance on the limitations of a range of metrics.
There is some concern that certain metrics can be “gamed” or manipulated. For example, authors might deliberately cite their own previous research when it is not the most appropriate reference in order to increase their citation-based metrics. Some platforms have the option to exclude self-citations to mitigate this practice. However, there are cases when it is legitimate for a researcher to reference their own work. Other citation manipulation practices include “citation cartels”, which is a group of authors who collaborate to cite each other to increase their citation metrics, and pressure from peer reviewers or editors of journals on authors to cite specific authors. Citation practices are considered part of research integrity, authorship and publication ethics. More information about citation manipulation is available from COPE (Committee on Publication Ethics).