Using indicators responsibly: a guide to good practice
Looking to follow good practice in your use of indicators? Follow these five key steps to make your process fair and valid. Further guidance can be found from the San Francisco Declaration on Research Assessment (DORA).
1. Consider what you value
It can be tempting to start with what is easy to measure, but easily available indicators do not always capture the heart of what you value in a researcher, research output, research group or institution. Start by identifying what you value and then identify a range of measures which reflect that perspective.
2. Explore creative and qualitative options
Publication metrics can appear to offer an objective picture, but they only really tell you how much research is cited, which is not a measure of true quality. You will often need to use qualitative measures and expert judgement to reflect true value. Peer review is an established way of drawing a qualitative picture. You may also consider asking researchers to indicate and evidence the value of their own research: for example asking a job applicant to identity their three best research publications for the interview panel to read. The idea of a biosketch in which researchers describe the strengths of their research career and publications is another format gaining traction: there are some useful examples on the NIH website.
3. Use multiple measures
If you are trying to understand the quality of research, you are likely to want to use more than one measure. There are multiple ways a piece of research can succeed including influencing the academic field, feeding into industry, developing new research methods or impacting on general society.
4. Check your ideas work in context
Check again at this stage: do your measures work in this context? Are you using citation data to evaluate early career researchers whose papers have only just been published? Are you including disciplines where journal articles are not the main research outputs? Do your local experts have the knowledge to undertake meaningful peer review?
5. Be transparent
Lastly, make sure it is clear what data you will be gathering, and how it will be used. This will help researchers to make sure that information about their research is up-to-date, and to proactively identify and explain gaps or limitations.