Policy-based evidence making

 Andrew Noyes, Director 

Much is written about evidence-based policy making in education but here I want to turn that notion on its head: policy-based evidence making.   

In my long experience of policy advisory work the research evidence needed to support those tasked with tackling the longstanding, ‘wicked problems’ of education – at different scales in the system – has often simply not existed. And where some evidence is available, it’s partial.

The recent expansion of large-scale experimental research (e.g. Education Endowment Fund trials), though valuable in many ways, doesn’t address the complex interconnected system challenges that we face in education. In trying to emulate ‘medical research’, education trials often focus on tightly defined solutions to specific issues (c.f. medicine trials) and not the holistic, messy education contexts of classroom, schools, MATs and regions. Education policy would benefit from large-scale observational studies in the same way that public health research has done. 

In 2018 the Royal Society and British Academy published Harnessing Education Research. The report’s first recommendation - on ‘supply and demand’ - assumes that what is being supplied by researchers is what is being demanded by policymakers.   

I’ve heard it said that ‘researchers produce many excellent answers to questions that no-one is asking’ - a little harsh perhaps but worth considering. Researchers who seek impact from their work could collaborate better with the users of their evidence to ensure better alignment of supply and demand. This is easy to say and much harder to do, not least because education – like all public services – is complex and replete with interdependencies. 

Education research is something of a cottage industry, typically comprising insights from many relatively small studies that accrete over time. This results in part from small funding pots and lone-researcher studies. Where there are major programmes of work, these still pale into insignificance when compared with the scale of the problems they are purporting to understand and address.   

Today’s publication by the Royal Society and British Academy estimates that only 0.05% of the public funding for education in the UK is committed to research and development (the equivalent for health is 34 times larger!). To improve evidence-based policy and implementation planning in education, there needs to be greater strategic investment and improved tactical coordination. 

In a 2016 paper my colleague Mike Adkins and I explored how a research result (‘those with A level maths earn 10% more’) became part of political discourse. This is the romantic view of many researchers, that their findings might get taken up in this way. This is rare, and in the paper we described the six conditions for such ‘successful’ uptake of research. We also noted that there are many occasions when research changes on its journey from academic paper or report to policy and practice: through decontextualization, partiality, overgeneralisation and misinterpretation. 

The pathway from research to real world impact is fraught with challenges, not least because researchers, policymakers and system leaders are often working on quite different kinds of problems. Perhaps it would be better for research ‘suppliers’ to liaise with those evidence ‘demanders’ from the outset to smooth – and in some sense reverse - this pathway. (The James Lind Alliance has an interesting model for developing “priority setting partnerships”, albeit with different combinations of stakeholders.) 

The Observatory for Mathematical Education is working across this research-policy-practice nexus, establishing constructive dialogue with decision makers at varied scales in the system from the outset. We recently hosted a meeting with a range of strategic thinkers, including policy advisors and DfE officials, to elucidate the kinds of questions that the Observatory’s ambitious programme might be able to answer.   

There was much excitement about the Observatory’s programme, and its commitment to better alignment of the priorities of researchers, policymakers and system leaders…to evidence-based policy making meeting policy-based evidence making. 

Author information

Andy is the Director of the Observatory for Mathematical Education. He is also the Chair of the Joint Mathematical Council of the UK and a member of the Royal Society’s Advisory Committee on Mathematics Education. 

Observatory for Mathematical Education team

Observatory for Mathematical Education on LinkedIn.