What can we learn from international comparison studies?
Gabriel Lee, Research Fellow
England has actively participated in international comparison studies such as PIAAC (Programme for the International Assessment of Adult Competencies), PISA (Programme for International Student Assessment) and TIMSS (Trends in International Mathematics and Science Study) for many years. PIAAC, PISA and TIMSS each involve a maths or numeracy test of their participants: 9-year-olds and 13-year-olds for TIMSS; 15-year-olds for PISA; and adults for PIAAC.
The results of PISA 2022 are in the public domain, and the reports of the latest cycle of PIAAC and TIMSS are expected in December 2024. In this post, I want to reflect on what can be learnt from reading across these studies and consider if one should pay them much attention – whether as a researcher, an educator, or a policymaker.
What have we learnt from international comparison studies?
In the most recent decade (2014–2023), England has repeatedly achieved above international averages in maths in PISA and TIMSS. For example, according to its PISA 2022 national report, England had a higher average score in maths than the OECD trend in PISA 2022 (492 > 475), 2018 (504 > 490) and 2015 (493 > 487).
As PISA and TIMSS also collect sociodemographic and attitudinal data, the studies have identified gaps in scores between characteristic groups, or associations between pupils’ scores in maths and their characteristics. For example, pupils from higher socio-economic backgrounds tend to score higher than pupils from lower socio-economic backgrounds, and pupils with more confidence in maths tend to score higher than pupils with less confidence. But we already knew that.
Having said that, PISA and TIMSS results disagree on differences between boys and girls: PISA results showed boys tended to score higher than girls, whereas TIMSS results showed that the difference in score between boys and girls was not significant. Differences such as this, together with issues around sampling and response rate, suggests that results of international comparison studies should be interpreted cautiously, rather than merely looking at the rankings (which is typically how some media handle the reporting of these studies) or cherry-picking elements of the analysis. Each assessment privileges different things so, as with all assessment, the analysis tells us something about the test as well as those tested.
Why should anyone care about international comparison studies?
International comparison studies generate large databases of information about pupils, teachers, schools and adult numeracy which can be useful for researchers and for theory building. They can tell us some things, but not everything.
International comparison studies have impacted on educational policies and practices in England. For example, the introduction of Teaching for Mastery Programme and Maths Hubs in 2013/14 can be traced to PISA and TIMSS and politicians' ambitions of replicating East-Asian successes. In addition, the interest in establishing national textbooks, as in Singapore, can be seen as an influence of international comparisons and policy-borrowing.
Similarly, the first cycle of PIAAC (2012) highlighted how England’s adults, especially 16-25-year-olds, tended to score lower in numeracy than the international average and, perhaps more interestingly, than older adults in England. Since then, discussions on improving adults’ numeracy skills have been ongoing and the DfE’s Multiply programme saw massive investment in adult numeracy.
We are now approaching a general election and a possible change of educational policies afterwards. Given that it is ten years since the introduction of international-comparison-inspired maths policies, and that we will soon see what impact these have had on the numeracy of 16-25-year-olds (in PIACC), 2024 will be a good year for appraising whether political attention to international comparison studies has paid off.
Furthermore, these are the first cycles of PIAAC and TIMSS after the COVID-19 pandemic. As PISA saw a drop in average maths score in England between 2018 and 2022, the latest results from PIAAC and TIMSS will also enable us to investigate how the pandemic impacted individuals’ skills in maths, on a global basis. (Interestingly, PIRLS, an international comparison of year-5 pupils’ reading skills, saw no significant change in average reading score in England between 2016 and 2021.)
These trends and international comparison are of core interest to the Observatory. We will be conducting detailed and longitudinal trend analysis on pupil outcomes and enriching these analyses with massive cohort studies that help to explain observed trends. We will monitor the results of forthcoming international comparison studies with interest.
Further reading
Bolton, P., Lewis, J., & Harrison, C. (2023). Estimates: Spending of the Department for Education on post-16 further education, colleges and adult education (Research Briefing). House of Commons LIbrary. https://commonslibrary.parliament.uk/research-briefings/cdp-2023-0147/
Jerrim, J. (2021). PISA 2018 in England, Northern Ireland, Scotland and Wales: Is the data really representative of all four corners of the UK? Review of Education, 9(3), e3270. https://doi.org/10.1002/rev3.3270
Lewis, J., & Maisuria, A. (2023). ‘Maths to 18’ in England (Research Briefing). House of Commons Library. https://commonslibrary.parliament.uk/research-briefings/cbp-9780/
Author information
Gabriel is a Research Fellow based in the School of Education and the Observatory for Mathematical Education.
Observatory for Mathematical Education team
Observatory for Mathematical Education on LinkedIn