Sampling schools

David Sirl, Senior Research Fellow and Chris Brignell, Deputy Director

At the Observatory for Mathematical Education, we are recruiting 150 secondary schools and 200 primary schools to take part in the largest ever cohort-tracking research study in mathematics education in England. Our initial question was simple: ‘How do we know which schools to invite?’. The answer was not so straightforward. 

The simplest approach is to download the list of schools from the Department for Education’s website and use your favourite random number generator (everyone has a favourite RNG, right?) to select schools entirely at random. While this approach is good for removing selection bias on our part, the question of what we mean by a ‘representative sample’ is not so clear. 

For example, if you select schools at random then a school with 30 pupils per cohort has the same probability of being picked as a school with 300 pupils per cohort. This is fine if you want a representative sample of schools, but an important point to keep in mind is that bigger schools contain more pupils. That point is so simple that you might wonder why we mention it, but it has important consequences like the following: while secondary schools with a cohort size above 270 comprise 11% of schools, over 18% of pupils attend a secondary school with a cohort size above 270. This raises the question of whether it is more important to get a representation of schools, or a representation of pupils within those schools?

Selecting at random is also not the same as selecting uniformly. On one of our dummy-runs no secondary schools in the southwest of England were selected. After careful checking of our computer code, it became apparent that this was purely by chance and not an error. (After all, unlikely things happen all the time: there are about 10²⁶³ different combinations of 150 secondary schools that we could sample and 10⁴⁴⁰ combinations of 200 primaries, so whichever combination we get will be an unlikely one. If the idea that unlikely things happen all the time has bent your mind in a way that you enjoyed, you can find out more about it by searching the web for “improbability principle”.) However, it prompted us to ask whether our survey would be valid in the eyes of key stakeholders such as policymakers, education leaders and other researchers if, at face-value, it looked like we had excluded a whole region. 

As anyone who’s worked with real data will know, it inevitably contains some errors and omissions. For example, our data sources suggested that there are 228 secondary schools in England for which the DfE apparently doesn’t know whether they’re boarding schools or not. Data checking also revealed numerous secondary schools officially listed as ‘mixed’ gender but who only recruited boys or girls in year 7. (The explanations for these issues are first that some data is supplied by schools and not checked so “not applicable” to a question about boarding is understandable for someone filling a form; and second that these schools were typically single-gender for years 7-11 but mixed-gender for sixth form.) Cross-checking different DfE data files (e.g. statistics releases, annual school census data, school performance tables) and individual school websites revealed and/or corrected more anomalies. From a practical viewpoint though, we had to stop at some point and get on with recruiting schools to our study. There will inevitably be some errors still left in our data, but we are confident that they are very few and so will have very little impact on our work. 

Overall, we compiled a list of around two dozen variables of potential interest for each school. Even though we’re conducting the largest ever study of its kind, 150 secondary schools and 200 primary schools is not enough to include a school with every possible permutation of the different variables. So we used our research expertise to narrow the field to the variables we think might influence maths attainment and attitudes to maths, including things like the size of school, the demographic makeup of the school (e.g. percentage receiving free school meals), geographical measures (e.g. urban or rural), types of school and so on. 

At this point there was lots of discussion between the statisticians and the education experts about what qualities of schools were of particular interest to us and thus which we should monitor more closely for representativeness and which we might stratify our sampling by. For most variables, our testing showed that a simple random sample would give a good cross-section of schools that were broadly representative. However, we decided to stratify the sampling of secondary schools according to gender and selectivity, to ensure that we sample enough single-sex schools and selective schools that we can compare them to their much more prevalent mixed and non-selective counterparts.  

Finally, we let our favourite random number generator do its job and our school sample was selected (including a healthy number of schools in the southwest). Job done! 

Well, not quite: not all invited schools will want to take part, so we also had to develop a procedure for replacing schools who decline. Schools are almost certainly not all equally likely to decline the invitation (for example, the recent PISA test included disproportionately many higher performing schools in England because of uneven response rate), so replacing declined schools with other schools chosen randomly will bias our sample. We need to make sure that, as far as possible, we replace declined schools on a like-for-like basis. Defining exactly what “like-for-like” means is challenging, but between the statisticians and the education experts we agreed a methodology which was reasonable, practical and transparent. The spirit of that process is as follows. If we need to replace a given school, we look at all the two-dozen variables for all the schools in England and match up those variables as best we can; with some prioritisation of which variables are more and less important to match closely. 

It’s said that a journey of a thousand miles starts with a single step, or in our case a 7-year cohort study starts with a single school. At the time of writing, we’ve had the first schools respond positively to our invitation. Our virtual ‘totaliser’ is increasing and we are closely monitoring the sample on our two-dozen variables of interest. We’ve taken these first steps on our journey with nervous excitement, knowing that getting to our destination depends on good sampling. Who will we meet? What will we find? We hope you’ll travel the journey with us. 

Author information

David is a former Associate Professor in Mathematics who now works as a consultant and as a statistician for the Observatory. David has played a major role in the sampling process for our cohort studies. 

Chris is the Deputy Director of the Observatory and an Associate Professor of Statistics in the School of Mathematical Sciences. 

Observatory for Mathematical Education team

Observatory for Mathematical Education on LinkedIn.