Search ‘never skip leg day’ and you’ll find a barrage of graphic warnings on the risk of working out your arms and upper body while ignoring your lower half. I wonder if Progress 8 has led to some schools ignoring ‘leg day’ – a solid academic core – in favour of superficial gains in the open bucket.
When Progress 8 was launched I remember thinking that this was the performance measure to end all performance measures – at last schools would be credited for the achievement of every pupil in every subject at every grade. No longer would schools limit their attention to the ‘key marginal’ pupils at the C/D borderline, writing off in the process those pupils deemed incapable of securing a pass and ignoring higher attaining pupils who might be able to gain exceptional grades with a bit of a push.
I still think that Progress 8 has its place in the performance measure mix, but I’m not sure that the predominance it has gained is justified, primarily because it’s too easily skewed by tactical behaviour in the open bucket and too far removed from the actual grades that pupils walk out with when they leave our schools.
First, is it right to say that Progress 8 has become the predominant measure of school improvement? I think so:
- The DFE’s floor standard for secondary schools is based on Progress 8 (lower than -0.5)
- The DFE’s Compare School Performance service gives a higher profile to Progress 8 than other measures e.g. if you enter the name of a school the first data you’ll see is that school’s Progress 8 score alongside a comment such as ‘well above average’ and if you create a comparison list of several schools, or compare all schools, the list is presented against Progress 8 scores by default
- The DFE’s Multi Academy Trust performance tables begin by comparing Progress 8 between trusts.
- Since its inception Progress 8 has topped the list of performance measures for secondary schools in DFE communication*
Separately, Ofsted’s Section 5 handbook explicitly states a preference for progress over attainment: ‘In judging achievement, inspectors will give most weight to pupils’ progress.’ It’s very clear from the rest of this paragraph that Ofsted is referring here to progress with a small ‘p’ rather than ‘Progress 8’ (e.g. the paragraph goes on to say that ‘inspectors will consider the progress of pupils in all year groups, not just those who have taken or are about to take examinations or national tests’) but with the DFE pushing its progress measure and Ofsted reaffirming its interest in progress, we can see why schools have reached the conclusion that Progress 8 is the performance measure that matters most.
When Progress 8 was introduced in 2015 it was – from my memory at least – schools with lower attaining intakes which welcomed its arrival most enthusiastically. At last, the key measure on which they would be judged would recognise pupils’ varying starting points and credit schools only for the progress made under their care.
It hasn’t quite turned out like that. In 2017 the average Progress 8 score for selective schools was 0.45 – well above average – while those schools languishing at the lower end of the P8 tables tend to be schools that also struggled under the previous attainment measures. If my Twitter feed is anything to go by, it’s now heads of schools with lower attaining white British intakes who feel most aggrieved by Progress 8, arguing that the playing field remains uneven.
So why have some people lost faith in Progress 8 so soon? Let’s take a few key issues:
- The Open Bucket – Despite the removal of ECDL (see previous post) there are still easier pickings to be found in the open bucket which can artificially inflate a school’s P8 score if they enter large number of pupils (thereby deflating the P8 score of schools that don’t). Whether it’s the LIBF Certificate in Personal Finance or the TLM Level 2 Certificate for IT User Skills in Open Systems and Enterprise, these qualifications benefit schools more than pupils, undermining our faith in performance tables in the process.
- The EAL effect – One striking feature of schools with exceptionally high P8 figures is that several of them have a high proportion of EAL pupils. If we go through the top 12 schools by P8 in 2018 then check the number of EAL pupils in their 2017 Y11 cohort (2018 EAL information is not yet available) we see EAL figures of 117 from a cohort of 119; 169 from 209; 17 from 111; 0 from 25; 105 from 118; 3 from 35; 68 from 75; 71 from 75; and 7 from 147 (data unavailable for three of the 12 schools because they didn’t have Y11 cohorts in 2017). The average here is 61% EAL. Switch the digits over and we get the national average of 16%. Perhaps the key point here is less about EAL and more that KS2 outcomes are not a great indicator of KS4 potential, particularly if performance at KS2 has been held back because pupils haven’t yet gained a secure grasp of English. Schools with lots of EAL pupils are potentially on to a winner: once their grasp of English is secure they motor ahead of first language English speakers. This is brilliant news, but it’s unclear why the schools they happen to go to should be credited for this.
- KS2 results – Just a hunch, but I wonder if disadvantaged pupils are more likely to have inflated SATs results compared to non-disadvantaged pupils. My thinking here is that disadvantaged pupils are more likely to attend primary schools at risk of poor KS2 performance, so these schools are likely to devote more time preparing for SATs rather than just teaching the normal curriculum. If this hypothesis holds water, then secondary schools that recruit a high proportion of disadvantaged pupils are recruiting pupils whose real attainment is weaker than their SATs results would suggest, making it difficult for these secondaries to secure higher rates of progress. Even if this is a bit far-fetched, we can probably agree with Dr Becky Allen that ‘Key stage 2 test score is quite a noisy measure of a child’s educational attainment at age 11‘ and that KS2 results depend on a wide range of factors such as the quality of teaching; the amount of preparation for KS2 tests; the way the tests are administered etc – so are fairly limited as indicators of a child’s ability at the age of 11, let alone their potential ability 5 years later. Given that one side of the Progress 8 ledger is based on KS2 outcomes we should probably avoid using P8 to make sweeping judgements about schools.
It’s the first point above – the problem with the open bucket – that troubles me the most. The DFE’s Compare School Performance service reveals the P8 score of each element (for 2017 at least, not yet for 2018), and it’s not unusual to find schools with modest P8 figures for English, Maths and EBacc, and then stunning P8 figures for the open bucket. We can assume with some confidence that this is because schools will be entering the whole cohort for two or even three vocational subjects, and then securing excellent outcomes in these subjects (of course the open bucket also includes subjects like RE, the English GCSE not already counted, as well as Art, Music and Drama etc, but it doesn’t really stand to reason that a school performs massively better in these subjects than in English, Maths, Science, languages and humanities, hence my confidence in suggesting that a disproportionately high Open Bucket P8 figure is likely to be based on large numbers of pupils doing vocational subjects).
If you’ve got a lower attaining cohort who all secure top grades in three vocational qualifications (in 2017 this would often have been ECDL, Business and Sport) then your P8 score would go through the roof. We’ve then got a strange situation where lower attaining pupils might only be making average progress in English or Maths – therefore leaving school without good grades in these subjects (because they started with low attainment and only made average progress) – yet the school’s P8 figure would indicate that the school is excelling. The school is rewarded for skipping leg day.
The first rule of league tables should surely be that they credit schools for the things that matter for pupils, thereby aligning the interests of the school and the interests of the child. The fact that Progress 8 rewards tactical behaviour in the open bucket is therefore hugely damaging for its integrity as a measure of school performance, and represents a dangerous drift away from the things that matter to pupils.
So despite the best of intentions in creating a measure that values every student, every subject and every grade, and recognising schools that add value from lower starting points, Progress 8 has not been the ‘measure to end all measures’ that I naively welcomed back in 2015. I think it still deserves its place in the mix, but I’m not sure it’s worthy of the first among equals status that it seems to hold at the moment. It would surely be better for school leaders to focus their energy on outcomes that matter to pupils – such as decent grades in English and Maths – rather than the school’s Progress 8 figure.
A simple solution to this is to remove the open bucket to create a ‘Progress 5’. We don’t need the DFE to do this for us – it’s something we do at United Learning to shine a light on the ‘core stability’ of schools, with the view that schools with strong performance in the academic core have a more secure basis for future success than schools relying on their open bucket.
The point of this post isn’t to attack schools that skip leg day, schools where the academic core lags behind the open bucket. It’s difficult for outsiders to know the challenges that a school might face in recruiting a stable team of English, Maths and Science teachers, for example, and I understand the drive to gain momentum at the beginning of a school improvement process. The open bucket offers quick wins to restore league table pride while longer term gains are made at the core of the curriculum.
Instead, the point of this post is to reflect on the way we rank and judge schools, and to think twice before heaping praise on schools that have secured a stunning Progress 8 without gaining decent grades in the academic core. These pupils might have starred distinctions in ECDL, Business BTEC and Sport BTEC, but unless they’ve also got 5s in English and Maths, they’ll struggle to get on to A Levels and degrees.
Where does this leave Progress 8? I’ve tried to argue that the two bits of data on which P8 is based – KS2 performance and performance across 8 subjects at KS4 – are both flawed, the former because KS2 performance depends a lot on how seriously primary schools take SATs and the latter because the P8 figure is easily skewed by the open bucket. As such, I don’t think Progress 8 is worthy of the status it seems to hold as the first among equals of our performance measures.
One of the criticisms of P8 that I considered but didn’t include here was the argument that a school’s P8 figure is easily distorted by a small number of ‘outliers’ – often pupils who have left school with no qualifications. Such pupils didn’t particularly affect a school’s figures under the old threshold attainment measures, such as 5A*-C with English and Maths, because although these pupils would be marked as failing on these measures, their impact on the school was no greater than ‘near miss’ pupils. Under Progress 8, pupils leaving school with nothing have a significant impact on a school’s league table performance.
I actually think this is a huge strength of the P8 measure. By incorporating the outcomes of all pupils in all subjects we’ve finally got a measure which encourages schools and the system as a whole to grapple with the ugly truth that thousands of pupils leave our schools each year with nothing to show for their education. The current and long-overdue discussions about off-rolling and exclusions are perhaps an unintended consequence of Progress 8, and might just represent Progress 8’s most important contribution to our collection of measures.
*KS4 performance measures, as stated by DFE in 2018
- Progress 8
- Attainment 8
- EBacc Average Point Score
- the percentage of pupils entering the EBacc
- the percentage of pupils achieving a grade 5 or above in English and maths
- the percentage of students staying in education or employment after key stage 4 (destinations).