Just asking the question - How reliable are GCSE and A-level grades?

The purpose of this article is -

  1. to provide you and your centre with some relevant background information on marking
  2. to seek your feedback on the current situation, and whether you think there is an issue to consider here or not, based on your own experience.
  3. This is the first exam session for the new GCSE programme so everyone should be aware that every new system needs time to bed in and establish itself.
  4. This article will be followed up by a short survey on this summers results and post results services.

Setting the Context:

You may have seen some articles in the press recently concerning the reliability – or otherwise – of GCSE and A level grades, for example

 

http://www.telegraph.co.uk/education/2017/08/20/thousands-students-set-receive-wrong-gcse-mark-new-system-experts/ 

 

http://www.telegraph.co.uk/authors/dennis-sherwood/

 

http://www.telegraph.co.uk/education/2017/08/21/exclusive-exam-watchdog-warned-new-gcse-grading-system-will/


http://www.telegraph.co.uk/education/2017/10/03/children-having-careers-blighted-due-inaccurate-exam-grades/

 

https://www.tes.com/news/school-news/breaking-news/heads-gathering-discuss-unreliable-exam-grading

 

An overview on grade reliability by Dennis Sherwood (reseracher)

We all know, and Ofqual acknowledges (see, for example, paragraph 5 of https://ofqual.blog.gov.uk/2016/06/03/gcse-as-and-a-level-marking-reviews-and-appeals-10-things-you-need-to-know/), that different markers can legitimately give the same script different marks. So, for example, a script might be marked 64/100 by marker X, and 66/100 by marker Y. Both marks are valid; no mistakes have been made; marker X is not ‘hard’; marker Y is not ‘soft’. We know all this.

 

But this ‘legitimate variability’ can cause a problem with grading: if grade B is 61 to 65 marks, and grade A is 66 to 70, then a script marked 64 by marker X is awarded grade B, but the same script, marked 66 by marker Y, is awarded grade A. Furthermore, if the script is awarded a B, and then fairly re-marked by marker Y on appeal, the B is up-graded to an A.

 

The fact that the grade awarded depends on whether marker X or marker Y marks the script first is hugely unfair in principle; the fact that a B might be upgraded to an A as the result of a challenge, even though the original mark was legitimate, casts doubt on the first-awarded grade, and erodes trust and confidence in the whole examination system.

 

This example is familiar. It is also important. It lies at the heart of the problem of grade (un)reliability. Grades that depend on who marks the script first, and grades that might be changed on appeal, are thought by some to be intrinsically unreliable and untrustworthy.

 

Just how reliable are today’s grades?

It might be thought that a measure of grade reliability could be deduced from the number of up-grades actually awarded. So, for example, for the summer 2016 GCSEs, 51,150 up-grades were awarded after challenge – rather less than 1% of the total number of certifications, 5,527,000 (see Ofqual’s 2016 Statistical Bulletin https://www.gov.uk/government/statistics/reviews-of-marking-and-moderation-for-gcse-and-a-level-summer-2016-exam-series, Tables 1 and 2).

 

In fact, this number – 1% of the total number of certifications - is not a valid measure of the reliability of grades, for it is based only on those who raised a challenge. If, for whatever reason (like the fee!), a candidate does not raise a challenge, then no up-grade can take place, and so it does not “count” in the official statistics – rather like a crime that is not reported.

 

This year, there were about 4,000,000 GCSE submissions. Of these, about 12.5% - some 500,000 – would be up-graded after a fair re-mark. That’s one in every 8 awards. Or, to put that another way, it is quite likely that every candidate who sat 8 or more GCSEs this summer has a certificate including at least one grade that would be up-graded after a fair re-mark. That’s every candidate. And this applies to A level too.

 

One more fact. 12.5% of the total number of 2016 certifications (5,527,000) is 690,875, this being the total number of up-grades that would have been awarded had everyone raised a challenge. In fact, the actual number of 2016 upgrades was 51,150 – about 7% of 690,875. That tells us something important: over 90% of those who were ‘eligible’ for an up-grade did not raise a challenge.

 

Conclusion:

So, what has this got to do with the exams officer community? You are the eyes and ears of the exams system in centres. You see and hear what students, parents and staff say about courses, exams taken and results given. You are one of the key people involved in the processing of results and post results on behalf of your centre.

 

Please share this information with your teaching colleagues and help them understand, the context in which they work and how you have to respond to their requests, given the context in which marking and grades are awarded.  

 

The iEOA will be issuing a short survey shortly for feedback on member's experiences of the results and post results services which will also try and capture some of this discussion about this summers results, but please do not hesitate to contact the iEOA office at – ieoa.office@examofficers.org.uk and share your experiences.