The RSS welcomes a commitment by the Office for Statistics Regulation (OSR) to review the process that developed the statistical model used to grade this year’s exam results.
The RSS
requested that the statistics watchdog carry out the review (PDF) after the exams regulator Ofqual was forced to abandon the algorithm it was using to estimate students’ grades due to ‘significant inconsistencies’ in the results.
While any review of the methodology used will not affect this year’s grades, which have now largely reverted to those estimated by their teachers, our letter urged that an OSR review could help to ‘set a benchmark’ should this situation arise again – whether in education or in other areas of government.
We reiterated concerns that
our Education Policy Advisory Group has expressed throughout this process around transparency, stating: ‘We do not believe that the development of the statistical adjustment methodology has been transparent enough to meet our concerns about statistical quality.’
After
agreeing to carrying out a review, the OSR director-general for regulation, Ed Humpherson, said it would focus on whether those developing the algorithm complied with the
Code of Practice for Statistics – a set of ‘gold standard’ guidelines that producers of official government statistics are required to follow. The findings are expected to be published in the autumn. We have offered our assistance to the OSR in this process.
The RSS letter also describes how two expert fellows that we put forward to help with the algorithm were asked to sign a non-disclosure agreement before they could be considered; this would in our view have prevented them from commenting on the choice of the model for some years after the exam results were released. The letter prompted a
response from Ofqual, objecting to the way the NDA was characterised. We responded robustly,
defending our characterisation.
Read our letter to OSR calling for a review.
Read the response from Ed Humpherson (18 August).
Read the
letter from Ofqual (21 August) and
our response.