Evaluating the Impact of One-to-one Reading
At Chapter One, we want to create a world where all children have the same opportunities to benefit from everything that reading provides, enabling children to thrive in life. We focus on supporting young children in the UK who face disadvantage. We develop fundamental reading skills and inspire a love for reading at an early age, helping to build confidence in children who might otherwise fall through the cracks. We offer a variety of programmes but our largest programme is the Online Reading Volunteers Programme. This programme takes place on our bespoke online platform, which connects corporate volunteers with young readers aged 5 to 8, facilitating weekly reading sessions over a whole academic year
The request
Chapter One records the reading ability of children who participate in the program at the start and end of the academic year.Although they can measure the improvement the child makes over the year, they cannot measure the effectiveness of the Chapter One intervention; as it is not possible to separate the impact of Chapter One from the impact of the classroom teaching.The charity wanted help with understanding how a control group could be used to quantify the impact of Chapter One, how it could be implemented and how reading attainment data could be analysed from a randomised control group study.
The approach
A collaborative approach was taken. The volunteer spent time with the lead contact at the charity to understand the nature of the Chapter One intervention, their processes and data and the potential implications of introducing a control group. The volunteer was then able to reflect on how this information affects evaluation, research suitable study design options and synthesise these two strands into a list of suitable control group options (such as simple randomisation, matched-pairs randomisation etc.). After further discussions a preferred option was selected. The volunteer then used their industry experience and further research to provide the charity with practical advice on implementation.
The result
The charity received advice on how they could set up a randomised control group study in a comprehensive report. The report discussed; the implications of how attainment is currently measured for evaluation, different types of control group designs, ethical considerations of introducing a control group, a check list of points to consider when running a trial, practical advice on implementing a control group and analysis support. The report was supplemented by a spreadsheet which illustrated the concepts outlined in the report (such as sample size estimation, randomisation and analysis) and now serves as a tool to aid setting up and running a randomised control group study
Impact and benefits
Chapter One had considered setting up a randomised control group study to evaluate the effectiveness of our Online Reading Volunteers Programme for a couple of years. Although we had some basic in-house knowledge about running a randomised control group study, we recognised that this is such a complex area and that it required expertise to set up. The volunteer statistician that we were paired with had an enormous amount of expertise in this area. The benefits for us as a charity were far reaching and went further than the initial project objectives. For example, discussions with the volunteer statistician prompted further thoughts and work within Chapter One about tightening up on our current way of measuring reading attainment in children and exploring other ways to measure reading attainment. With regards to the benefits from the project itself, the volunteer statistician provided us with invaluable knowledge and practical ideas and tools on how to plan and implement a randomised control group study. For Chapter One, this new knowledge has meant that we feel empowered to be able to try out such an in-house study in the future – something which felt like an extremely daunting task before this project.
Volunteer: Jenny Thomas
Charity lead: Julie Bennett (Evidence & Impact Manager)