Our recommendations for addressing gaps in poverty data

---

This project finds that the UK has valuable data on poverty, but it is often hard to access, incomplete, and difficult to use. As a result, researchers, charities, and policymakers cannot fully understand poverty or design the best solutions for alleviating it. 

To address this, the report sets out three main recommendations for government and statistics producers: 

A major problem is that the most useful data—especially government administrative data—is too difficult to access. Many researchers, particularly outside academia, are effectively locked out of current access routes. 

Our recommendations in this domain centre around: 

• Reforming and clarifying laws and policies to broaden who can access data. 
• Making approval processes faster, clearer, and less bureaucratic. 
• ​Expanding alternative access routes to accessing data, especially for non-academic researchers. 

Better access would allow more organisations to answer important questions about poverty, leading to better-informed policy and support for vulnerable groups.
 

• For the full set of recommendations in this area read below

  BACKGROUND: The existing system for data access excludes many would-be data users from using linked administrative datasets, which promise to be the most illuminating sources for advancing our understanding poverty and how to alleviate it. Currently, access to those datasets is restricted to accredited researchers working on accredited projects in TREs. Accreditation for projects can take months or even years to complete, and the process tends to be opaque and frustrating for researchers. Security protocols are understandable given the sensitive nature of these data, but they also come with a cost that must be accounted for: many projects that are reasonably within the public interest are unable to get off the ground. 

  We applaud the work done by entities like Administrative Data Research UK (ADR UK) for their groundbreaking work to open up access to these powerful datasets, especially for academic research purposes. We also applaud and acknowledge the fact that statistics producers themselves are involved in those efforts. We spoke to many researchers who spoke highly of the considerable progress that has been made in recent years, with examples like the Longitudinal Educational Outcomes (LEO) dataset being praised frequently.   

  At the same time, we recognise that there is no equivalent to ADR UK focused specifically on non-academic data users. The anti-poverty research space would greatly benefit from a dedicated champion that can drive forward their unique data access needs in a way perhaps parallel to ADR UK’s work for academic researchers. 

  1. Legislators and the UKSA should clarify what counts as “research” in the context of the Digital Economy Act 2017. In our conversations with people involved in accrediting research projects, there was uncertainty about the exact scope of what counts as “research” for the purposes of the legislation. Specifically, non-academic uses of the data were not clearly understood to be allowed (or definitively disallowed) by the Act, even though many are likely to be in the public interest.1 Although we take no position on the legal interpretation of the Act as it currently exists, as a matter of policy a restrictive framework strikes us as unnecessarily exclusive. Legislators and the UKSA should therefore clarify and/or broaden the definition of "research" to be more explicitly inclusive of non-academic projects that may want to use data held in TREs. 

  2. Legislators and the UKSA should take a leading role in ensuring a broader set of access pathways are available in the future. Producers often told us that they would like to be more open with their publication and sharing of data, but that they are constrained by the narrow range of existing legal permissive gateways for doing so. Policymakers have a role to play in opening up new gateways and clarifying the limits of existing ones. 

  3. DWP and HMRC should scale up indirect pathways to accessing their administrative data. There are already promising models for this, including DWP’s Employment Data Lab—which lets charities submit their own data to DWP staff who will create a matched comparison group and light analysis using administrative records—or HMRC’s Datalab, which allows researchers to propose novel analyses using HMRC data. These departments should think more ambitiously about the scale and remit of these models, with an aim to use them to better leverage the full potential of their internal data. 

  4. DWP, ONS, and HMRC should collaboratively scope the development of an “intermediary” model to help more people benefit from the UK’s best datasets. The purpose of this intermediary would be to help bridge the chasm of data access between civil society and academic researchers who can work on projects using linked administrative datasets held in TREs. As noted in the write-up of our Improving Data Access Workshop, this could look like a set of trusted organisations or universities acting as an agile analytical force to support enquiries from outside researchers. However, more work is needed to understand exactly what the possibilities are and what would be most effective, and the scoping should directly involve representatives from civil society and academia. 

  5. The ONS Secure Research Service (SRS) and other TREs should work with departmental data owners to speed up approval timelines for projects seeking to gain access to the data they hold. One option here might be to fast-track applications from certain organisations or applicants with proven track records of safe handling of data. Continued development of public, synthetic versions of the data will also be welcome to enable applicants to better understand the possibilities of the datasets and appropriate methodologies that can be applied to them. Additionally, there are likely “indirect” ways of opening up this data in the public interest without compromising on security, as suggested in Recommendation 3. 

Even where data exists, there are gaps in what we know, some of which are especially critical for certain topics and groups. We highlight several areas that are clearly in need of improvement, such as homelessness, disability, ethnicity, and wealth. 

Here, our recommendations focus on: 

• Encouraging producers to take stronger leadership roles to improve data quality and consistency 
• Creating working groups with expert users and producers to solve complex issues like disability measurement 
• Continuing advances in the use of linked administrative data and new data sources 
• Working with private and third-sector partners to fill gaps where possible. 

Without good data in these areas, key forms of disadvantage remain invisible, making it harder to target policies effectively. 

• For the full set of recommendations in this area read below

  BACKGROUND: Although many gaps covered in this document are systemic in nature, there are also many issues that affect research in acute ways, relating to specific topics. For instance, in our interim report, we highlighted that areas like homelessness, migration, gendered poverty, wealth, and disability struggle with data gaps that are more fundamental to their domains. Rather than listing all those gaps here, these recommendations cover large gaps that impact fields in a broader way. Readers should consult and contribute to our Data Gaps Explorer, which highlights specific gaps mentioned by civil society researchers.  

  6. Producers like the Ministry of Housing, Communities, and Local Government (MHCLG) and others should take a proactive role in stewarding improvements in areas like homelessness that suffer from poor data quality and standardisation. Statistics producers and government departments like MHCLG are the best placed to encourage standardisation of this data by mandating that local authorities collect and report data in more consistent ways. This would ideally be aimed at producing datasets that are linkable and generally more legible to researchers than they are now.  

  7. ONS and DWP should lead on the establishment of a permanent working group for the purpose of designing and implementing solutions to shared problems with disability data. The need for a permanent working group here is particularly strong because there are longstanding debates about conceptual issues that impede research. For example, there is little agreement about the proper way of identifying disabled people in surveys, which are themselves limited in their ability to reveal information about small populations. However, our experience of convening disability researchers for this project clearly showed that there is willingness to positively and pragmatically work together on these issues. A co-productive approach involving civil society and academics is therefore preferred. 

  8. ONS and DWP should launch and deliver a scoping project on additional costs associated with disability, ideally within the framework suggested in Recommendation 7. This is a conceptually challenging topic, and one objective should be to determine whether determining extra costs is theoretically and practically feasible given existing data, and if not, what would be necessary to enable it.   

  9. ONS should lead a working group to develop and scale innovative ways of using existing data on wealth and assets to complement what is available in the UK’s large-scale surveys. Possibly building on the work being done by Smart Data Foundry and others, data from a wide range of private-sector partners could be leveraged to produce highly comprehensive pictures of wealth in close to real time. Particularly for individuals and households at the bottom of the economic ladder, having a high-frequency view into fluctuations in savings or spending behaviour can help us to understand patterns of disadvantage and craft better policies that can give people a stronger buffer in times of need. Indeed, producers should generally think creatively about how to work with private and third-sector partners to fill gaps in official data. 

  10. Producers should continue to prioritise efforts to use administrative records where there are gaps in or limitations to survey data, including poor coverage of certain populations or a lack of granularity rooted in small sample sizes. Initiatives like the FRS Transformation at DWP are welcome*, and we hope that further work is done to maximise the potential value of administrative records elsewhere. For instance, people living in institutional settings tend to be excluded from surveys, and administrative records may give us the best chance of understanding living standards for these people in detail.   
      
      *We note that there are limitations to administrative data that should not be overlooked, including that they can suffer from measurement error and bias in ways that mean they cannot fully replace surveys and that they will struggle to capture information about important questions like perceptions of wellbeing or deprivation measures. Therefore, while administrative datasets are immensely promising, producers must be clear-eyed about them.

We heard from many researchers that although the UK holds a vast and useful range of data that relate to poverty, it is often hard to navigate, with key data hard to find, understand, or use for some users. 

To help address this issue, our recommendations concentrate on: 

• Investing in clear, up-to-date documentation and data catalogues 
• Providing better user support and guidance, including about what data is available 
• Creating regular forums and engagement structures for users 
• Treating data more like a user-focused product, with ongoing investment in usability. 

The benefits of improvement here will apply to both users and producers, as they could reduce wasted time and effort on both sides and enable more people to use data effectively to inform decisions and research. 

• For the full set of recommendations in this area read below

  BACKGROUND: In interviews with expert users and in our workshops, we heard that the reliability of documentation was inconsistent, even for major datasets held in the ONS Secure Research Service. Documentation was often non-existent or out of date, meaning users could find that variables had changed or been added or dropped without explanation. This is a serious point of friction for users that can slow down research or certain analyses being abandoned. 

  The issue is compounded by a perceived lack of support for queries. One person we interviewed said that, even when they did get personalised support from the ONS, they did not have confidence that the support worker understood the issue or data themselves.  

  We also heard repeatedly from users of all technical backgrounds that the landscape of datasets held by government relevant to poverty-related research is challenging to navigate. Data is not always easily discoverable for those without expert backgrounds. There is currently substantial duplication of effort trying to find and use data from disparate sources. 

  During our workshops, both users and producers showed clear appetite for deeper and more regular engagement. This would foster transparency and mutual understanding and let users meaningfully feed into producers’ work. However, current engagement is often ad hoc and infrequent. Third sector organisations also struggle to coordinate their engagement, due to lack of expertise and resources which makes it harder for producers to prioritise user needs  and to handle feedback that is not neatly packaged into specific requests.  

  11. ONS and DWP should prioritise and invest in producing, updating and communicating data documentation across the board. In particular, producers should provide support staff with time protected for handling data-related queries, and clear, straightforward, and well-advertised mechanisms for users to seek support. Particularly in secure environments that are already time-consuming to access, this should be a priority. There is likely also scope to develop new AI-supported search and support tools to help field user queries. Additionally, for more advanced users, a welcome improvement would be more thorough and explicit documentation of the statistical methodologies that are applied when, for instance, variables are imputed in surveys. 

  12. ONS and DWP should create their own unified data catalogues to collate and signpost information about the datasets they make available, how to access them, and the variables they contain, alongside examples of analytical work done using them. This should be as user-friendly as possible, to account for the diversity of users that may engage with it. Good examples from other producers include the Department for Education’s data catalogue, HM Land Registry’s land and property data catalogue, and Administrative Data Research UK’s (ADR UK) data documentation.  

  13. DWP should broaden the scope of Stat-Xplore or build supplementary tools to improve the usability and findability of DWP's data. We heard from many users that the data provided via Stat-Xplore is valuable, but there is desire for it to be expanded. For instance, DWP currently use Stat-Xplore as a public-facing tool for the exploration of official statistics only, but anti-poverty work would benefit from a more general window into aggregated data held by DWP. Moreover, usability could be improved by making it easier to both find and link to datasets, and to understand what datasets contain. Creative technological solutions within Stat-Xplore or possibly the development of new systems to enable flexible linking of datasets and smoother navigation could be helpful here, especially given the large number of relevant datasets held by DWP. 

  14. The ONS Secure Research Service and other Trusted Research Environments should publish data about project applications seeking to use the data they hold (e.g. time to decision, reason for rejection, stage of the process at which a rejection is handed out, etc.). This will help applicants understand where applications are most likely to fail and help set expectations about timelines while supporting all users, but particularly first-time or less well-connected users. TREs should also publish clear timelines and criteria for approval; ensure that their timelines are reasonable; and publish details of successful applications. 

  15. Producers should invest in modern user research to improve the usability and findability of their statistics and data. They should hire dedicated user researchers to proactively develop data resources, define user groups, and do user testing. Producers should treat their statistics and data as a product, with a product owner, user research team, and development resource to make them easier to use. Organisations that do this are much more likely to see real engagement with their data and see it used. 

  16. DWP, ONS, and the UKDS should engage with and help facilitate the creation of bodies such as the institutional consortiums mentioned in the Introduction. This can be done in part by committing to ongoing, regularly scheduled workshops or conferences on topics of relevance to users of data on poverty, deprivation, living standards, benefits, etc. Departments sometimes already facilitate more structured engagement for specific datasets—the Family Finance Surveys User Conference organised by the UKDS, DWP, and ONS is a positive model that could be built on. These forums can create opportunities for feedback on specific data issues and encourage conversations on more strategic, cross-cutting, and philosophical matters relating to data on the above topics.  

Across all three areas, it will be key to promote a more collaborative environment between government, academics, and other researchers. We believe that, if implemented, our recommendations will go a long way towards building such an environment. 

For the most part, the 16 specific recommendations above are aimed at parts of government, since they tend to be best positioned to make lasting changes. Nevertheless, positive progress can also be achieved by others, like civil society organisations, academics, and other researchers.  

• For our commentary on how others can support better anti-poverty work, read below.

  For instance, the academic community may wish to reconsider how poverty-relevant research is conducted and supported. Current conceptions of “impact” are often too narrowly focused on dissemination of academic research findings. There is clear and definite value in that. Yet, in many cases, greater impact could be achieved through more agile and embedded forms of support for non-academic researchers. For example, academics have the skills and knowledge to offer hands-on support for organisations to collect, analyse, and interpret data in ways that directly benefit their work.  

  We would therefore welcome some shift towards more collaborative and practice-oriented research models, particularly those that involve closer working between academic researchers and non-academic organisations. Research funders have a key role to play in enabling this shift, including by supporting projects that prioritise practical utility and collaboration, even where these do not necessarily lead to conventional academic outputs. 

  There is also room for institutional consortiums or partnerships that bring together academics, civil society organisations, and other poverty researchers to coordinate their data needs and requests and share resources or best practice between their members. Currently, there is a coordination problem that faces the vast number of people and organisations working in the poverty space, many of which lack resources to act in a joined-up way to effect systemic change. There is, however, plenty of demand for structured, permanent forums for engagement with each other and with government—the response we received at our workshops and roundtables showed us this clearly.  

  But beyond willingness to participate, there needs to deliberate investment into that kind of infrastructure to help the sector overcome this coordination barrier. More coordinated models would allow users to better organise their needs, articulate them more clearly, and engage with producers in a more systematic way. Coordinating bodies should also act as intermediaries, connecting lived experience and frontline service insights with technical data expertise, and translating these into actionable requests for data producers.  

  As we note in Recommendation 16, some of that investment is likely to be best made by government departments that already have an interest in user engagement (like DWP and ONS, for example). It is possible that organisations like the Royal Statistical Society could fulfil this intermediary role with enough institutional and sectoral support.

Finally, we thank the Joseph Rowntree Foundation Insight Infrastructure team for funding this research and for serving as valued advisors throughout. They aim to democratise access to high-quality quantitative and qualitative data and evidence through open collaboration and innovation, working towards a more equitable and just future, free from poverty.