Principles to support statisticians making trade-offs in pressurised situations

In the first stage of this initiative, we published a range of case studies illustrating instances in which compromises had to be made for data to be able to inform decisions. We explored the factors that were balanced, the impact of making these trade-offs, and the lessons learned. Looking across these case studies and following discussion with a range of stakeholders, we have developed a set of principles outlining the aspects to consider when making trade-offs.  

We hope these principles are useful to statisticians and analysts for both previous and future work. The principles can be referred back to when reflecting on past instances in which compromises had to be made, in order to support the decisions made or consider alternative steps that could have been taken. The principles can serve as a useful checklist of factors to consider in future work, though we recognise that they do not provide context-specific detailed guidance. We also hope these principles can be helpful in instances when certain trade-offs may not be suitable – to help reach this conclusion and explain why.  

The principles build on and link to existing work in this space. The RSS has long called for an increased focus on user engagement and transparent communication when producing statistics, to meet the vision in the OSR’s Code of Practice for Statistics of trustworthiness, quality and value. Value depends – among other factors – on relevance to users, and user engagement is a key component of the RSS call for more public statistics. This work also builds on other government work acknowledging the trade-offs that must be made, for example guidance on the quality of statistics and sharing data in crises.    

Here we set out five principles – each related to the others – that should be considered when balancing competing demands to enable evidence-informed decision-making. 

Image illustrating an example relationship between work (time/resource) and output quality. The first dotted line indicates sufficient quality for intended purpose, which depends on the work invested. The second dotted line indicates that there will be a point at which further work makes no appreciable improvement to quality.  

Explore the principles  

 

Read the full principles   

• Principles to support statisticians making trade-offs 

  Principles	Factors to consider
  1. Understanding output purpose and stakeholder needs  You have understood the purpose of the output and the needs of relevant stakeholders, to ensure the output is proportionate and fit for purpose.	Who needs this and what for (eg internal decision-making versus for publication)?   Have you engaged with key stakeholders to understand their needs and the priorities they place on them (including separating user ‘needs’ from user ‘wants’), and used this to inform output development?  Is the output proportionate to need with respect to key domains, including, for example, timeliness, resource, certainty, data collection, level of accuracy (eg number of decimal points versus giving an idea of order of magnitude)?  Is your work proportionate to need with respect to effort input for output? At what point do your efforts no longer lead to meaningful improvements (‘return on investment’) with regards to the purpose of the output?  What sort of output quality is expected – is the output of sufficient quality to inform decision-making?      Do you have all of the information you need to be able to assess whether the output is fit for purpose?  If relevant, have you considered the relative weight likely to be given to statistical evidence versus other forms of evidence feeding into decision-making, and factored this into ensuring that the statistics are fit for purpose?
  2. Prioritisation with justification  You have considered the core aspects necessary for the output (linking back to its purpose) and have justified your choices and documented the reasons for these decisions.	Time or resource are often limiting factors in pressurised situations – given these limitations, what is the core ‘minimum viable product’ that can serve its purpose?     What is ‘necessary’ versus ‘nice to have’?  Can you take a staged approach, starting smaller and adding/improving as resource allows?  How is limited time or resource best spent, to maximise the practical utility and usefulness of the output within time or resource constraints?  Have you considered how far to go in key domains, eg with respect to: number or frequency of publications; timing of publications; resource; level of quality assurance; granularity; coverage, etc.  If the output requested is not appropriate or feasible, have you considered alternatives that could be offered, and ensured that the differences are understood?
  3. Risk mitigation You have identified the possible risks and their impact, assessed the appetite for risk, and considered how to mitigate risks.	What are the risks – have you considered both short-term and longer-term risks? Have you considered the wider context, including eg the financial, resource, legal, ethical, and safety risks?  How much appetite is there for risk in this project?   What are the likelihood and impact of these risks (eg high /medium/low)? Have you considered the extent to which decisions regarding trade-offs are likely to attract comment or controversy?  Have you considered how you can minimise or mitigate the risks?    Have you documented your work, including both the justifications behind your decisions, as well as the methodology – providing a demonstrable audit trail and enabling your results to be reproduced? Have you sought independent review, even if this is in a more rapid form than would be standard in non-pressurised circumstances?  Have you implemented processes to support staff and avoid burnout?
  4. Transparent and accessible communication  You have communicated with relevant stakeholders, ensuring that they understand the trade-offs, limitations and uncertainty associated with the output (or communicated why it is not feasible to produce the output, if relevant).	Are you communicating with your key stakeholders in ways that are appropriate and that they will understand? (eg audience of analysts versus public)   Have you identified any additional stakeholders for whom alternative communication styles might be necessary?    Have you followed best practice for communication about trade-offs, considering intelligent transparency; accessibility (eg colours, format); and the perspective (including prior knowledge and any biases) of your audience?  Have you kept stakeholders informed of progress and decisions as the work progresses?  Is your communication about the trade-offs, limitations and uncertainty clear, easily comprehensible and accessible?  Are your stakeholders aware of the limitations, the need for prioritisation and trade-offs, and the resulting uncertainty in the output?    Have you considered what wording is appropriate to explain your decisions and outputs, along with any wording that should be avoided?   Have you factored in the capacity implications of stakeholder engagement (both prior to publication as well as following)?
  5. Monitor and adapt  Where appropriate, you are monitoring the output (in terms of quality and reception) and will consider adapting it to ensure it remains fit for purpose – especially in situations where outputs are published on an ongoing basis and the context is rapidly changing.	Are you monitoring the quality of your output, as well as reflecting on the wider process (including stakeholder engagement, decision-making, communication etc)?   Are you monitoring the response to your output and working to understand the factors influencing this?  Can you refine and improve over time – as capacity allows, or as context and demands change?    Are you effectively communicating the reasons for the changes?  Have you considered the cost-benefit ratio of extra work on the output – when does extra work cease to meaningfully improve the output? Would the proposed extra piece of work change the interpretation or the decisions it is used to inform?   Have you considered the impact of adapting your output over time – including balancing improvement with a need for comparability and consistency, and understanding user needs in this regard?