To launch this new guidance, PUBLIC hosted a panel of experts from the public sector, academia and private sector advisors to discuss important considerations for evaluating digital projects in the public sector and how evaluators and practitioners can work together to drive evidence informed policy making. The panel included Tom O'Keeffe, Senior Researcher at BIT; Oliver Hauser, Professor of Economics at the University of Exeter and a Senior Special Advisor to the Evaluation Task Force; Lizzie Shelmerdine, Senior Evaluation Advisor to the Evaluation Task Force; and Theo French, Economic Advisor to MHCLG.

Now is an important time for public sector practitioners to start thinking about evaluations. The arrival of a new government gives an opportunity for project teams to start considering what has gone well, and therefore what they want to continue doing, and what has gone less well, to decide where funding should be reallocated. And this is particularly important in the technology sector. Tens of billions of pounds are spent every year on technology and innovation projects across the public sector, and we need good quality data to understand their impact and potential.

In this article, we summarise some of the main takeaways for teams evaluating digital projects from our expert panel.
 

1. Evaluation design needs to come at the beginning of new projects

All of the panellists agreed on one thing: the earlier we bring in evaluation design into the project, the better our prospects are. There is often a pressure on digital teams to push ahead with rapid development, roll-out and implementation. But quicker isn’t always better and digital delivery teams need to pause to consider the direction of their projects and what success should look like. Bringing evaluators to the table early on, to develop a theory of change following a business case, enables practitioners to identify and refine a project’s goals and expectations, building a solid foundation for success. Most importantly, it offers more scope to design randomised roll-out, or other QED methods into the project design.
‍

2. Digital teams and evaluators need to start speaking the same language

There is often an asymmetry of knowledge between digital teams and those responsible for evaluating project outcomes, with economists and evaluators speaking a different language to developers and data scientists. To address this, there is a need for more cross-team collaboration, with economists and digital experts working side-by-side from project  inception. Similarly, policy teams often still need to be convinced of a project's success and the evidence needed to prove this.  Teaching evaluation methods early in the process could help practitioners become better consumers of evidence, recognising what high-quality evidence looks like, and how to communicate the value of their projects to policymakers and vie for future funding.
‍

3. Evaluations can give us insights into behaviour, where data alone can’t

Data on digital projects is often ubiquitous, but it doesn’t always tell the whole story. While digital data provides insights into how systems function, the true measure of success lies in user adoption and experience. However, the level of adoption doesn't always tell us an awful lot about the reasons for adoption and non-adoption, or the behaviours of a digital user. Practitioners are challenged with bridging the gap between raw data and user experience, connecting quantitative measures to qualitative insights on user behaviour. Theory-based evaluation methods can support digital practitioners to understand user behaviour and adoption by identifying signifiers of that desired behaviour and uncovering the observable, tangible and measurable things to demonstrate the impact of a digital tool.
‍

4. Running Randomised Control Trials can be straightforward for citizen-facing digital services

Digital projects offer an opportunity to conduct evaluations using RCTs (usually considered the ‘gold standard’ of evaluation). Panellists agreed that these forms of evaluation are very similar to the kind of  A/B testing, multivariate testing and path testing that are very common in the private tech sector (the example of Uber’s frequent A/B trials was given). Although running RCTs may seem daunting initially, and practitioners may not find them easy, the value and insight that comes from running trials is immense. Where the digital service being evaluated is a citizen-facing service (rather than a complex programme, or fund, etc.), it can be straightforward to design an RCT, in the form of an A/B test. 50% of the users might be assigned to the new digital service when they click a button, and 50% to the old service. Or hundreds of other A/B combinations could be designed in the same way. With such testing approaches so common at Uber, Meta, Google and other technology companies, the public sector could do well from learning from these methods and embedding rapid RCT tests.
‍

5. Where overall trials are not feasible, consider looking at ‘micro-trials’

Another theme discussed was that in some cases, it may not be feasible, or even desirable to conduct an old-vs-new RCT in the way described above. It might be that you don’t want to stagger the roll-out of the new service, or that you are arriving at the evaluation midway through. In these cases, the panel pointed out that there is value in running ‘micro-trials’, which test specific features or product combinations. Although this might not allow direct comparison between the entire-old service and the entire-new service, it allows teams to better understand the causal contribution of different features. This is in line with the A/B testing approach recommended above: running trials of different product combinations and capabilities allows you to build up a more nuanced understanding of what is working, and how the service connects to different outcome areas.
‍

6. Digital teams need to pause and think about evaluation before rushing off quickly

Developing digital projects requires agility, but too much flexibility without a clear framework for evaluation can lead to projects that lack direction or measurable outcomes. Often the race to keep up with technology advancements can overshadow the fundamental question: “Are we implementing the right innovation at the right time?” As a result, it is important to ensure that projects align with a long-term vision of impact and measurable end-result, even if it means taking a moment to slow down and evaluate. This means breaking down goals into achievable time frames and narrowing the scope to targets that are both measurable and meaningful. One piece of advice given on the panel was to make sure that Theories of Change are flexible documents that are reviewed and updated regularly: ‘a compass, rather than a map’.
‍

7. If teams build a culture of collecting data by default, evaluation becomes much easier

It is important to build a culture of evidence-based decision-making in the public sector.  Senior digital leaders and policy makers, especially, need to champion evaluation and evidence-based decision-making by setting requirements that demand robust evidence to justify the success of digital projects. Evaluations should not be seen merely as a box-ticking exercise but a meaningful process to understand what works, what doesn’t, and why. The panel agreed that there is still more work to be done here: for digital teams to understand the importance of these processes, and for evaluation teams to develop more flexible and agile approaches.

We wanted to thank our fantastic panel for their insights and contributions, and all of those who joined us for the day. We look forward to continuing the conversation about how to use evaluation tools in technology and innovation projects, and hope our new guidance can help.