Interview

Key Takeaways at a Glance

Join the conversation

This conversation touches on a theme we often explore here at Real World Data Science: the idea that the future of data science and AI will not be defined by technical capability alone, but by how well we integrate human judgment, context, and responsibility into our systems. The position paper—and RSS: Data Science and AI more broadly—is an open invitation to engage with these questions. Whether through research, case studies, or reflections from practice, there is a clear call for contributions that connect technical work with real-world impact.

As Neil suggests, the answers are unlikely to come from any single discipline or organisation. They will emerge from a broader conversation across the data science community.

Now is the time to be part of that conversation: answer RSS: Data Science and AI’s call for submissions.

Start with What We Are Trying to Achieve

Too often, data science projects begin with vague aims such as “build a model” or “forecast sales.” These are activities, not outcomes. What matters is the result the organisation is striving for—for example, increasing upsell revenue by £2M this quarter or preventing 500 contract cancellations per month through timely intervention. Asking the right questions early is essential for designing solutions that can actually be implemented. For instance, a retention model might flag 1,000 customers at high risk of leaving, but if capacity allows only 50 calls per week, the key question becomes: which 50 should be prioritised, and does contacting them actually improve retention compared to a control group?

Before writing a single line of code, it is essential to gather as much context as possible:

• What problems is the business actually solving?
• How does the model fit into the wider business process?
• Who will use the outputs, and what actions will follow?
• How will success be measured—commercially, operationally, behaviourally?
• What trade-offs are acceptable in cost, complexity, or speed?
• How will performance be monitored over time?
• What are the operational constraints?

Once the essentials are understood (to the extent they can be), the vision for the project and the success metrics must be agreed collectively. All key stakeholders—technical, operational, financial, and strategic—need to be involved in defining what success looks like. Without this shared vision, each group risks optimising for its own priorities rather than the organisation’s overall goals. Crucially, the vision should extend beyond performance metrics: it should tell the story of the problem being solved and what success will mean in practice. This shared narrative becomes the project’s guiding star. To keep it on course, data science and AI teams, working with stakeholders, must guard against scope creep and shifting success criteria, ensuring that any new requests fit within the agreed scope. Flexibility still has a place—experimentation and design changes are healthy—but only when they remain consistent with the original vision and are aligning to achieving the success metrics.

The Power of Test-and-Learn

Evaluation and monitoring must be built in from the beginning. Doing so ensures that systems are designed to capture the right metrics for monitoring, rather than scrambling to measure impact after the fact. This means defining not only technical performance measures but also organisational impact measures, all aligned to clear, measurable success metrics. These metrics should be developed collaboratively with stakeholders, and while data scientists may not set them alone, they play a critical role in shaping and challenging them where needed.

A test-and-learn approach is particularly powerful because it generates direct evidence of what works under real-world conditions. For example, a simple test-and-control design splitting customers into two groups, one acted on and one left as business-as-usual, provides incremental evidence of benefit that is far more persuasive than retrospective accuracy scores. Unlike abstract metrics, this method shows whether interventions truly drive the desired outcomes, and it allows organisations to learn, adapt, and refine strategies over time.

Ultimately, evaluation is about measuring decision performance in practice, while monitoring ensures that impact remains robust as circumstances evolve.

Map the current business process end to end, noting all user interactions and data collection points. Then overlay where the model will integrate into that process— the inputs into the model pipeline, who receives the model outputs, how they are acted on, and how outcomes flow back into the system. This makes clear both the operational impact of the model and what changes are needed for it to deliver value.

Design for Value, Not Novelty

Data science is not about building impressive models for their own sake. It is about solving valuable problems in ways that make business sense.

If a model improves accuracy by two percent but costs ten times more to run, is it worth it? The answer depends on whether those extra points translate into measurable financial impact.

Ask:

• Could a simpler model deliver “good enough” accuracy at lower cost?
• What is the marginal value of added complexity?
• Does the design reflect operational constraints?

Here, the product mindset for data science and AI practitioners becomes critical. Treating an AI solution as a product reframes the goal from “building a model” to “delivering value.” Like any product, an AI system has costs to design, build, deploy, and maintain. Its worth lies not in technical elegance but in whether the return justifies those costs. That means asking early: is the investment worth it?

One practical way to answer that question is by forecasting scenarios. Before scaling, estimate the expected impact under different conditions: a base case, a best case, and a worst case. For example, in a retention project, you might forecast incremental revenue by combining churn rates, average customer value, intervention costs, and expected uplift. This makes assumptions explicit and gives decision-makers a clear view of risk and upside. A solution is rarely a guaranteed win, but scenario planning allows stakeholders to judge whether the likely outcomes justify the investment.

Build for Adoption

Adoption must be planned from the start. Trust develops gradually, and regular check-ins with stakeholders help sustain it by keeping the project aligned with its agreed vision. These sessions are not box-ticking exercises but chances to test assumptions, surface blockers, have continuous feedback and make timely adjustments. Ultimately, a model succeeds only if people use it — so adoption depends on seamless integration into existing processes while delivering something users can see a clear benefit from.

Instead of starting with purely technical questions—such as “will I need to export this to a CSV?”—it is often more effective to begin by considering the user journey. For example, if the end goal is for users to view the results in a dashboard, that should frame the discussion from the outset. Once the user’s needs are clear, the practitioner can then work with the data engineering team to determine the most appropriate technical solution, such as the optimal data format or storage approach.

Hence it is important to ask early:

• Where will predictions appear (CRM, dashboard, alert)?
• Will outputs be delivered in tools people already use?
• What training or support is required?
• How will impact be made visible to leadership?
• How best should the outputs of the model be presented to ensure they are usable and actionable for the next stage of the business process.

Thinking about these questions early prevents the familiar fate of a technically brilliant model that sits idle.

Adoption is strongest when development is iterative. Rather than disappearing into a three-month build, teams should work in cycles: release a minimum viable product (MVP), test it with users, gather feedback, and refine. The first iteration of the MVP should be the simplest form of the product while testing the core principle of what is trying to be achieved. An MVP could be as simple as a weekly spreadsheet with a risk score; if it proves valuable, the team can then invest in automation, dashboards, or more advanced models. This staged approach reduces risk, delivers value early, and builds trust among stakeholders. Crucially, reaching an MVP quickly lets both technical and business teams see what works—and what doesn’t—in practice, instead of relying on endless planning meetings where edge cases are difficult to anticipate.

Communication is critical. Just as one study on doctor–patient interactions found that 91% of patients preferred doctors who avoided jargon [1], stakeholders respond more positively when practitioners present results in plain language. Clear explanations build understanding, and understanding builds trust. It is also important to explain, in accessible terms, how a model or tool works “under the hood,” so users can better grasp how decisions are being made. Adoption can be further strengthened by having champions within the business—trusted and respected leaders in the business area who engage end users, promote new tools, and support day-to-day use through training and guidance.

The Bottom Line

Great models rarely fail because of poor algorithms; they fail because they are disconnected from the goals, workflows, strategies, and people they are meant to serve.

To avoid the fate of the Vanishing Model, projects must begin with a clear vision — one that is co-created with stakeholders and sustained through regular check-ins. Frame every project around measurable business outcomes and define success before writing a single line of code.

Prove value under real-world conditions with well designed and measurable evaluation plans such as test-and-control approaches. Weigh technical ambition against practical trade-offs—cost, complexity, deployment speed, and maintainability. Translate precision, recall, and ROC curves into outcomes the business understands: contracts retained, revenue gained, costs reduced. And above all, plan for adoption from day one, so that predictions are not just accurate but usable, trusted, and embedded in daily decisions.

In the end, the mark of a great model is not the elegance of its algorithm but its ability to have a positive impact.

For a broader, strategic view of why organisations struggle to realise value from AI—and how leadership and structure can change the odds—check out “Why 95% of AI Projects Fail.”

Sources: [1]Allen, K. A., Charpentier, V., Hendrickson, M. A., Kessler, M., Gotlieb, R., Marmet, J., Hause, E., Praska, C., Lunos, S., & Pitt, M. B. (2023). Jargon Be Gone – Patient Preference in Doctor Communication. Journal of Patient Experience, 10, Article 23743735231158942. DOI: 10.1177/23743735231158942.

Interview: Beyond Quantification and Uncertainty in AI

Key Takeaways at a Glance

Join the conversation

RSS: Data Science and Artificial Intelligence has an open call for submissions responding to the paper.

Sylvie Delacroix’s work is a call to action for data scientists, designers, and professionals alike. We have a window of opportunity to shape AI systems that encourage humans to keep re-articulating the values they care about.

We want to hear from you. As AI tools become more integrated into high-stakes professions, how can we ensure that systems support human judgment in all its facets rather than simply optimising for efficiency?

Read the full paper here, or our accessible digest here, and join the conversation about building AI tools that truly serve people, not just processes.

Current State of Play with Generative AI

While data analyses based on text data such as NLP (natural language processing) have been around for a while, a new type of approach has quickly become widespread with the rapid rise of generative AI methods. These models, known as LLMs (large language models), have entered the everyday vernacular as all organizations grapple with how to use these tools. With the debut of ChatGPT in 2023, these LLMs have become increasingly sophisticated, trained on a wider variety of data sources and optimized for a variety of different scenarios.

The accuracy of these models depends on the quality of the training data used. While LLMs are known to hallucinate or give inaccurate answers, they tend to perform better in situations where the training data is precise and exact, with less nuance than language often carries. As a result, LLMs can produce large amounts of computer code based on large amounts of training data based on accurate computer code. Some specialized LLMs (e.g. Claude Code) have been specifically designed to generate accurate code that can access, clean, combine, analyze and visualize data. While LLMs are not perfect in code generation, they can increase the efficiency of an experienced coder.

As a result of these LLMs, less knowledge is required to analyze and work with large datasets. A user can provide a specific prompt on the business question or research objective, upload the relevant data and have an LLM provide a relevant data analysis, complete with the underlying code used to generate that analysis.

The “New” Data Scientist

As LLMs improve in their ability to quickly generate vast amounts of accurate code, what does that mean for a discipline which has prided itself on its data wrangling skills? As Davenport and Patil noted “data scientists’ most basic, universal skill is the ability to write code” (1). Data science has seen coding as a viable career path.

When a skill becomes accessible to a wider variety of people and can be automated, how does one distinguish themselves in an organization? When coding can be done by AI tools, what happens to those who are known for their coding abilities? For a discipline to be recognized as a discipline, it must have some distinguishing characteristic that defines it as different from other disciplines. In addition, disciplines become more valuable and prominent as their contribution to society grows.

So, what are the skills that data scientists have that can’t be done well by AI? While AI capabilities are rapidly increasing, we do believe there are things that may be beyond the reach of AI for the time being.

We believe that the greatest factor limiting AI is data quality. There seems to be a growing consensus of the importance of data quality as noted in Davenport, Hoerl, and Redman (3), Davenport and Tiwari (4), and Redman (5) — as well as recent Real World Data Science pieces. As the adage goes, “garbage in, garbage out”. With poor, inaccurate data used in training, the resulting AI output will also be poor and inaccurate. A related fear is that, as AI models start to use AI generated content as a data source, a recursive loop happens that degrades the quality of any AI output (6).

Discussions of data quality are limited in books, university courses and training programs. When they do occur, they are restricted to the question of “are the data right?” and discussions of data cleaning. Data cleaning is often focused on eliminating outliers or invalid points. While there is often a reason to remove invalid points, those outliers can sometimes be the source of valuable insights.

However, data quality is much more than data cleaning or checking to make sure the data are accurate. There is an element of contextual understanding and process knowledge that enables the data scientist to properly prepare the data for analysis. We are skeptical of AI’s ability to fully understand context and the nuances of assumptions that go into data analysis. In an earlier piece, Jensen provided some examples of the limitations of AI when it comes to proper data cleaning (7). For any set of data, subject matter knowledge of how the data were collected and what they represent is crucial to a proper analysis.

This creates an opportunity for data scientists to become more valuable. By employing probing questions to better understand the context of the data, they will be in a better position to identify data quality issues and ways to improve the data quality, thus leading to better model output.

Conclusion

As coding becomes easier in an AI-enabled world - where anyone can code and analyze data - the skill set of a data scientist becomes less unique. Data scientists were once in high demand because they set themselves apart as coding wizards who could wrangle large datasets and extract insights. To remain successful and continue to deliver value, data scientists must now pivot their skillset. The real limiting factor in successful data science is data quality. A renewed focus on owning, improving and governing data quality will not only strengthen outcomes but also provide future job security and increase the value data scientists bring to organisations.

Note that this article is based on the following paper and contains some of the same ideas: Hoerl, Roger W. 2025. “The Future of Statistics in an AI Era.” Quality Engineering, published September 10, 2025.

You can find out more about the Real World Data Science stance on data quality from our article ‘Why We Should All Be Data Quality Detectives’.

Two Foundational Elements of Data Science

“Data Science” contains two essential components, “data” and “science”. The field of Data Science requires holding both components in equilibrium. Data is the raw material molded in the service of Science. While Data is first, Science is no less important. Data is the foundation and Science gives it purpose.

What do we mean by Science? We’re referring specifically to the scientific method as an approach to gain knowledge. It is the process of formulating ideas and hypotheses about the world around us and collecting data to determine the validity of those ideas. By hypotheses we’re not limiting the definition to strict statistical hypothesis tests, but rather the general process of formulating a research question, gathering appropriate data and advancing human knowledge, regardless of the statistical techniques or machine learning algorithms employed. Science, at its core, is about using data to gain insights and understanding about the complex universe we inhabit.

The scientific method has a long history and is generally defined in terms of steps such as these noted in Wikipedia (Scientific Method, 2025) as:

1. Characterizations (observations, definitions, and measurements of the subject of inquiry)

2. Hypotheses (theoretical, hypothetical explanations of observations and measurements of the subject)

3. Predictions (inductive and deductive reasoning from the hypothesis or theory)

4. Experiments (tests of all of the above)

The relationship between Data and Science is cyclical. Performing good science requires gathering good data, informed by proper experimental design techniques, which in turn requires appropriate analysis and interpretation in the context of the science. And good research (science) often generates more questions than it answers, giving way to the need to gather more data, and so on. As such, data science is more than just confirming hypotheses or generating insights, it becomes the application of the scientific method.

The scientific method should be the scaffold supporting what data scientists do. While data scientists come from a variety of backgrounds, many have more training in computer science than statistical methodology, and have more experience in software tools than they do in executing the scientific method.

In an influential, relevant paper Shmueli (2010) described two major types of statistical modeling, I) explanatory models which attempt to determine causal effects, and II) predictive models which seek accurate predictions. While predictive models can lead to understanding and possible explanatory models, explanatory models tend to be preferred by those seeking more scientific explanations for phenomena.

Data Modeling Culture Versus Algorithmic Culture

Leo Breiman in a famous paper described two paradigms: I) the data modelling approach, which assumes the model that generates the data, and II) the algorithm approach, which relies on flexible methods without making assumptions about an underlying data generating model or how the data are generated. Breiman(2001a) felt that the statistics discipline was missing out on opportunities by focusing more on data modeling approaches and not using algorithmic approaches. He practiced what he preached, developing new algorithmic approaches and encouraging the field to increase its focus on algorithms. For example, he introduced Random Forests, starting a cascade of more algorithmic approaches to modeling (Breiman, 2001b).

This wise counsel from Breiman encouraged those working with data to build more expertise in algorithms, promoting the algorithmic culture as a way to harness the power of new computational techniques. We would equate Breiman’s algorithmic approach with a greater focus on the Data side of Data Science and the data modeling approach with a greater focus on the Science side of Data Science.

Balancing Data and Science

Just as pendulums slowly swing back and forth, so too has the pendulum swung too hard towards predictive accuracy (the Data side) at the expense of contextual interpretation (the Science side) . This pendulum swing is evidenced by the growing demand for explainable ML methods (see Alangariret al. 2023 as an example) . One such method is the use of Shapley values to elicit and rank the most important features in a ML model (see Rozemberczkiet al 2022 for an introduction). It seems ironic that, in the rush to gain model accuracy with sophisticated models containing hundreds of features, end users of the models still want something they can understand and explain. In other words, they still want scientific knowledge and understanding of cause and effect, even for complicated problems.

So what is the best approach from a scientific perspective? Throw as many features into a model that you can think of and see which ones show up to be the most important? Or is there some thought and care that can go into feature selection, considering what might be important given your knowledge of the science behind a problem?

We’re not suggesting that it is bad to include many features in a model. We’re suggesting that considering the context of the problem can provide insight on features that might matter. Of course, we don’t want to jump to conclusions on what we think is important and miss opportunities to learn. We seek to maintain some balance between using our previous knowledge and experience while not increasing the risk of confirmation bias in the feature selection process.

In software engineering, there is a well-known warning: “premature optimisation is the root of all evil”. The same applies in Data Science. Too often, teams rush to optimise models, tuning hyper-parameters, stacking architectures, and searching for marginal gains, before clearly defining the scientific question or validating whether the data and assumptions are appropriate. This tendency leads to models that are mathematically elegant but scientifically ungrounded. Optimisation should follow understanding, not precede it. A model that captures the right question with moderate accuracy is far more valuable than one that optimises the wrong target to perfection. This limitation of models is reflected in the famous aphorism “All models are wrong, but some are useful,” most commonly associated with the British Statistician George Box, who wrote (Box 1976):

“Since all models are wrong, the scientist cannot obtain a”correct” one by excessive elaboration. On the contrary, following William of Occam, he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist, so overelaboration and overparameterisation is often the mark of mediocrity.”

Is it okay to use black box models where the model accuracy is paramount and ignore the explainability of the model? Yes, for some problems. But should we use black box models for all problems? No. The key to being a good data scientist/statistician is to recognize when one provides more value than another and use the best approach for the problem at hand.

So how does one give more attention to the Science side of Data Science? It starts with more attention on the question of interest. It doesn’t matter so much the type of question - whether it is a research question, a business problem to solve, or something sparked by curiosity. And it is often more than a single question. Often, it is not a single question but a series of cascading questions, each one digging deeper to get at the root causes. To manage this complexity effectively, it helps to adopt a modular approach, structuring analytical work into well-defined, interlinked components that mirror the scientific process. Each module focuses on a specific purpose: formulating and refining hypotheses, understanding data provenance and quality, developing and validating models, and translating findings into meaningful actions. Such modularisations keep the process transparent and iterative, prevents premature optimisation, and ensures that model development remains anchored to the underlying scientific inquiry rather than drifting towards technical over-engineering. With this increased attention, we believe sampling methods and experimental design will continue to be fundamental.

Here’s one example loosely based on our work experience. A business executive has some reports that show an increase in turnover at their organization, which is driving up hiring and recruiting costs, making the company less profitable. We find that the turnover is higher for those who are newer to the company, which leads to the question: “Why these new ones?” This leads to an additional hypothesis that perhaps these newer employees are not getting the leadership support they need, which leads to questions about the effectiveness of leadership training programs, which in turn leads to questions around how we measure the effectiveness of training programs. By continuing to ask questions, we can get a more targeted effort at a root cause and thus increase the impact of our work.

With the availability of many algorithms and approaches that are able to process large amounts of data, it can be tempting to gravitate towards them. When teaching analysis/ applied statistics courses, it is important to look beyond the methods and consider the overall aim. There are a set of frameworks that have been around for a while that can be helpful in finding the right balance. One example is PPDAC (Mckayet al. 2000) that emphasises all the steps beyond the modelling part and the importance of considering them all. Using such frameworks can help decide whether a black box approach is suitable in the situation or whether this won’t achieve the overall intended aim.

Finding Balance

So how do we ensure a good balance between Data and Science in Data Science?

One way is to ask “so what” with any analysis that you do and any model that you build. Ideally, you would ask that at the beginning of a project to reduce wasted effort, but it should be clear how the analysis output will be used. And it would not be sufficient to say “so that we can publish the output in a paper”. You have to think about the impact of the analysis. Will it change a decision that is being made? Does it create a new insight that can be acted upon? Does it lead to a process improvement or a new product innovation? Does it lead to a new way of running an organization? Data science that doesn’t lead to some action or insight is just computation for computations sake. Vance et al. (2022) provides additional resources and advice for how to ask good questions.

A second way is to consider the potential explanations and meaning behind any model. Don’t become too enamored with the predictive accuracy of the model (which isn’t inherently a bad thing) at the expense of asking whether there are potential scientific explanations based on the features used in any model. Use a predictive model as a starting point for digging deeper and finding a smaller set of features that provide deeper insight on potential causal relationships to explore.Sometimes a simpler model that is easier to “explain” or one that uses trusted data provides more value than the latest and greatest algorithm.

A third way is continuing emphasis on the reproducibility of the results. Clean code, documentation of results, version control of analysis code and open sharing of the code with its underlying assumptions are best practices to ensure that others can replicate the findings of any data science output. Sassoon (2023) provides additional guidance for ensuring reproducibility and transparency of results.

A final way to find the balance is to better understand how the data are generated. Hoerl (2025) touched on this issue in calling for statisticians to focus more on data quality. We believe this advice to be equally relevant for data scientists. By recognizing the crucial importance of the data generation process, data scientists will better be able to use the right data that matches the problem of interest and push for changes as needed to ensure high quality data.

Conclusions

We encourage data scientists to live up to their name and become experts in both Data and Science. As they find a proper balance between those two areas, their impact and influence will increase and the quality of their rigorous work will stand up to scrutiny and advance human knowledge.

References

Alangari, N., Menai, M. E. B.,Mathkour, H., &Almosallam, I. (2023).Exploring evaluation methods for interpretable machine learning: A survey.Information,14(8), 469.

Box, George (December 1976). “Science and Statistics”. Journal of the American Statistical Association. 71 (356): 791–799. doi:10.1080/01621459.1976.10480949. Retrieved 2025-11-28.

Breiman, L. (2001a). Statistical modeling: The two cultures (with comments and a rejoinder by the author).Statistical Science,16(3), 199-231.

Breiman, L. (2001b). Random forests.Machine Learning,45, 5-32.

Hoerl, R. W. (2025). The future of statistics in an AI era.Quality Engineering. Advance online publication.

Hyde, R. (2009). The fallacy of premature optimization.Ubiquity,2009(February).

MacKay, R. J., & Oldford, R. W. (2000). Scientific method, statistical method and the speed of light.Statistical Science,15(3), 254-278.

Rozemberczki, B., Watson, L., Bayer, P., Yang, H. T., Kiss, O., Nilsson, S., & Sarkar, R. (2022). Theshapleyvalue in machine learning. InThe 31st International Joint Conference on Artificial Intelligence and the 25th European Conference on Artificial Intelligence(pp. 5572-5579). International Joint Conferences on Artificial Intelligence Organization.

Sassoon, I. (2023, November 6). How to ‘open science’: A brief guide to principles and practices.Real World Data Science. https://realworlddatascience.net/foundation-frontiers/posts/2023/11/06/how-to-open-science.html

Scientific method. (2025, November). InWikipedia.

Shmueli, G. (2010). To explain or to predict?Statistical Science,25(3), 289-310.

Vance, E. A., Trumble, I. M., Alzen, J. L., & Smith, H. S. (2022). Asking great questions.Stat,11(1), e471.

The Appetite for AI in the Third Sector

Q: What led you to start working on Knowbot in the first place?

I used to be a business entrepreneur with several tech and fintech businesses. That came to an end when I sold those businesses and then, when Covid started, it seemed like there was an opportunity to give something back. I founded TestRAMP, my first nonprofit, and that experience was so enjoyable and so productive. Once the pandemic eased, we had a foundation in place but no clear mission. Then ChatGPT launched, the world got excited about large language models, and we thought: could we use our tech backgrounds and some funding to build something for the social good?

Knowbot grew out of a simple question: Is there an appetite within the world of nonprofits for LLMs —and is there a use case for them? We needed a very simple, low-risk, easy-to-grasp AI use case that could act as a gateway for organisations cautiously exploring LLMs. Knowbot became that gateway.

We pushed it out to some of the non-profits we were already working with and it’s been interesting. There is some appetite for it, and the appetite is increasing. We use Knowbot as a jumping-off point to start a conversation about AI.

Q: Are there patterns in which charities are more open to adopting AI?

The biggest differentiator isn’t size or budget — it’s culture. We are finding it most productive to work with medium-to-large nonprofits with a scientific or research-oriented culture, where the internal decision-makers tend to “get” AI more quickly. Because our single biggest challenge, far and away above any technical or scaling challenges, or anything to do with IT or LLM, has been accessing the right decision-makers inside organisations.

The sector is understandably cautious and we are new kids on the block — Knowbot didn’t exist a year ago. And we have realised that word of mouth recommendations will be key to our growth: credibility has to be built one relationship at a time with the right nonprofit partners.

Designing for Maximum Ease

Q: How did you approach technical deployment?

From day one, we knew getting anything onto a nonprofit’s website would be difficult. So, technically speaking, we’ve made it as easy as we possibly can. Knowbot runs almost entirely on our servers. On a partner’s website, it appears as a javascript button in the corner of a user’s screen which when clicked loads a small Knowbot window, where they can ask questions. Knowbot loads its interface from our servers in Frankfurt.There’s no client-side coding or backend integration required. That was deliberately done to allow a very straightforward deployment, and it should only take new partners a couple of hours to implement. We’ve recently made an even simpler option: some partners now just link to a branded page that looks like their website but which actually exists on our servers.

Behind the scenes, Knowbot is written mostly in Python and uses LLMs from Anthropic, OpenAI, Meta, Google, and Perplexity. We don’t develop our own models — few organisations on Earth have the budget for that. Instead, we “build the car around the engine,” and it’s a slightly different car for each non-profit that we work with.

Ethics by Design: Domain Restriction as a Safety Mechanism

Q: As an AI-for-good nonprofit, how do you approach issues of risk, bias, and governance?

With nonprofits, we are hyper-aware of these issues. Some of the organisations we work with handle sensitive information, so we need to be sure we’re not introducing new risks. With a non-profit that is working in healthcare, for example, if there is a possibility that Knowbot may be used for medical-adjacent questions, then we need to think carefully about whether that is something we should be doing and, if so, whether we can do it safely.

One major choice we made early on was domain restriction. Whereas most of the big answer engines out there, like ChatGPT, search the full public internet, Knowbot will only use its internal knowledge and specific website(s) upon which it’s based (i.e. the non-profit’s own website(s)). That means the knowledge is curated and the nonprofit knows exactly what information Knowbot can draw on. That dramatically reduces the risk of hallucination, misinformation, or unsafe advice.

We also adjust our prompts continually based on feedback. For example, we hadn’t anticipated that users would ask questions like “Who are you?” or “What is Knowbot?” Because the model had no context, it responded unpredictably. So we now require partners to include a “What is Knowbot?” page on their site, which Knowbow can reference.

Evolving with the Technology

Q: What technical challenges have you encountered?

I think we have been really lucky in terms of timing. To do what we’re doing now five years ago would have been completely impossible, because the LLMs weren’t there. From an infrastructure perspective, newer services such as cloud hosting tools like Render now let us deploy servers in minutes. That’s just a breath of fresh air. It takes away a lot of the operational heartache. And coding has become dramatically easier—AI assistance means we can build things now that would have taken us weeks before.

We’ve also been lucky in that LLM technology has improved fast enough that we’ve been able to incorporate new functionality almost as quickly as our nonprofit partners have requested it. For example, we now allow partners to restrict Knowbot to particular sections or topics within a website, or to sit across multiple websites. This has only become practical as models and retrieval tools have matured. Development has become faster, too: for example, we can now submit an entire codebase into our coding LLMs and ask questions about it. By comparison, when ChatGPT first launched we could only submit a small section of a program at a time.

Impact: What Knowbot Is Changing

What impact have you seen so far?

There hasn’t been anybody who has stopped and decided that it’s not for them, which is encouraging. Most of our nonprofit partners have started small — monitoring answers, getting staff comfortable — and have then expanded deployment. Feedback has been invaluable and very positive.

We can measure impact partly by usage volume, but more importantly by value. Getting 1,000 questions about “where is the ice cream stall?” on a venue website is fine, but the real impact is when researchers or decision-makers can extract complex information from, say, a conservation charity or a healthcare resource site. That’s where AI becomes transformational.

We want to add more value with more nonprofits and other for-good actors. We think there are some obvious things that should be done more generally. Whether we do them through Knowbot or whether we flag other suitable tech partners isn’t clear yet, and probably isn’t that important. But, for example, long-term, we would like to see tools like Knowbot become standard on sites like NHS.uk, gov.uk, NICE, and others. These sites hold high-quality knowledge but can be very difficult to navigate due to the sheer volume of information they contain. Traditional on-website search tools simply aren’t up to the job and LLM-based retrieval is a natural upgrade. The sooner these sites start adopting tools such as Knowbot, the better information people are going to get. Whether that means members of the public, or whether it means professionals that are looking for technical advice, the biggest wins come from the right information helping people make valuable decisions.

Advice for Nonprofits Considering LLMs

What advice would you give nonprofits thinking about adopting LLMs?

Start small. Don’t try to design a global solution from day one. There is so much hype around AI — some justified, some not — that it feels like a high-risk decision. LLM AI is still new and evolving rapidly. Understand that you don’t know what you don’t know, and be prepared to experiment on a small scale, before building out.

Use low-impact tools first. Let different teams build familiarity. Learn what the models can and can’t do in your context. Build internal confidence gradually.

And for practitioners inside nonprofits facing resistance: the conversation is the same one we have externally. Be clear about risks, mitigation, and the fact that this is a learning process. Build trust.

Q: Where do you see the next big opportunity for AI in the public interest?

We’re at the beginning of another new chapter in generative AI. Until now, most LLMs have been about retrieval and synthesis. The next transformational phase is agentic AI: systems that can do things - take actions autonomously, or semi-autonomously. That will be incredibly consequential for society, with new risks and huge potential benefits. Getting that right is absolutely essential. Future technology aside, there remain enormous public interest opportunities even for today’s tech. There is always a lag between new technology and its adoption and implementation.

Q: Anything you’d like readers to know about Knowbot?

Yes! Come and talk to us. Every conversation teaches us something new, whether or not the organisation ends up using Knowbot. We’re eager to collaborate with nonprofits and with tech companies building the next generation of models.

Mike Hudson is an entrepreneur in technology & electronic markets. He now uses his expertise to help solve social problems. Mike founded TestRAMP, a pandemic nonprofit social market described as a “major contribution to Covid PCR testing & genomic sequencing” & donated its £2.4mn profits for charity. Mike is a Fellow of ZSL & adviser to its CEO. He is an honorary Research Fellow at City, University of London. Mike is a member of the Responsible AI Institute. He is a Foundation Fellow at St Antony’s College, University of Oxford.

Why this matters now

The authors warn that the problem becomes even more serious as we move toward agentic AI systems—multiple AI agents interacting and making decisions together. If one system miscommunicates uncertainty, the error may ripple through an entire network.

To address this, the authors propose shifting away from trying to algorithmically “solve” uncertainty, and instead enabling professionals themselves to shape how AI expresses it.

In summary

This paper argues that if AI is to genuinely assist professionals, it must go beyond quantification. Numbers alone cannot capture the ethical, interpretive, and contextual uncertainties that define professional practice. Instead, AI should help preserve and enrich human judgment by communicating uncertainty in ways co-designed with the communities who rely on it. AI should not just be accurate —it should be appropriately uncertain.

The Case for Data Quality

If you search for “data quality disasters” on any search engine, you will find many results. Similarly, literature on data quality measures offers abundant advice. But within the scientific research community, data quality is often ignored. For example, how often have you encountered the term “data quality” in the guidelines when submitting or reviewing an academic paper? We would venture to say, hardly ever (or never).

This is puzzling, because high-quality data (i.e., data that is fit for purpose) is essential; without it, results become almost meaningless. Data serves as the foundation of our work. So why isn’t its quality given the prominence it deserves? Why aren’t we, as statisticians and data scientists, advocating data quality more vocally?

Recent publications may shed some light: it appears that “Everyone wants to do the model work, not the data work” [1]¹, and that statisticians may feel uneasy with elements that are not easily quantifiable [2]². Or perhaps we are all guilty of “premature enumeration” (as Tim Harford puts it), rushing into data analysis without having a good look at the data first. Whatever the case, data quality work or “data cleaning/wrangling” is not seen as fun.

For us, as self-confessed “data quality detectives”, the reverse is true, and we began the workshop by reframing data quality not as a tedious chore, but as an empowering and even enjoyable part of the analytical process. We spend hours looking at the data, enjoying the delayed gratification of finally getting to (trustable) results.

In Rosemary’s case, her attitude was shaped by key experiences early in her statistical career. Her doctoral research focused on developing methods to automatically classify magnetic resonance spectra of human leg adipose tissue based on diet—specifically distinguishing between vegans and omnivores. The study recruited 33 vegans, while the control group included 34 omnivores and 8 vegetarians, primarily staff from the MRI unit at Hammersmith Hospital. With limited experience at the time, she began experimenting with various techniques, starting with k-means cluster analysis. Although she hoped the clusters would reflect dietary groups, the analysis instead produced two distinct clusters—one containing just two spectra and the other containing the rest. After consulting colleagues, she learned that the two outlier spectra had been acquired using a different protocol and were mistakenly included in the dataset. While she may have identified the error later, catching it early saved her several weeks of work — and won her some kudos with colleagues.

Detective Work at the Tables

During the workshop, we split into six groups to investigate two questions: Why does data quality get overlooked? What strategies can raise its profile?

The discussions were rich and revealing. Many pointed to organisational gaps — no clear strategy, limited training, and confusion over who is responsible for data quality. Others highlighted cultural issues: time pressures, lack of curiosity, and a tendency to assume someone else has already checked the data.

Simple Excel errors are also common. We heard an example case of a study comparing a new, advanced imaging machine with an older model. The results were presented in a spreadsheet, which included several measurements. As expected, the correlation matrix showed strong correlations between most columns—except for the first, which was the main measure of interest. It quickly became apparent that the sort function had been applied to that column, scrambling the values and rendering them effectively random. Unfortunately, the researcher had not kept a backup of the original data, meaning the entire experiment was compromised. During the COVID-19 pandemic, a similar technical mistake involving Excel led to thousands of positive cases being omitted from the UK’s official daily figures. These are the kinds of simple issues that could have been caught with a basic data check.

Other examples were given of data quality issues arising when datasets were used for a specific research focus, and the quality checks applied were tailored too narrowly to that focus. Additional problems only became apparent when the same data was later used for a different research purpose. Conclusion: you can’t be complacent about the quality of the data you’re using.

Strategies for Change

The second question sparked even more ideas. Suggestions ranged from embedding data quality education early (even at school level) to implementing cultural changes that lead to great transparency. Participants called for:

• Training and upskilling across roles
• Transparent reporting of errors and limitations
• Positive feedback loops for data collectors
• Rewarding quality work and error detection
• Modernising systems and improving interoperability
• Using AI and automation to support quality checks
• Publications including recommendations for more transparent reporting of “initial data analysis” in their guidelines.

One standout idea: organisations could promote a “data amnesty” culture where errors can be acknowledged without blame. This is something Roger experienced during his time as Chief Statistician for the Scottish Government. There, he occasionally encountered serious data quality issues that required official statistics to be revised or delayed. Being transparent with users about these issues was a key principle of the Code of Practice for Official Statistics. A conscious effort was made — through training and through taking a certain approach to handling such situations — to foster a culture of openness and accountability. Staff were supported to create and implement plans to address the problems, learn from them, and communicate clearly with users. This transparency was essential to maintaining trust in both our processes and the statistics we produced.

A Call to Action

We walked away from the workshop with a clear conclusion: data quality needs a culture shift. It’s not enough to care — we need to prioritise and celebrate the work of those who keep our data trustworthy, while educating other stakeholders about what it involves.

Shaping the next steps will require keeping this conversation going within the data community and Real World Data Science can play an integral role in that. As a direct result of this piece, we have updated our submission guidelines to include recommendations for transparent data reporting and we would like to publish more stories of data disasters – or disasters averted through careful attention to data quality.

As one attendee put it, “We need to challenge the data and learn best practice from the get-go.” It’s time to embrace our inner data detectives; the integrity of our insights depends on it.

Please share your own data disaster stories in the comments, or in the Real World Data Science inbox.

Data in the Veterinary Profession: Challenges and Opportunities

If Artificial Intelligence (AI) is to be used in any sphere, it needs to be trained on data. The data used to train should be relevant, complete, structured, accurate, consistently formatted, and labelled. Achieving this standard is a challenge not only in veterinary medicine but also in many other fields where data are fragmented and inconsistently recorded. In the veterinary profession, unlike centralised NHS data, the veterinary data are often stored in individual practices or farms. These may use different formats and scales (such as imperial or metric), US or UK date formats, and twelve or twenty-four hour clocks. These records may also fail to follow the animal if it is sold or moves to a new practice. Such inconsistencies mirror the difficulties faced in other domains, and can make the adoption of AI in veterinary medicine particularly complex..

On the other hand, animal data has fewer constraints than human data. Article 4 of the UK General Data Protection Regulation (GDPR) makes it clear that the act applies to ‘personal data’ and specifies that ‘an identifiable natural person is one who can be identified’, which means that there is potentially more freedom to use data related to animals than humans. (It is worth noting that the GDPR would apply to the farmer, pet owner, or veterinary staff involved, so some consideration might still be required.) Given this data is an asset, it is worth considering whether it is owned by the animal’s owner or the veterinary professional (or their employer) in any given circumstance.

How AI is Already Transforming Veterinary Practice

AI is becoming an affordable and widely used tool in veterinary medicine. It’s now commonly applied in areas like diagnostics, treatment, and disease monitoring and prediction, despite the misconception that it’s rarely used. Preventative healthcare has always been a key aim within veterinary medicine. The obligation to ensure that both animal health and welfare and public health are accounted for is reflected by point 6.1 of the Code of Professional Conduct for Veterinary Surgeons and RVNs: ‘6.1 Veterinary surgeons must seek to ensure the protection of public health and animal health and welfare’.

Diagnostics
Diagnosis and prediction of diseases is one key area where AI is being used in veterinary medicine in farm animals, companion animals and beyond.

For example, in companion animals AI has been used to assist in the diagnosis of canine hypoadrenocorticism, an endocrine disease. Additionally, machine learning algorithms have potential for improving the prediction and diagnosis of leptospirosis, an infectious zoonotic disease. Additionally, by combining MRI data with facial image analysis, an AI tool can assist in predicting the likelihood of chiari like malformation (CM) and syringomyelia (SM) from images of the dog’s head obtained via an owner’s smartphone. And finally, AI can also assist with faecal analysis: images are analysed by proprietary Artificial Intelligence models which reference the images against the Telenostic Reference Library, a company specialising in parasitology diagnostic solutions. The image recognition software identifies each specific parasite species and the number of parasitic eggs or oocysts present.

These are just a few examples of AI use in companion animals currently.

Disease Monitoring and Prediction Disease monitoring and prediction are exciting because they can help us act earlier—sometimes even preventing illness. This not only improves animal health and welfare, but also supports antimicrobial stewardship by reducing unnecessary treatments, helping to combat antimicrobial resistance—a serious global threat to both animals and humans.

An area that demonstrates compelling evidence of these positive outcomes is farming and agriculture, where farmers are able to use AI to monitor herds, and act promptly to treat disease, before it would have been evident and notable by human monitoring. Examples, which will be explored in more detail below, include body condition technology, lameness technology, disease recognition, grazing, land and pasture management, biosensors and biochips and more.

Body Condition Technology The agricultural industry typically relies on subjective visual observation, human recording and manual reporting of all the key health and welfare traits, including Body Condition Score (BCS). Despite these individuals being highly skilled professionals, there is inevitable human error, paired with the constraints of busy farm management which can lead to cases getting picked up later in their disease process. BCS is a major indicator of metabolic performance in dairy cows and directly related to fertility performance and health traits. Technologies such as Herdvision use a 2D and 3D camera system to monitor BCS, resulting in improvement in cattle heath and fertility, less premature culling, and savings on feeding costs.

Lameness Technology
Lameness is considered one of the top cattle health and welfare challenges. A 2013 study noted that almost 70% of the dairy farmers expressed an intention to take action for improving dairy cow foot health. Cattle naturally mask the signs of pain, and as with body condition scoring we have relied on subjective visual observation, human recording and manual reporting of all the key health and welfare traits. Technology that can pick up lameness earlier, with more objectivity and with less labour intensity is hugely beneficial to both the animals’ health and welfare and the farm’s profitability.

Disease Recognition As with lameness assessments, monitoring of pain in the UK pig industry relies on human observation, either in person or via video footage, to detect disease.

An interdisciplinary team at the Newcastle University have used artificial intelligence to develop automated systems to analyse and monitor pig behaviour and health. The algorithm was tested in a controlled environment where infection and disease were present, assessing footage of pigs captured by cameras and pinpointing and quantifying changes in behaviours to identify links to disease.

Other computer vision and AI-based approaches have allowed the automatic scoring of pigs in relation to posture, aggressive episodes, tail-biting episodes, fouling, diarrhoea, stress prediction in piglets, weight estimation, and body size – all providing animal farmers increased insight into the health of their population.

Grazing, Land and Pasture Management The use of AI has allowed more efficient pasture and grazing management, allowing movement of livestock onto new pastures when the grazing quality and quantity depletes below a certain threshold.

There are numerous methods of using Agri-Tech to monitor animals, such as the SheepIT project, an initiative where an automated IoT-based system controls grazing sheep. Typically, such solutions are split into two main groups: location monitoring and behaviour and activity monitoring. Location monitoring allows farmers to keep track of animals, inferring preferred pasturing areas and grazing times, and even detecting absent animals. Behaviour and activity monitoring focuses on detecting the type and duration of an animal’s activities – for example resting, eating or running - based on accelerometry and audiometry.

Biosensors and Biochips In human medicine, advances in molecular medicine and cell biology have driven the interest in electrochemical systems to detect disease biomarkers and therapeutic compounds (medications for example). Currently in human literature, implantable biosensors have been noted in glucose monitoring, DNA detection and cultures, among others. Microelectronic technology offers powerful circuits and systems to develop innovative and miniaturised biochips for sensing at the molecular level; these have numerous applications in veterinary medicine from hormone detection, pathogenic microorganism and infection monitoring and homeostatic mechanism surveillance (homeostasis being the bodies regulatory mechanisms that controls many functions and maintain stability) such as being applied to pathogen detection in cattle mastitis.

Paul Horwood, Farm Vet and Founder of AI(Live), a conference on the development of AI applications in the livestock industry, sees this as a time of opportunity for the profession:

“The farm vet’s role continues to evolve from “problem-solver after the fact” to “strategic advisor at the heart of herd health planning.” Technology is helping us get there by giving us earlier insights, better data, and stronger evidence for the decisions we make every day. We’re at a pivotal moment. The technology is here. The challenge is knowing how to use it and how to lead with it. As a farming nation, we have always been innovative; as farm animal veterinary surgeons, we can either wait to be brought in at the end of the conversation, or step forward now to shape how AI is used on UK farms. Let’s choose the latter.”

Shared Frontiers: Common Threads in AI Adoption Across Sectors

The veterinary sector, like every other industry, is on a journey when it comes to the use of artificial intelligence, and many of the themes that are emerging are common to other sectors.

For veterinary professionals this includes:

• The need to radically change training of new vets and RVNs to ensure that they are prepared to embrace the new opportunities that AI will bring.
  
• The need to upskill existing vets and RVNs to enable them to use these new opportunities.
• Working with stakeholders, such as in this case farmers and pet owners, to evolve the business model to ensure that all parties benefit from the change.
• A change in the attitude to data, in which it becomes seen as a business asset when it is well managed, with the ultimate benefit in this sector of promoting the wellbeing of animals.

These recurring patterns offer a blueprint for understanding how professions evolve in response to developments in the field, and a reminder that AI isn’t just transforming high-tech labs and Fortune 500 boardrooms – it is quietly revolutionising industries across every sector. By looking at how specific professions, like veterinary, are navigating this shift, we can better understand the broader dynamics at play when machine learning meets existing practice.

Bridging Disciplines: Unlocking Value Through Interdisciplinary Collaboration

This is a pivotal moment where the intersection of data science and veterinary medicine offers a unique opportunity for cross-sector collaboration, driving progress in both fields.

The field of data science has much to offer industries currently experiencing these inflection points. Although many data scientists come from ‘traditional’ backgrounds such as statistics, mathematics, or computer science graduates, many more diverse routes to data science roles now exist. These routes include people who wouldn’t necessarily call themselves data scientists who work in other professions who use data science in their working life, upskilling themselves through training, or even trial and error. The authors are already aware of veterinary professionals who are skilled data scientists, even if they may not identify as such, applying data science to veterinary research in academia or industry. The RSS, other professional bodies within the Alliance for Data Science Professionals, and Data Science departments in universities may find offering Continuous Professional Development opportunities to the veterinary profession worthwhile. Certainly, the certifications offered by the RSS of Data Science Practitioner and Advanced Data Science Practitioner are open to veterinary professionals who have developed such skills.

The veterinary profession may also provide benefits to the data science community, by providing data sets that can be applied in many ways without major GDPR issues, as well as opportunities to showcase the benefits of data science to society and animal health and welfare through examples similar to those above.

One vehicle for more cross-pollination could be joint conferences. A near-term opportunity is AI (Live) in September 2025, which aims to start the debate and establish the principles by which AI and livestock farming can derive the maximum benefits, with a focus on education, governance and application.

By fostering collaboration across disciplines, we can ensure that the benefits of this data revolution are shared—by all creatures, great, small, and artificial. And we are happy to report that Sparkle, whose illness sparked this article, has made a full recovery and in fact recently celebrated her ninth birthday!

Scope and type of papers

RSS: Data Science and Artificial Intelligence is seeking high quality papers from across the breadth of these disciplines which encompass statistics, machine learning, deep learning, econometrics, bioinformatics, engineering, computational social sciences and beyond.

As well as three primary paper types - method papers, applications papers and behind-the-scenes papers - RSS: Data Science and Artificial Intelligence will publish editorials, op-eds, interviews, and reviews/perspectives in line with its goal to become a primary destination for data scientists

Why Publish?

RSS: Data Science and Artificial Intelligence offers an exciting open access venue for your work with a broad reach and is peer reviewed by editors esteemed in their field. Discover more about why the new journal is the ideal platform for showcasing your research

Submit a paper

Find out how to prepare your manuscript for submission and visit our submission site to submit your paper

Editors

View the full editorial board here: Editorial Board | RSS Data Science | Oxford Academic (oup.com)

Open Access

RSS: Data Science and Artificial Intelligence is fully open access (OA) and is published by Oxford University Press (OUP). Your research will be free to read and can be accessed globally. An OA license increases the visibility of your research and creates more opportunities for fellow researchers to read, share, cite, and build upon your findings.

The cost of publishing Open Access may be covered under a Read and Publish agreement between OUP and the corresponding author’s institution. Find out if your institution is participating. Members of the Royal Statistical Society can submit papers at a reduced cost.

Explore the journal’s website now www.academic.oup.com/rssdat

Collective phenomena

Hopfield networks brought some of the mathematical toolbox long applied to extract “collective phenomena” from vast numbers of essentially identical parts such as atoms in a gas or atomic spins in magnetic materials. Although there maybe too many particles to track each individually, properties like temperature and magnetic field can be extracted using statistical physics. Hopfield showed that similarly a useful phenomenon he described as “associative memory” could be constructed from large numbers of artificial neurons by defining a “minimum energy”, which describes the network of neurons. The energy is determined by connections between neurons, which store information about patterns. Thus the network can retrieve the memorized patterns by minimizing that energy, just as stable conformations of atomic spins might be found in a magnetic material¹. As the energy of the network is then subsequently minimised the pattern gets closer to the one that was memorised, just as when recalling a word or someone’s name we might first run through similar sounding words or names.

These Hopfield networks proved a seminal step in progressing AI algorithms, enabling a kind of pattern recognition from multiple stored patterns. However, it turned out that the number of patterns that could be stored was fundamentally limited due to what are known as “local” minima. You can imagine a ball rolling down a hill – it will reach the bottom of the hill fine so long as there are no dips for it to get stuck in en route. Algorithms based on Hopfield networks were prone to getting stuck in such dips or undesirable local minima, until Hopfield and Krotov put their heads together to find a way around it. Krotov describes himself as “incredibly lucky” that his research interests aligned so well with Hopfield. “He’s just such a smart and genuine person, and he has been in the field for many years,” he tells Real World Data Science. “He just knows things that no one else in the world knows.” Together they worked out they could address the problem of local minima by toggling the “activation function”.

In a Hopfield network all the neurons are connected to all the other neurons, however originally the algorithm only considered interactions between two neurons at each point, i.e. the interaction between neuron 1 and neuron 2, neuron 1 and neuron 3 and neuron 2 and neuron 3, but not the interactions among all three altogether. By including such “higher order” interactions between more than two neurons, Krotov and Hopfield found they made the basins of attraction for the true minimum energy states deeper. You can think of it a little like the ball rolling down a steeper hill so that it picks up more momentum along the slope of the main hill and is less prone to falling in little dips en route. This way Krotov and Hopfield increased the memory of Hopfield networks in what they called the Dense Associative Memory, which they described in 2016². Long before then, however, Geoffrey Hinton had found a different tack to follow to increase the power of this kind of neural network.

Generative AI

Geoffrey Hinton showed that by defining some neurons as a hidden layer and some as a visible layer (a Boltzmann machine³) and limiting the connections so that neurons are only connected with neurons in other layers (a restricted Boltzmann machine⁴), finding the most likely network would generate networks with meaningful similarities – a type of generative AI. This and many other contributions by Geoffrey Hinton also proved incredibly useful in the progress of machine learning. However, the generative AI algorithms grabbing headlines today have actually been devised using a “transformer” architecture, which differs from Hopfield networks and Boltzmann machines, or so it seemed initially.

Transformer algorithms first emerged as a type of language model and were defined by a characteristic termed “attention”. “They say that each word represents a token, and essentially the task of attention is to learn long-range correlations between those tokens,” Krotov explains using the word “bank” as an example. Whether the word means the edge of a river or a financial institution can only be ascertained from the context in which it appears. “You learn these long-range correlations, and that allows you to contextualize and understand the meaning of every word.” The approach was first reported in 2017 in a paper titled “Attention is all you need”⁵ by researchers at Google Brain and Google Research.

It was not long before people figured out that the approach would enable powerful algorithms for tasks beyond language manipulation, including Demis Hassabis and John Jumper at Deep Mind as they worked to figure out an algorithm that could predict the folding conformations of proteins. The algorithm they landed on in 2020 – AlphaFold2 – was capable of protein conformation prediction with a 90% accuracy, way ahead of any other algorithm at the time, including Deep Mind’s previous attempt AlphaFold, which although streaks ahead of the field at the time it was developed in 2018, still only achieved an accuracy of 60%. It was for the extraordinary predictive powers for protein conformations achieved by AlphaFold2 that Hassabis and Jumper were awarded half the 2024 Nobel Prize for Chemistry.

Connecting the dots

Transformer architectures are undoubtedly hugely powerful but how they operate can seem something of a dark art as although computer scientists know how they are programmed, even they cannot tell how they reach their conclusions in operation. Instead they query the algorithm and add to it to try and get some pointers as to what the trail of logic might have been. Here Hopfield networks have an advantage because people can hope to get a grasp on what energy minima they are converging to, and that way get a handle on their working out. However, in their paper “Hopfield networks is all you need”⁶, researchers in Austria and Norway showed that the activation function, which Hopfield and Krotov had toggled to make Hopfield networks store more memories, can also link them to transformer architectures – essentially if the function is exponential they can reduce to the same thing.

“We think about attention as learning long-range correlations, and this dense associative memory interpretation of attention tells you that each word creates a basin of attraction,” Krotov explains. “Essentially, the contextualization of the unknown word happens through the attraction to these different memories,” he adds. “That kind of lens of thinking about transformers through the prism of energy landscapes – it’s opened up this whole new world where you can think about what transformers are doing computationally, and how they perform that computation.”

“I think it’s great that the power of these tools is being recognised for the impact that they can have in accelerating innovation in new ways,” says Janet Bastiman, RSS Data Science and AI Section Chair and Chief Data Scientist at financial crimes compliance solutions company Napier AI, as she comments on the Nobel Prize awards. Bastiman’s most recent work has been on adding explanation to networks. She notes how the report Hopfield networks is all you need highlights “the difference that layers can have on the final outcomes for specific tasks and a clear need for understanding some of the principles of the layers of networks in order to validate results and be aware of potential difficulties and “best” scenarios for different use cases.”

Krotov also points out that since Hopfield networks are rooted in neurobiological interpretations, it helps to find “neurobiological ways of interpreting their computation” for transformer algorithms too. As such the vein Hopfield and Hinton tapped into with their seminal advances is proving ever richer in what Krotov describes as “the emerging field of the physics of neural computation”.

Gender disparities in internet access

A turning point in the conversation around digital technology and gender equality came in 2018 with work by Kashyap and collaborators in the US and Qatar at the time. They found that where traditional survey-based data on internet and mobile gender gaps was available, it correlated well with the gender gap on Facebook, using data extracted for Facebook’s ad platform: When Facebook’s aggregate user counts did not show women, it provided a good signal that women were not online altogether in those countries. As such, the work revealed a potentially useful proxy to gauge the digital gender gap in countries where little traditional survey data was available³. The results revealed an unexpectedly large gender gap, particularly in parts of South Asia and certain countries in Africa where men were up to twice as likely to have access to the Internet compared with women.

‘In some sense it was perhaps not surprising,’’ says Kashyap highlighting that having a mobile phone or similar device that grants access to the internet amounts to a kind of asset ownership, and studies for other assets indicate women are less likely to own them. ‘This is broadly reflective of economic gender inequality,’ she adds. Perhaps more surprising is that the gaps have changed very little in the five years since their website, which monitors the digital gender gap, was first released, particularly in view of the pace of technological progress in general, and the importance placed on closing the gap. Citing India as an example, Kashyap points out that in 2019 the ratio of access to the internet for men versus women was 0.619 – fewer than two women had access for every three men with access. In the subsequent half decade this digital gender gap has closed by just 7.1% to a ratio of 0.663.

In countries where the gender disparity for access to the internet is large, there is evidence to suggest that those women who do have access are of the more affluent echelons of society. Analysis of the type of device used, which can also be retrieved from the Facebook ad platform, highlighted that where women are less likely to be online, the relative proportion of iOS users tends to be higher among women than among men, and as Kashyap points out, ‘iOS users are on average wealthier’. Fortunately, among the stakeholders starting to see the benefit of closing the gap in access to the internet between the genders are the mobile network providers, who are looking for ways to tap into this part of the market through incentives and discounts on SIMs for women. However, it is unclear to what extent these types of schemes are ultimately beneficial in closing the wider gap.

Kashyap and her colleagues also found that a key predictor of the digital gender gap was the gender gap in educational attainment. ‘I think that’s quite telling, because it’s showing that accessing education and going to educational institutions is also a pathway to becoming more digitally integrated,’ says Kashyap, flagging that schools and educational institutions are where women and girls often access computers and digital technologies. She highlights that beyond giving people a device ‘more of the challenge’ is helping them make good use of it by ‘giving people skills to feel that this is actually meaningful for them, and allows them to do things that they wouldn’t be able to do otherwise, and feeling confident and safe and secure.’ She emphasises the importance of men valuing gender equality, highlighting work from South Asia that shows that even when women have a device, their use of it may be curtailed or scrutinised by male members of the household, sometimes on the grounds of doubts over women’s safety online.

Gender disparities in ‘fears’ of online harms

Safety can be a knotty issue when it comes to enabling women to have a voice online. A study by the Alan Turing Institute⁵ earlier this year suggested just 23% of women in general feel comfortable expressing political opinions online, compared with 40% of men. This might be down to women in general being exposed to online violence more than men, as previous studies of online harms have suggested. Indeed, a key takeaway from the Alan Turing Institute’s study was that women reported greater fears of exposure for all categories of harm, although this included types of harm that women reported experiencing less frequently than men.

Previous studies have largely surveyed women-only sample-groups so that their conclusions were drawn without data on men against which to compare. In contrast, the researchers at the Alan Turing Institute, including researcher Tvesha Sippy, took a nationally representative survey of 2,000 men and women. They investigated whether they had been exposed to various types of online harms, their fears surrounding such exposure, the psychological impact of those experiences in general, tendencies to use protective tools for digital activities, and how comfortable they felt with online behaviours such as expressing opinions and sharing information online. The study revealed that women were more likely to report experiencing some harms, such as online misogyny, cyberflashing, cyberstalking, image-based abuse and eating disorder content to a significantly greater extent than men. However, there were several harms that men reported being the direct targets of to a greater extent than women, such as hate speech, misinformation, trolling and threats of physical violence.

By using a representative cohort, the Alan Turing Institute study tells a more nuanced story than those sampling women only and highlights challenges in similar assessments for minority groups. For example, those identifying as non-binary were excluded from the analysis by the Alan Turing Institute because, although as Sippy emphasises, ‘We do want to look at minoritised genders,’ they did not have sufficient numbers of respondents in this category within their nationally representative survey to do any meaningful analysis. Ultimately, a higher budget enabling larger samples would allow analysis of minority groups as well.

As for the greater fears for all online harms reported by women, ‘it’s a very complex phenomenon,’’ Sippy tells Real World Data Science, highlighting the need for further research. She points to several possible explanations such as differences in the impacts of the harms experienced more by women versus men, as well as innate fearfulness potentially from the offline world translating to behaviour online. Sippy also highlights the differences in how men and women experience online harms, which may offer clues. Women were more likely to report that their fears stem from the experience of a public figure (35% of the women surveyed compared with 26% of the men) or a female friend (37% of the women compared with 27% of the men). Furthermore, the experience of a male friend was much less often cited as the source of online fears for both groups (8% of the women and 14% of the men). There is also the possibility that women’s adaptive behaviours make them less exposed to future online harms than men, since women were more likely to make use of protective tools from disabling location-sharing on a device, and limiting who can engage with images, posts and tweets, or even find their profile. While protective, such adaptive behaviours could also dampen the influence women have in online discourse.

Rather than relying on adaptive behaviour for self-protection, it would seem a lot of people are keen to see more action from social media companies and governments to help people to feel safer online. In 2023, researchers at the Turing Institute led by senior research associate Florence Enock published a study investigating attitudes to online interventions. They found that 79% thought social media platforms should ban or suspend users who create harmful content and 73% thought that platforms should remove harmful content. According to the report ‘this was consistent across age, gender, educational background, income and political ideology.’

There are some complications for social media companies who need to balance privacy needs with protection, as well as having the resources required to handle multilingual posts when investigating what action to take. However, Sippy feels there remains a need to have a civil remedy in place so that a user can request a platform take down content which is harmful without having to pursue criminal proceedings and get the police involved. Where the additional resources needed for social media companies to take corrective action and a lack of business incentive pose an obstacle, government legislation may help. The same study into attitudes to online interventions also reported that for platforms that fail to deal with harmful content online more than 70% of respondents felt the government should be able to issue large fines, and 66% thought that legal action should be taken.

‘The Online Safety Act is a really good start,’ adds Sippy, also highlighting the importance of proposals by the previous UK government to criminalise the creation of sexually explicit deep fakes. She points to a 2019 report by AI firm Deeptrace, suggesting that of 15,000 deep fake videos they found online, 96% constituted nonconsensual pornography with women disproportionately targeted⁶. In a recent Alan Turing Institute survey 90% of respondents expressed concerns about deepfakes increasing misogyny and online violence against women and girls⁷. ‘I do see there’s more advocacy, but it remains to be seen what approach the new Government will take.’

Gender disparities for making an impact online

Challenges to women being heard online seem to go beyond safety issues. Recent research by Kashyap and collaborators at the University of Oxford and collaborators in Iran and Germany has also highlighted differences in how influential women’s professional networks are relative to male counterparts⁸. In previous work with Florianne Verkroost, also at the University of Oxford, Kashyap had investigated the gender gaps in those who have a LinkedIn profile to see how they vary across industries⁹. They found that use of the platform broadly mirrors female-to-male ratios of representation in technical and managerial professions. In reference 8, they then investigated what insights LinkedIn data might provide as to the cause of some of the gender disparities in these professions, and ultimately why women are not progressing in technical and professional jobs as well as male counterparts.

‘One argument is that that’s often because they don’t have advantageous networks,’ says Kashyap, adding that women may be restricted by the need to resume care commitments at home instead of staying for drinks after work or travelling to attend conferences. One might expect online avenues for networking would be able to mitigate such obstacles. In fact, studies of LinkedIn data did suggest that although women are less likely to be in professional and technical occupations as reflected in the platform’s data, in some instances their numbers exceeded them. Kashyap suggests this could be ‘where they’re using online platforms to make themselves more visible, because other fine forms of networking are less available, or they have less time for it.’ Indeed, women who were on LinkedIn were more likely to report a promotion than their male counterparts, suggesting an element of positive selection among the female LinkedIn user population. However, the potential equalising impact of moving professional networking online seems to have its limits.

Their study of LinkedIn data showed women were less likely to report a relocation for work, which Kashyap suggests, ‘is a sign that the work family trade-off is probably still remaining acute for this highly selected group.’ In another 2023 study Kashyap and colleagues had also reported a lower mobility for women, specifically among published scientists, researchers and academics based on bibliometric data from over 33 million Scopus publications¹⁰. In addition, when Kashyap and her colleagues looked at women on LinkedIn working in the tech sector, they found that they had a lower chance of being connected to those working in one of the “big five” firms in the tech sector than men, when not working in one themselves. ‘One way to interpret that is to say that they have maybe less influential online social networks, right, even when they are on the platform.’

Kashyap suggests several reasons why women may have less influential networks online. For one, online networks are still likely to be influenced by the scenarios playing out offline, since referrals on these networks are based on the people you already know. The difference may also be based on the types of companies women tend to work in and the positions they hold. For instance, women are more likely to work in IT service support than programming-intensive occupations, and here once again Kashyap suggests the work family trade off plays a role in women seeking less intensive or more flexible jobs. She highlights that girls equal or exceed the achievement of male counterparts through school and continue to match them in their early careers before their numbers start to drop off dramatically. ‘I think now there’s a growing recognition that this is actually a real conflict, the work family conflict,’ she tells Real World Data Science. Today’s young women are socialised to have ‘high achieving aspirations’, which can be hard to reconcile with ‘regressive norms’ for women to shoulder the bulk of caring responsibilities, particularly when starting a family.

Real world gender disparities in career development

Neuroscientist Joanne Kenney has also been following data on the gender gap in the science and tech sectors and co-authored ‘A Snapshot of Female Representation in Twelve Academic Psychiatry Institutions Around the World’¹¹ with Assistant Professor of Psychiatry at Harvard Medical School Elisabetta del Re. The figures published here also show that globally women represent a large majority of early career scientists, but their numbers steadily decrease towards the mid and senior career stages so that there is a negative correlation between career stages and female presence in science, often referred to as the ‘leaky pipeline’ or ‘sticky floor’. ‘You don’t always hear their stories or the reasons why they’ve left,’ says Kenney who highlights that in her experience in academia exit interviews are rare. Just 24% of the UK total workforce in the tech sector are women, while black women account for only 0.7% of IT professionals according to the 2024 UN Women UK and Kearney Consulting report ‘Gap to Gateway: diversity in tech as the key to the future’ for which Kenney was an external collaborator. Kenney is currently working on another project with a team of scientists from Europe, Africa, and North and South America led by del Re to gather stories from women and other underrepresented groups in academic institutions around the world through focus groups aimed at better understanding their experiences of working in science.

For those who stick at it, the career path appears to be a steeper hike for women than their male counterparts. There is a citation-bias favouring male-authored articles¹². Women also take on average nine years to transition to senior author whereas men take five¹³, and women are less likely to be promoted to leadership positions¹⁴. While women in science bear a measurably unequal career impact on entering parenthood¹⁵, some of these inequalities may also stem from sexism, which can range from fewer opportunities for mentorship and collaboration to outright harassment¹⁶.

‘I think a lack of mentorship and sponsorship are two big ones,’ says Kenney when it comes to the key discouraging factors for women at the mid-career point in tech and academia. In AI, in particular, less than 3% of venture capital funding deals involving AI startups go to women-founded companies. The gender pay gap, which at 16% in the sector exceeds the overall pay gap of 11.6% may be another disincentive.

In short there is evidence of various patriarchal subcultures at play, both in the tech and science sectors and the world in general that can still pose a significant disadvantage to women. As Sippy points out, ‘Those subcultures also translate to the online world.’ Ultimately while digital technologies may offer creative loopholes for side-stepping some aspects of gender bias and disadvantage, gender inequality needs to be tackled in both spaces in tandem.

Model framework improved flexibility and accuracy:

The 3 steps in the framework are chronological steps to be performed in which the first step is pre-selection of the independent variables with the highest predictive power. The independent variables from the first step are then summarised into a few factors using factor extraction methodology in the second step. The final step consists of using the factors from step 2 to perform regression.

Figure 2 summarises the various methods employed at each step in the 3 step framework. In their report Chinn, Meunier and Stumpner aim to propose the best techniques for pre-selection, factor extraction and regression. As such their 3-step framework comprises performing pre-selection using Least Angle Regression (LARS), factor extraction using Principal Component Analysis (PCA) and employing a Macroeconomic Random Forest (MRF) machine learning technique for nowcasting.

The model performance or accuracy of MRF is compared with traditional methods using Root Mean Square Error (RMSE), a measure of the deviation between the actual data and the predicted data. The 3-step framework model accuracy is tested by holding the preselection and factor extraction fixed to isolate the impact of regression techniques.

Figure 3 compares the RMSE of traditional methods, machine learning tree and machine learning regression model for backcasting (t-2 and t-1), nowcasting (t) and forecasting (t+1). It highlights the greater model accuracy of MRF and Gradient Boosting compared with traditional models and tree models for backcasting, nowcasting and forecasting.

What’s Next?

Organisations such as The Nowcasting Lab provide GDP estimates for European countries. Such nowcasting techniques have been employed by humanitarian agencies including the United Nations Refugee Agency (UNHCR) which uses nowcasting to estimate the actual forced displaced population. The nowcasting techniques, dashboards and tools have been implemented and accepted as a reliable source of information at government organisations for policy making, central banks, and financial organisations. The 3-step framework, proposed by Chinn, Meunier and Stumpner, is easily adaptable, flexible and provides higher accuracy, which will be valuable to a range of fields employing nowcasting.

High stakes

When Open AI publicised their Voice Engine’s ability to clone human voices from just 15s of audio, they too flagged the potential benefit for people with poor health conditions, since those with deteriorating speech could find a means to have their speech restored. However, voice clones had already been used to make robot calls to voters imitating the voice of President Joe Biden and telling voters to stay at home.

“The question you have to ask there is what’s the societal benefit of that tool? And what are the risks,” associate director at the Ada Lovelace Institute Andrew Strait told Real World Data Science. “They thankfully decided to not fully release it,” he adds, highlighting how the timing “right before an election year with 40 democracies across the world” could have made the release particularly problematic.

While OpenAI’s voice engine might have made voice cloning more accessible had they proceeded with a full release, voice cloning is clearly still well within reach for some already. Strait cites the experiences of hundreds of performing artists in the UK over the past few months that have been brought to the attention of the Ada Lovelace Institute. “They’re brought into a room; they’re asked to record their voice and have their face and likeness scanned; and that’s the end of their career,” says Strait. The sums paid to artists on these transactions are not large either. “They are never going to be asked to come back for audition again, because they [the companies] can generate their likeness, that voice doing anything that a producer wants without any sense of attribution, further payments, or consent to be used in that way.”

Customer service is another sector where jobs have been threatened with replacement by a generative AI chatbot, however the technology can run into problems since gen-AI is known to “hallucinate”, generating false information. Air Canada has just lost a case defending its use of a chatbot that misinformed a customer that they could apply for a bereavement fare retroactively, which is not the case according to Air Canada’s bereavement fare policy. In their defence Air Canada flagged that the chatbot had supplied a link to a webpage with the correct information but the court ruled that there was no reason to believe the webpage information over the chatbot, or for the customer to double check the information they had been supplied. While there are ways to mitigate problems with gen-AI with the right teams in place , other industries have also hit problems with the accuracy and reliability of gen-AI, which may dampen the impact AI has on the labour market. All in all the wider picture of how AI deployment may affect jobs is largely a matter of speculation. Here a US piloted scheme may soon provide framework for a more data informed approach to tackling AI’s impact on the workforce.

Strait highlights that conversations that centre around efficiency when weighing up the possible advantages of introducing AI can be ill informed. “If we’re talking about an allocation of resources in which we’re spending an increasing amount of money on automating certain parts of the NHS, or healthcare or the education system, or public sector services, how are we making the decisions that are determining if that is worth the value for money? Instead of investing in more doctors, more teachers, more social workers?” He tells Real World Data Science that these are the questions he and his colleagues at the Ada Lovelace Institute are often pushing governments to try to answer and evidence rather than to just assume the benefits will accrue. When it comes to measures of success of an AI model, Strait says “It’s often defined in terms of how many staff can be cut and still deliver some kind of service…This is not a good metric of success,” he adds. “We don’t want to just get rid of as many jobs as we can, right, we want to actually see improvements in care, improvements in service.”

Michael Katell, ethics fellow in the Turing’s Public Policy Programme and a visiting Senior lecturer at the Digital Environment Research Institute (DERI) at Queen Mary University of London suggests the problems may go deeper still when looking at the use of generative AI in creative industries. “There are definitely parallels with prior waves of disruption,” he says citing as an example the move to drum-based and eventually laser printing as opposed to manual typesetting. “A key difference, though, is that, in the creative arts, we’re talking about contributions to culture, and culture is something that, I think we often take for granted.” He highlights the often overlooked role cultural practices that enable and empower shared experiences have in holding society together. These may come in various forms from works of art to theatre, and the working and living practices among the wider community may play an important role too. While acknowledging there may be interesting and fascinating uses of AI in art to explore, Katell adds, “If we’re not attending to maintaining some aspects, or trying to manage the changes that are happening in our culture, I think we’ll see societal level effects that are much greater than the elimination of some jobs.”

The need for legislation

These stakes all highlight the need for regulatory interventions. However, most governments, bar China and the EU, have so far favoured “voluntary commitments” towards AI safety, which would seem to fall short of providing the kind of governance over the sector that can be robustly enforced. In a recent blog Strait alongside the Ada Lovelace Institute’s UK public policy lead Matt Davies and associate director (Law & Policy) Michael Birtwhistle, “evaluate the evaluations” of the UK’s AI Safety Institute for companies that have opted in for these voluntary commitments. They highlight that on the whole the companies planning to release the product hold too much control over how the evaluation can take place, ultimately empowering them to direct tests in their favour, which inhibits efforts at robust monitoring. Furthermore, there is usually no avenue for the necessary scrutiny of training data sets. Even withstanding these limitations, Davies, Strait and Birtwhistle conclude that “conducting evaluations and assessments is meaningless without the necessary enforcement powers to block the release of dangerous or high-risk models, or to remove unsafe products from the market.”

The reticence to implement firmer regulation might be attributed in some part to the perceived benefits to the state when their AI companies succeed. One often perceived benefit is that the percolating profits these companies accrue may benefit the economic buoyancy of the societies they function within. There is also cause for sovereign state competitiveness in “AI prowess” that stems from the potential for AI-based technology to underpin all aspects of society, prompting what has been described as an “AI arms race”. Here the UK may well regret allowing Google to acquire Deep Mind, whose output is responsible for bolstering the “UK’s share” of citations in the top 100 recent AI papers from 1.9% to 7.2%. However, a lack of robust regulation may prove as much a disservice to the companies releasing AI products as it is to society as a whole.

“The medicine sector here [in the UK] is thriving, not in spite of regulation, but because of regulation,” says Strait. “People trust that the products you develop here are safe.” Katell, highlights the impact of pollution legislation on the automotive industry. “It jumped forward invention and discovery in automotive technology,” he tells Real World Data Science. “It seems prosaic in hindsight, but it wasn’t, it was a major innovation that was promoted by regulators, promoted by legislators.” The UK government’s chief scientific advisor Angela McLean seems to agree. “Good regulation is good for innovation,” she replied when asked about balancing regulation with favourable conditions for a flourishing AI sector at an Association of British Science Writers’ event in May. “We’re not there yet,” she added. The challenge is pinning down what good regulation looks like.

Regulatory ecosystems

As has been emphasised throughout the series, making a success of an AI project requires a unique skillset that combines expertise in AI with the domain expertise for the sector the project is contributing to, and there is often a dearth of people that straddle both camps. The same hunt for “unicorns” with useful expertise in the tech sector and policymakers can also be an obstacle for developing “good regulation”. One solution is to bring people from the different disciplines together to develop legislation collaboratively, as was arguably the case with the roll out of General Data Protection Regulations (GDPR) in 2018. “Policymakers and academics, they worked very closely together in the crafting of that law,” says Katell. “It was one of those rare moments in which we saw the boundaries really dissolve between policy and academia in a way that delivered something that I think we can agree was largely a positive outcome.”

When it comes to AI, an obstacle to that kind of collaboration has been the lack of a common language. In “Defining AI in policy and practice” in 2020¹, Katell alongside Peaks Krafft at the University of Oxford and co-authors found that AI researchers favoured definitions of AI that “emphasise technical functionality”, whereas policy-makers tended towards definitions that “compare systems to human thinking and behavior”, which AI systems remain far from achieving. Strait also highlights a recurring theme among those without experience of actually making AI systems in overselling AI capabilities in suggestions that it will “help solve climate change” or “cure cancer”. “How are you measuring that?” he asks. “How are we making a clear sense of the efficacy, the proof behind those kinds of statements? Where are the case studies that actually work, and how are we determining that’s working?”

As Krafft et al. point out in their 2020 paper, such exaggerated perceptions of AI capabilities can also hamper regulation. “As a result of this gap,” they write, “ethical and regulatory efforts may overemphasise concern about future technologies at the expense of pressing issues with existing deployed technologies.” Here a better understanding of what AI is can be helpful to focus attention on the problems that exist now – not just the potential workforce impact, but the carbon cost of training large language models, activities like nonconsensual gen-AI porn aggravating online gender inequality, and a widening digital divide disadvantaging pupils, workers and citizens who cannot afford all the latest AI tools, among others.

Fortunately, there has already been progress to breach the language divide between policy makers and the tech sector. “The current definitions [championed in policy circles] say things like technologies that can perform tasks that require intelligence when humans do them,” says Katell, which he describes as a far more sober and realistic definition than likening technologies to the way humans think and work. “This is really important,” he adds. “Because some of the problems that we see with AI now are symptomatic of the fact that they’re not humans and that they don’t have the same experience of the world.” As an example he describes someone driving a car with child in the car seat, calling on all their training and experience of road use to navigate roads and other traffic, while juggling their attention between driving and the child. “Things that AI is too brittle, to accomplish,” he adds, highlighting how a simple model may identify school buses in images quite impressively until it is presented with an image of a bus upside down. “The flexibility and adaptability, the softness of human reason, is actually its strength, its power.”

Getting everybody on the same page can also help provide a more multimodal approach to governance. Empowering independent assessors of AI product safety prior to release is one thing but as Strait points out, “It could be more like the environmental sector, where we have a whole ecosystem of environmental impact assessments, organisations and consultancies that do this kind of work for different organisations and companies.” Internal teams within companies can play an important role too so long as they work sufficiently independently from the companies themselves. When set up with the right balance of expertise they can be better placed to understand and hence assess the technology and practical elements of its implementation. Although such teams can be expensive, getting the technical evaluation and consideration of ethical issues right can pose a competitive advantage for the companies themselves as well as providing a more thorough safeguard for society at large. Nonetheless there are also obvious advantages in having external regulatory bodies, which do not need to take into account the company’s profit margins or shareholders’ needs. An ideal set up might incorporate both approaches. In fact in their appraisal of the current UK AI Safety Institute arrangement, Davies, Strait and Birtwistle first highlight the need to integrate the AI Safety Institute “into a regulatory structure with complementary parts that can provide appropriate, context-specific assurance that AI systems are safe and effective for their intended use.”

Prosperity for all

With all the precedents in other sectors from environmental impact checks to pharmacology, an organised framework or ecosystem for robust, independent and meaningful evaluation of AI product safety seems an inevitable imperative, albeit potentially expensive. (Davies, Strait and Birtwistle cite £100 million a year as a typical cost for safety driven regulatory systems in the UK², and the expertise demands of AI could further increase costs.) However, such regulatory reform will likely slow down the pace of technological development and the route to market. While the breathing space to adjust to the societal changes they bring with them may be welcomed by some, the delay can be quite unpopular in a tech sector where the ethos is famed for embracing a “move fast, break things” mentality. As Katell points out that ideal is based on the notion that the things being broken were unimportant – when it’s vulnerable people and societies that is “unacceptable breakage”.

Strait also highlights the cultural mismatch between the companies developing AI products – where the research to market pipeline is extremely fast – and the sectors those tools are intended to serve, such as social care, education and health. Although Open AI eventually decided against full release of the Voice Engine, when it comes to the ethos of some AI technology companies , “The default is to put things out there and to not think through the ethical and societal implications,” says Strait who has direct experience of working for a company producing AI tools in the past. “I think it’s so critical for data scientists and ethicists to explore, and do that translation and interrogation of what are the ethics of the sector that we’re working in?”

Katell voices concern shared by many that at present AI is under the control of a very small handful of very large, powerful technology companies, and as a result the AI releases making the most impact are targeting the agendas of the companies releasing them and their current and anticipated customer base, as opposed to the needs of society. The potential for such large tech agents to become too big to fail poses additional regulatory challenges. While many may lament the tension between a demand for open source data sets for testing AI models versus the need to respect data privacy, security and confidentiality, there have already been widely mooted instances where certain companies may not have met expectations for respecting copyright and terms of service. In fact the tech giants are not the only people developing AI models and the open source community have been known to pose valuable competition that may temper the tendency for AI to concentrate a lot of power into the hands of a small few³. However, open source developers can also pose a certain amount of regulatory complexity.

There is also an argument that these efforts should broaden their scope beyond baseline AI safety and aim to focus efforts in AI development towards tools that actively promote greater wellbeing and prosperity to the many. “We need to bring in other values like fairness, justice, and simple things like explainability, gender equity, racial equity,” says Katell, highlighting some of the other qualities that demand attention among others. Taking explainability as an example, there is increasing awareness of the need to understand how certain outputs are reached in order for people to feel comfortable with the technology, and the outputs requiring explanations differ from person to person. Although it can be hard to explain AI outputs, progress is being made in this direction. As Katell says, “We’re not helpless in managing these types of disruptions. It’s a matter of societies coming together and deciding that they can be managed.”

How bad is the problem?

The lack of data results in conflicting information. Some reports have warned of apocalyptic takeovers of the job market for many professions. Indeed, a heavily cited report by Goldman Sachs ⁴ predicted that AI could replace 300 million jobs. But the same BBC report that cited the Goldman Sachs prediction quoted the future-of-work director at Oxford University, Carl Benedikt Frey as saying “The only thing I am sure of is that there is no way of knowing how many jobs will be replaced by generative AI”. Simply put, as the former US Federal CIO, Suzette Kent, said “we lack useful information for informing strategic decisions for national workforce matters.”

So just how much of a problem is it that there is no information on how investments in science and technology affect the labor market? Why should we worry if we cannot accurately predict the impact of AI on workers, firms, and jobs? One reason is to avoid the mistakes of the past, in which both workers and firms have borne the consequences of bad information. Just in recent history, digitization and globalization resulted in a devastating loss of jobs in many countries. And geographic inequality soared as jobs in the midwestern and northeastern urban centers were lost and a service economy on the coasts burgeoned. Efforts to reduce the loss of jobs and earnings came too little, too late ^{5 6}. Another reason is to make evidence based policy recommendations. For example, the US National AI Research Resources Taskforce, which was directly charged by the President and Congress with recommending ways to invest in AI research to strengthen and democratize the U.S. AI innovation ecosystem did not have joined up data between science investment and the workforce to inform their final recommendations. ⁷

In other words, governments need more timely, local, and actionable data so that they can understand changes in the tasks that employers need performed, which types of jobs and firms will be affected, and where. Concomitantly, data will be needed about the effects of AI on different population groups and different geographic areas so that the costs of change are not unfairly distributed. Armed with such information, policy makers can make investments that mitigate or counteract negative impacts and workers can be trained in the new necessary skills and matched with the firms that need them. But the swift pace of change in AI means that the urgency to create timely, local, and actionable labor market information to guide these investments has never been greater.

A new approach

The IRIS approach, called the “Industry of Ideas” builds on the “economics of ideas” framework for which Paul Romer received the 2018 Nobel Prize in Economics”. ^{8 9}. People who create ideas – new technologies – that can be reused, form the foundations of new industries. In other words, “the discovery of new ideas lie at center of economic growth…” (Charles Jones describing Paul Romer’s conceptual framework) ¹⁰.

The project recognizes that, as Robert Oppenheimer said “the best way to transmit knowledge is to wrap it up in a human being”. ¹¹ It uses people-centric methods for following the movement of ideas from investments in research into the marketplace. The approach identifies businesses that employ people with deep skills in AI and other emerging technology areas and developing early, never-before-available indicators that can provide alerts associated with potential impacts on current and future workforce. Initially focused on the artificial intelligence and electric vehicle industries in Ohio, the pilot is creating a data system that can be expanded and applied to other industries and other states across the country.

The new tools are innovative because they build on new opportunities to produce usable information that is local, about relevant industries, and that directly tie investments in new technologies, such as AI, to labor market impacts.

Another key aspect of the NSF piloted “Industry of Ideas” is the focus on tying innovation at its source - individual data on university research activities - to the local workforce data reported by firms to their state departments of labor. The need for local data is critical because so many labor markets are local, not national in scope. Even in a global economy, many businesses and workers are locally based – as are the training providers that work to ensure that labor demand and supply are well matched. Thus the Industry of Ideas pilot provides policy makers, workers, firms, and educational institutions with access to an array of local, timely, granular, actionable resources to help them make decisions. That way, local leaders who need labor market data don’t need to rely on national unemployment figures, which are reported once a month.

Connecting science investments with jobs

The Industry of Ideas approach directly connects investment in science and the labor market, moving beyond the current approach for evaluating investment by studying scientific papers and publications ^{12 13} which are disconnected from workers and jobs. The data seeds were sown almost two decades ago. President Bush’s Science Advisor, John Marburger III, who, quite sensibly was unconvinced of the scientific and practical value of relying primarily on document-based, bibliometric approaches to studying science to understand its practical effects, called for a “Science of Science Policy” ^{14 15}.

The Industry of Ideas is testing the potential to securely combine university and state data to measure the link between federal investments on local and regional economies for AI. It uses people-centric data generated by the administrative processes at universities and firms. With this data the Industry of Ideas project can capture the organization of people in science at multiple levels (e.g. individuals, teams, projects, and institutions), their multiple sources of funding (federal scientific and programmatic agencies, philanthropic foundations, industry, and state and local government), inputs into science from vendors (such as computing services, instruments, biological specimens), as well as the dynamics of their careers across time (individual career earnings and employment trajectories).

The IRIS infrastructure, developed over the past decade, provides administrative records on more than 41% of U.S. total R & D spending at universities ¹⁶. The infrastructure also provides links to survey data, as well as data from private sector suppliers ¹⁷, and can trace the flows of university funded researchers into the private sector ¹⁸ by joining up the university administrative data with state workforce data.

Tying information about AI to skills needs

How is it possible to tie changes in AI to changing needs for skills? State leaders in workforce and education agencies have identified new ways to collaborate, build staff capacity, and develop solutions, services, and products that respond to local need. An example of how to use data to get better information that more accurately connects workers with firms in the swiftly changing labor market is the New Jersey Career Navigator. It provides job seekers recommendations on new careers, available job postings, and relevant training programs based on skills similarity, labor market demand, and wage impacts observed in the underlying data. These recommendations, which are in themselvers generated by AI, show how AI technology can be used to navigate the changes in the labour market AI may cause. The New Jersey Career Navigator draws on millions of wage records, providing earnings and industry information on all workers covered by unemployment insurance in New Jersey firms; employment and wage outcomes from hundreds of thousands of graduates of occupational skills training programs in New Jersey; several years of online job postings from the National Labor Exchange Research Hub (NLx); and the resumes of 400,000 New Jersey residents.

In other words, as the Industry of Ideas pilot evolves, new ideas from states like New Jersey can be used not only to trace the flows of ideas from academia to the workplace but also to develop a new system that targets reskilling efforts once the type and location of skills needs have been identified. The new joined up data and evidence can be used to address challenges such as low labor force participation, and supplies education and training providers the data they need to align their programs with the needs of the labor market. Such a system would help government, business, educators, and workers adjust regional talent pipelines continuously in response to the changes in AI and enable workers to successfully navigate the changes that it brings.

New approaches to classifying industries: “Industries of Ideas”

An important outcome of the new NSF pilot is the potential to transform the way in which we classify firms into industries. The current industry classifications are rule based. They are designed for the economy as it was organized 40 years ago, so are not designed to describe AI. A case in point is the state of Texas – a state that anecdotally has generated a lot of high tech jobs. Current industry data for Texas is limited because firms are grouped into industries that are defined by what they produce, or how they produce it, rather than describing what new technology is being developed or utilized by those firms. As a result, the main source of labor market data in Texas provides an implausibly low picture of AI activity ¹⁹.

The Industries of Ideas approach could provide states with a new way to classify firms, based on clever new ideas of how firms can do their business, and by grouping firms by the people who created and use the technologies they will adopt ²⁰. Examples just for Ohio include funding to use AI to improve the ways in which medicine is delivered, and advancing digital agricuture , which includes things like precision livestock farming, or precision agriculture that reduces waste and improves productivity more generally. As they interact with farmers, the clustering of university researchers and the ideas embodied in them alongside the farms that adopt those ideas represents this new type of industry cluster . Such a classification framework is a sea change from earlier industrial classifications based on what goods are physically produced - like manufacturing and agriculture ²¹.

The Future

Such a bottom-up classification and analysis system, based on local links between researchers and firms, could be designed locally but scaled nationally. It could address the challenges identified at the beginning of this piece. The definition of AI firms could evolve and be defined by the links between AI researchers and the firms with which they work. The lack of information about the needed AI capabilities would be resolved by the direct mapping of firm skill demand and their hiring patterns, as exemplified in New Jersey. The same New Jersey mapping could tie AI capabilities to occupational skills. And the direct impact of AI on job replacement or job augmentation could be mapped from the joined up university and workforce data.

Of course, much needs to be done. The implementation will depend on the success of the pilot, and the ability to build on existing assets. Not all states and universities have the capacity to build a similar system, but the fact that 30 universities and 15 state agencies are participating in advisory boards for the NSF Industry of Ideas pilot is grounds for hope. Indeed, a new generation of data leaders is leading the way, not only at the local and regional government level but also at universities and professional associations (Advisory Committee on Data for Evidence Building) ²².

We began this paper by noting that the urgency to create timely, local, and actionable labor market information has never been greater. We close by arguing that our capacity to fundamentally change the way in which we can use data and information to understand the demand for skills and the structure of work has also never been greater. The opportunity is ours for the taking.

Know your goals

The first point to note is that checking how well the output from an AI model matches the data in the training set is not an adequate indication of how well it will perform once deployed on other data. The problem can be exemplified by considering a simple model based on an equation that best fits a training data set. Data values are inevitably subject to measurement uncertainties and local conditions that add various types of noise, so taking the line defined by the equation identified as best matching the training data and measuring how good that match is falls short of adequate evaluation - the more perfectly a model matches this noisy data, the less perfectly it will fit an alternative set of data, a scenario described as “overfitting”. Similarly, what a machine learning or AI algorithm or model learns when it optimises its fit to the training data may not be generalisable.

There are a number of possible approaches and factors to take into account when sourcing test data but the first thing to consider when drawing up a process for evaluating an AI model is its objective. With this objective in mind it is then possible to pin down an appropriate measure of performance, which will shape how to use the test data to evaluate the model performance. Among the distinguishing factors between different measures used to evaluate how a model performs on test data, some will be suitable when the objective is to classify (e.g high or low risk based on health data?) while others are useful for models that estimate or predict (e.g What is the estimated height of a child given their parents’ heights).

Classification model performance can be measured using accuracy, confusion matrices, sensitivity, specificity and the receiver operating characteristic (ROC). Classification accuracy summarises the performance of a classification model as the number of cases in which the model correctly classifies divided by the total number of cases used in the test set. However, this can be a blunt tool as there are cases where there is a different cost or consequence depending on the direction of the error. Confusion matrices are helpful to explore how the model performs in correctly classifying the different classes. The confusion matrix sums up the number of cases the model classifies correctly within each of the classes, for example how many actual high-risk cases are correctly classified as high risk by the model. The number of cases the model classifies as high risk, for example, that are not high risk is referred to as the False Positives. In the context of medical tests (e.g the covid lateral flow tests) testing positive for a condition that is not actually there is potentially less damaging than testing negative when the condition is there.

Additionally, the sensitivity and specificity can provide a more detailed look at model performance. The sensitivity refers to the proportion of cases labelled as positive that are classified as positive by the model, whereas specificity refers to the proportion labelled as negative that it classifies as negative. It is useful to visualise model performance and the receiver operating characteristic (ROC) provides a method to do just that. The ROC plots the True Positive rate against the False Positive rate for the model. This can be further summarised in one value as the area under the curve (AUC). The larger the AUC the better the model is performing.

Deciding whether accuracy is enough or whether there is a need to delve into the directions of the errors depends on the context of the model’s deployment. Other examples in medicine include the risk models that were developed to assess an individual’s risk of a specific medical condition, such as QRISK ¹ which calculates a person’s risk of developing a heart attack or stroke over the next 10 years. Here model performance needs to go beyond accuracy and consider the direction of the errors it makes. A good overview of performance evaluation is outlined by Flach (2019) ². Is it better to tell someone they may be at risk of disease X, run a blood test and rule it out (False Positive) than to tell them they are not at risk and not check (False Negative)? All this needs to be considered and factored into the validation of the model. It is worth noting that a systematic direction for its errors can also cause an algorithm to hit ethical problems.

When evaluating the performance of models that are estimating a numerical value (e.g height of child from height of parents) the measures used are based on how far off the model’s estimate is from the actual value (which is known for testing data). There are then a multitude of ways of summarising that quantity. The mean square error (MSE) is computed by taking the average squared difference between the estimated values from the model and the actual value in the data. Other variations include root mean square error (RMSE) and mean absolute error (MAE). The RMSE is computed in the same way as the MSE but the value is square rooted. The MAE takes a different approach by summing up the absolute errors (i.e. the error magnitude). Each of these three measures involve dividing the value obtained by the number of rows in the data. Depending on the context one of these measures may be better suited than others. For example the MSE is sensitive to outliers so can be easily skewed by a small number of extreme values, which may be useful to highlight them, whereas the RMSE has the advantage of being measured in the same units as the original variable the model is designed to estimate.

Large Language Models (LLMs, e.g. Gemini, ChatGPT) are also models trained on a data set and as such also need to be evaluated and monitored. Whereas in the models discussed so far there are some standard metrics, evaluating LLMs is more challenging as there are a multitude of benchmarks and metrics³. When LLMs are used to answer questions (when you ask a chatbot a question) then monitoring the performance of the model (the trained LLM) can involve anything. Is the answer correct? Is the answer clear? Is the answer biased? The possible metrics are varied and not as simple to capture in one measure. It is also possible to use a LLM to evaluate or score another LLMs’ answer to a question. However this adds its own risk as LLMs are not 100% accurate or consistent themselves, and they can hallucinate.

Getting data right

Not only is separate test data needed for an evaluation, but care is needed to ensure the test data is suitably representative. Similar requirements apply for test data as for the original training data to ensure the dataset is representative of the context the model will be deployed in. For instance, if an algorithm is being developed to handle photos from the UK, training and testing it on photos where the sun always shines may cause problems. The model needs to be trained and tested on a set of photos that include rain and clouds otherwise it cannot be assumed it will reliably classify such photos if they appear during deployment in the real world. Getting the training and test data set right may mean using a smaller more curated set than simply one that contains everything available.

These data sets also need to have reliable labelling i.e. the rows of data need to be accurate so that the model’s performance can be assessed objectively against a trusted “ground truth”. For example, if we want to evaluate the performance of a fraud transaction classification model using accuracy as the performance metric, then we need a reliable training data set with true fraud transactions to evaluate how good the model is at detecting them. A data set with a list of transactions that are not accurately identified (or labelled) as fraud or not is not helpful. Thinking about how some commercial LLMs are trained on all the data in the “internet” it is worth asking whether a smaller more curated and specific training set would be better for model performance as well as being more ethical and safer.

Several approaches for generating test data sets take training and test data as distinct subsets from the same initial data set ⁴. There are different ways of doing this to make the most of the data to evaluate the model as systematically and exhaustively as possible. Perhaps the simplest example is using a hold-out set, which involves taking all the data available and taking a random subset of the data to use for testing the model. Depending on how much data is available then this can be 50% or less.

A slightly more sophisticated approach is k-fold cross validation, which involves splitting all the data you have available into k subsets and then doing k iterations where in each iteration a different kth of the data is used as the testing data for evaluation of the model built by training it on the remaining (k-1/k) of the data. This is repeated k times each time using a different one of the k subsets for testing. The performance of the model can then be averaged over the k iterations. (The measure of performance can be, say, accuracy or sensitivity depending on the context). For example, if k is 3 then the data is split into 3, and each iteration will take a different 2/3 of the data as training data to build the model, and the remaining 1/3 as testing data to evaluate the model.

Bootstrap is a more computationally intensive approach and it involves creating multiple samples by randomly sampling with replacement from the original data. Typically, hundreds or thousands of and such samples are generated, each will be different. These multiple samples provide multiple versions of the training and testing data so the model can be evaluated on all these variations. As bootstrap relies on sampling with replacement this means that each row of data in the original data can appear multiple times in a sample training or test data during each iteration, or not appear in other samples. As with k-fold cross validation the performance of the model can be then averaged over these multiple iterations. It is important that bootstrap does not rely on only a handful of iterations. Both bootstrap and cross validation offer an opportunity to see how sensitive the model’s performance is to the characteristics of the test data, but when the data sets available are small, the use of the bootstrap approach provides a more robust way of estimating the model’s performance.

An approach that can be useful to test whether the performance of the model is sensitive to time is time-based splits. This involves taking a “sliding window” ensuring that data is split into back-to-back time periods. Using a back-to-back (sliding window) further ensures that the data the model is trained on is separate from the one it is tested on.

Maintained monitoring

Once an algorithm has been let loose it can be challenging to maintain any rigorous monitoring, but it is worth highlighting the importance of taking on the challenge of ongoing monitoring and promising approaches to it. Some of the same metrics will apply to keep a handle on the myriad of issues that could arise. These range from the banal, such as data input errors, to the complex as is the case in model drift.

In the first case, if a model makes use of data that is fed into it from another system (e.g. a billing system) any update to this other system can affect model performance. Identifying this involves checking that the characteristics of the data used to train the model and the latest data fed into the model are not too dissimilar, since a difference in the data such as an increase by a factor of 10 or a hundred can cause the algorithm to fail. The magnitude of acceptable change in the data will depend on the context. Such a step change (due to source system update) in one of the model inputs can be identified and can potentially be an easy fix.

Model drift is more complex as real-world data evolves over time. There are two types of model drift: data drift and concept drift. Data drift refers to the change that can occur to data over time, whilst concept drift⁵ is a deterioration or change in the relationship between the target variable and input variables of a model. An example of data drift could be in the context of billing data the addition of new price plans or phones to the data, whilst an example of concept drift can arise when there is a change in the relationship between the effect (for instance, leaving one mobile phone provider for another) and underlying factors changes. In the context of the mobile phone provider market, a concept drift may mean that leaving for another provider is no longer dictated so much by price sensitivity as the type of network. Both types of drift lead to a deterioration in performance of the model as time goes by. Performance monitoring of the model is key to detecting model drift but differentiating between data or concept drift requires additional specialist approaches. Some of these are outlined in (Rotalinti, 2022)⁶ and (Davis, 2020)⁷.

In some cases, refreshing a model to account for the change in the underlying data (both training and test) can be quick and easy. However, if concept drift is detected, then it may take more than just a model refresh as the relationships between the variable we are trying to model, and the explanatory data has changed. This may involve finding new data sources and could lead to significant changes in the model, for example moving from a regression model to a neural network. Deciding to rebuild or retrain a model can also in some cases have environmental impact (particularly for the more resource intensive models such as deep learning and LLMs). Either way, where models are subject to peer review or some form of governance this can be a more onerous task.

Even with each step in a model’s evaluation stringently adhered to it is also important to assess the context for its deployment for risks and rogue scenarios that might break or in the case of Tay despoil it. And like all other stages of the evaluation this should not just be at the time of deployment but also over time. When models (machine learning or other) are used to inform or make important decisions providing information on how and when the model was evaluated, and how it is monitored should be standard practice not just to avoid the wasted expense of another broken AI model (algorithm) left on the shelf but more importantly to safeguard the welfare of those who come into contact with it.

Bias in bias out

In 2016 when ProPublica launched an investigation into potential biases in a ‘risk assessment’ algorithm used by the US criminal justice system, it was the first independent investigation of its kind. This was despite the widespread use of the algorithm and its power to influence a judge’s sentence, in one instance doubling the duration while increasing the severity of the imprisonment. On examining 7000 risk assessment scores and the records detailing whether the subjects of those scores had reoffended in the subsequent two years, Propublica found “Only 20 percent of the people predicted to commit violent crimes actually went on to do so”. Even when the full range of crimes was taken into account “the algorithm was somewhat more accurate than a coin flip” at 61%. Part of the enthusiasm for these algorithms had been the expectation that they might bypass the prejudices and unconscious biases of human judges, enabling fairer justice. However, while many might baulk at the thought of tossing a coin to determine someone’s prison sentence, it turns out this might be a fairer approach than the algorithm, which was found to “falsely flag black defendants as future criminals” at twice the rate of white defendants.

Since Propublica’s investigation there have been multiple reports highlighting problems with algorithms trained on historic data for use in the criminal justice system. The risk illustrated here, which can be generalised, is that such algorithms will tend to propagate social biases. In this case it means that those from ethnic minorities and lower socioeconomic backgrounds are awarded harsher sentences. Compounding the problem was the proprietary nature of the algorithms involved, which made it difficult to launch independent investigations. However, in the case of the algorithm investigated by Propublica, the input data, which is taken from questions put to the defendant and their prison records, did provide clues as to the scope for unfair outcomes. Although race is not explicitly identified, it likely correlates with other data that is used as input. This meant that the outcomes would be biased with respect to race all the same. A lot more work is needed to mitigate the effects of historical social injustices in how the criminal justice system uses data. Innovators in this area need to have confidence in what will be affected by their evidence base, as well as support from independent legal and ethical reviewers, and from regulators, to determine what will make a good innovation, and what will not.

Consent, human rights, and data provenance

The testing and training of AI algorithms can also run into other ethical questions about the ratio of public to private benefits from data and who governs those benefits. On the eve of the UK’s AI Summit in 2023, Joe Cuddeford of Smart Data Research UK wrote: “Many AI systems rely on data collected passively from individuals, raising questions about transparency, privacy, and who benefits from these data-driven advancements.”

Large scale AI models, such as generative AI models, are often trained on web-scraped data from online platforms. This leads to questions about the fairness of internet data, the ownership of it (e.g. potential violation of copyright law), and methods for users’ consent and human rights to be embedded and respected. There are, once again, questions about accuracy and bias: what do algorithms “learn” from data scraped from the internet, and is the information appropriately curated for use?

Civil liberties concerns also similarly arise when people are compelled to give up data about themselves by powerful arms of the state. For example, the national Facial Verification Testing program run by the National Institute of Standards and Technology, a part of the U.S. Department of Commerce, has held and made use of images of vulnerable individuals to test and validate the performance of commercialised AI technologies. The data used by the agency for testing include ‘mugshots’ or facial images from arrests or from other encounters with law enforcement. ¹ An additional programme focuses on testing the performance of facial recognition algorithms against an image database of sexually exploited children (CHEXIA-FACE). Having statistics from this kind of agency testing has clear commercial benefit: it helped win the case for the vendors who could match those statistics when the London Metropolitan police purchased live facial recognition technology. However, the interests of the people that have been documented do not come up for discussion in this form of data governance. There are many participatory methods that could be used for more ethical stewardship of the data that people are compelled to give. ²

To address the scope for minorities and vulnerable groups to play their part in data collection, it should be possible for data scientists to adopt strategies that consciously address the bias in data collection. Eun Seo Jo (from Stanford University) and Timnit Gebru (formerly at Google) have suggested library and archival approaches. In Strategies for Collecting Sociocultural Data in Machine Learning, ³ they note that internet data is subject to historical and representative biases. Recognising and mitigating biases will “start with a statement of commitment to collecting the cultural remains of certain concepts, topics, or demographic groups.” A public mission statement, which highlights the interests of communities and minorities they plan to support, “forces [archival] researchers to reckon with their data composition.”

These strategies also need to be supported by good management of data collection and curation. A report by the Royal Academy of Engineering (2021) Towards trusted data sharing: implications for policy and practice has highlighted that, to support the use of data for research, good data management must exist among the data owners. Strong relationships with data owners predicated on data quality and ethics will help researchers to specify what data sets they are looking for and how they can best be curated for AI purposes. Good data management will not only help AI developers but also all potential users (as well as the public) to understand the scope and the quality of what’s being shared. “Defining the requirements for data quality, and ensuring these requirements are delivered, remains a central challenge.” (RAE report)

Advocates for accurate and fair data and machine learning have worked hard to clarify what good data management and sharing looks like, which is cause for optimism. However there is the sense that this is the area in which AI has the furthest to go, as data sets currently available fall far wide from the standards recommended by their work. Nonetheless the rise of Trusted Research Environments, Data Safe Havens and other methods of open-source transparency enable more AI innovators to disclose their sources without placing any of the personal information they use at further risk, as discussed previously in the AI series. Leadership on ethical standards for data sharing may yet help to improve the robustness, security, safety, and fairness of AI tools, which the OECD advocates as key principles for AI.

Openness, explainability, and the scope to challenge AI decisions

A principle that many data science communities have been working on, is towards ensuring transparency and explainability of AI (OECD AI Principle). In OECD parlance that is in part “to ensure that people understand when they are engaging with [artificial intelligence] and can challenge outcomes.” In acknowledgement that some AI applications make this disclosure harder and more unappealing, the OECD suggests that the fact that AI is in use should be disclosed “with proportion to the importance of the outcome … so that consumers, for example, can make more informed choices”. The OECD emphasises the importance of the “explainability” of the algorithms, which it defines as “enabling people affected by the outcome of an AI system to understand how it was arrived at. … notably – to the extent practicable – the factors and logic that led to an outcome.”

The tens of millions of digital ‘platform workers’ that now live all over the world are a case in point for where explainability is needed. They perform short-term, freelance, or temporary work through digital platforms or apps in the “gig economy”. There is little transparency about how algorithms and AI influence outcomes for gig workers, and whether platform algorithms are contributing systematically to unfair outcomes. Platform workers themselves have come together to share their data to understand more about the outcomes of the algorithms, or AI, which is shaping their lives.

It follows that where the use of an AI system does not affect outcomes for people, there may be less of a demand to publicly justify how AI arrived at its outcomes. For example, where AI is used to simulate something, or to research a decision, rather than to make a decision, there could be less weight placed on explaining the model publicly.

François Candelon, Theodoros Evgeniou, and David Martens, writing for the Harvard Business Review have outlined that their preference is for accuracy as well as explainability. Often, to strike this balance, they will prefer ‘white box’ models which are transparent and interpretable. But not always. “In [complex] applications such as face-detection for cameras, vision systems in autonomous vehicles, facial recognition, image-based medical diagnostic devices, illegal/toxic content detection, and most recently, generative AI tools like ChatGPT and DALL-E, a black box approach may be advantageous or even the only feasible option.”

Even where the algorithm is too large and complicated to be interpretable, work like that conducted by the Alan Turing Institute in Project ExplAIn finds ways of extracting some kind of explanation, for instance by embedding layers in the coding. The case for opening up AI in this way has to be balanced against concerns for intellectual property, information security and privacy. There can be cybersecurity issues with making the different layers of an AI model more open to interrogation. Nonetheless, experiments with transparent and explainable models enable developers to advance their understanding of AI, as well as to consider whether its use for decision-making is ethically sound. The OECD principles make clear that it is important that AI doesn’t elude human insight, checks and balances. As Andrew Ng highlighted in the RSS fireside chat in 2021: “AI is increasing concentration of power like never before…governments and regulators need to look at that and think of what to do.”

Appropriate, human-centred governance

When school exams in England were cancelled during the Covid-19 pandemic, the government’s Department for Education decided that an algorithm should be used to allot grades to A-Level students, partly as a measure to counter grade inflation (a trend in which the grades awarded for the same standard of work will tend to rise, year on year). Algorithms had been used before in previous years to adjust the marks that were awarded for exams and coursework. Here instead of exams and coursework, the input data was gathered from Ofqual’s historical records about how particular schools’ pupils had performed in previous years, and some was generated by teachers. Efforts had been made at transparency in terms of how the new algorithm would arrive at these decisions (it was a relatively simple, white box algorithm). But there were ‘outliers’ acknowledged in the model even prior to deployment. Coupled with the widespread downgrading of teacher-estimated grades to fit a curve that would avoid grade inflation, there was not a clear process by which students and schools could appeal to change their grades. Dissatisfaction with the grades awarded in the absence of exams or coursework was rife, as young people regarded as academically talented by their schools fell short of the grades their teachers had predicted, and lost university places.

In the resulting furore, the Department for Education determined that its original policy was wrong and adopted the teacher estimated grades with an appeal process in place. The incident demonstrates that achieving the functional transparency of an algorithm is only one step in due process. Controversial policies could be using an algorithm to apportion losses across the population (e.g. to try to reduce grade inflation) in ways that are abhorred by individuals.

Vested interests also surfaced during investigation of an algorithm brought into use to tackle fraud in India’s welfare system. “From 2014 to 2019, the government of Telangana “cancelled more than 1.86 million existing food security cards and rejected 142,086 fresh applications without any notice.” reported Al Jazeera in January of this year. Despite the government’s initial claims that the cancelled food security cards were fraudulent, critical data scholarship in India and elsewhere has established discrepancies and errors in the algorithms used, such as, confusing the records of a valid claimant with a car-owning citizen by the same name. (Under the government’s policies, SUV owners cannot receive food aid.) Further investigations revealed that at least 7.5 per cent of the food security cards were wrongly cancelled. The investigations highlight what can be a common problem: a focus on reducing the costs of welfare programmes tends to lead services to identify false positives - wrongful claimants – rather than false negatives. Thus efforts to correct sloppy data may meet resistance if this leads to fewer “frauds” being identified, even when citizens bring evidence to challenge it.

There is a similar type of example in the UK’s Post Office scandal, in which many sub-postmasters were wrongfully prosecuted for false accounting, after the Post Office adopted accounting software that contained significant bugs, which were covered up for many years. This similarly goes to show how far organisations can pursue wrongful judgements, and the life-changing consequences.

The EU’s new AI Act advocates a risk-based approach, to balance the desire to minimise the burden of compliance while ensuring the safety of people who may be affected by the implementation of AI algorithms. Systems assessed as high risk according to specific criteria are then “subject to strict obligations before they can be put on the market”.

Governments across the industrialised world have raised their hopes for AI that will help to drive increases in productivity, and do so safely in ways that are fairly constructed, making use of legitimate data sources, and with fair outcomes for society. The work of data scientists is integral to the foundations by which AI can be used for social good, from establishing protocols for data management and sharing, to understanding the workings of complex algorithms, and the use of large and unstructured data sources. Data scientists and researchers are getting closer to understanding what good looks like, not just in terms of the ethical values to uphold but the technicalities of the code and data involved. However, a great deal of not only data work but also other work also needs to be maintained to uphold the ideal of ‘AI ethics’. Support for well-established ethical and legal rights and principles, to meaningfully involve people in policies that will be affected by AI use, and to develop data governance and infrastructure. It is always possible that when we’re working on AI ethics, we find that there are fairer and more ethical approaches that should precede the use of AI.

“AI development raises a range of ethical questions for data practitioners, whether they are data scientists, econometricians, analysts, or statisticians,” Daniel Gibbons, Vice Chair of the Royal Statistical Society’s Data Ethics and Governance Section told Real World Data Science. Today, many data scientists would urge that ethical considerations precede the development of an AI algorithm and must inform its design and use, particularly for processes that significantly affect people, to ensure it does not propagate errors and injustices.

Obviating the obvious errors

Duplicated records can mask existing diversities within the data, diminishing the representativeness of important subgroups and leading to a biased training set and model outcomes. If duplication originates from data labelling issues, it can lead to fundamental challenges during the training of supervised models. ² In healthcare data, this issue can arise when linking data across multiple sources where each source holds different labels for the same data. ³

Missingness directly leads to loss of information required for training the algorithms on various real-world scenarios. It is typically addressed via two primary routes: deletion of missing rows or imputation. Deleting missing rows results in a reduction of the sample size and bias. For instance, when it comes to health data, using electronic medical records from a single health provider such as general practice may give rise to a lot of missingness in other aspects of an individual’s health such as their hospital records, pathology testing data or medical imaging. On the one hand, structured missingness can serve as an informative feature to explore within the data. However in cases where missing data causes pixelation in the comprehensive health picture we are attempting to construct, it often conceals an underlying narrative. ⁴ For instance, the COVID-19 response involved many initiatives across the AI community, however, during the early stages of the pandemic partial availability of data pixelated the picture and impacted models predictive ability which resulted in a minimal improvement according to the UK’s national centre for data science and AI report. ⁵

Imputing missing values can preserve the whole sample. However, the introduction of noise during the imputation process may compromise the quality of fitted models, contingent upon the proportion of missing records. One way to offset this may be to use larger and larger data sets that might inevitably include a fuller training picture to the algorithm, just as adding dots to a pixelated image makes it more and more clear what is depicted.

It is often perceived that for certain instances, particularly in the context of deep learning models, such as neural networks, the model itself is capable of handling missing values without explicit imputation or deletion procedures. ⁶ Is this really true in a real-world scenario? Are these models advanced enough to achieve an optimised performance even when data quality is not at an optimised level? ⁷ Köse et al. (2020) investigated the effect of two conventional imputation approaches: Multiple imputation (MICE) ⁸ and Factor analysis of mixed data (FAMD) ⁹ on performance of deep learning models. Their study endorsed the use of such explicit imputation approaches, showing an enhancement in model performance. ¹⁰

Data imbalance issues, arise within datasets where there is a disproportionate amount of information pertaining to a specific aspect. When imbalanced data, rich in focused information, is used to train an AI model, the model becomes adept at learning about the specific aspect but may struggle to generalise its findings to diverse scenarios, thus fostering overfitting where the model achieves accurate prediction on the training data but loses accuracy for any new test dataset.

Overfitting severely undermines the predictive performance of AI models on data beyond their training set, defeating the primary purpose of these models. For instance, out of all strokes that occur, approximately 85% are ischaemic, caused by blockage of blood supply to part of the brain, and 15% are haemorrhagic, caused by bleeding into the brain. Development of machine-learning and AI-based stroke predictive models are therefore being affected by this natural imbalance in the two types of stroke. ¹¹ This type of imbalance also exists in population wide studies where stroke itself is only present in a minority subsection of a healthy population. ¹²

The circumstances of data collection can lead to bias, so care is needed at early stages to ensure that datasets are representative of the real world. These types of error can be picked up at an initial Quality Assurance (QA) stage conducted to reveal any unexpected errors in data used by AI models. QA checks often involve principle checks on presented values to ensure they are the right data type and within the expected range and have the expected temporal coverage.

Finally the choice of features included in a data set requires consideration since this can also have implications on an algorithm. Taking another example from the COVID-19 pandemic, a group of researchers trained an algorithm on radiological imaging of COVID-19 patients where the position of the patient during radiology was present as a feature in the dataset. However, since more severe cases were lying down and less severe cases were standing up, the existence of this feature resulted in an algorithm that predicted COVID-19 risk based on the position of the patients. Here although the data included was correct, its inclusion in the dataset proved to be “wrong”.

Things get complicated

When AI algorithms encounter complex, unstructured data, the task of quality assurance suddenly balloons beyond tackling the three main errors highlighted above. Such circumstances require some kind of quality enhancement procedure, where datasets in the form of images or unstructured text go through a curation process which involves enhancement of the quality and standardisation of the format to the level required for integration into AI algorithms. This process of standardization of data is paramount across various domains, especially in healthcare where complex, unstructured health data holds transformative potential for AI driven advances that revolutionise diagnosis, treatment, and prognosis of diseases. From electronic health records to magnetic resonance imaging (MRI) scans and genetic sequences, this data offers a wealth of insights for AI models to learn from. Adopting standardised formats not only facilitates seamless integration of diverse datasets but also streamlines the development and deployment of AI models. However, unlocking this potential requires a strong foundation in high-quality, processed data which begins with standardisation.

One category of complex health data is neuroimaging of which a prime example is MRI. Different institutions will often employ different acquisition protocols and different ways of collecting and storing neuroimaging data. Above all, this can make it very difficult to integrate into existing workflows and processing pipelines, but it also makes it challenging to understand, compare and combine with other datasets. To address these challenges, the neuroimaging community has adopted the Brain Imaging Data Structure (BIDS) ¹³ – a standardised format for organising and naming MRI data which allows compliant data to integrate smoothly with existing workflows and AI models, streamlining processing and analysis. By embracing standardisation, we can pave the way for common processing tools to enable the generation of AI-ready data.

Next, comes pre-processing. Sticking with the neuroimaging example, MRI scans are susceptible to various forms of noise and artifacts, which can appear as blurring or distortions which, without proper processing, can be misinterpreted by AI models. Pre-processing typically includes steps for spatial normalisation and image registration, involving alignment of brain images from different individuals into a common reference model and alignment of different images of the same subject to a common template. This standardisation facilitates inter-subject and inter-study comparisons, enabling AI models to generalise effectively across diverse datasets. However, the multi-layer aspect of this process means that aligning data to a common template is dependent on the choice of template which itself can introduce bias if the template brain doesn’t accurately reflect the patient’s anatomy (due to age, ethnicity, or disease for example).

Once pre-processing is complete, you may want to combine datasets to increase sample sizes for your AI model. This is where harmonisation techniques ^{14 15} come in to deal with inconsistencies and variations in acquisition which can add noise and bias into a model. A typical technique for harmonisation in neuroimaging, known as ComBat ¹⁶, works by modelling data using a hierarchical Bayesian model and followed by empirical Bayes to infer the distribution of the unknown factors. The method is actually borrowed from genomics data but is applicable to situations where multiple features of the same type are collected for each participant, whether that be expression levels for genes, or imaging derived measures such as volumes from different regions. This is a crucial step for combining datasets to enable AI models to focus on learning the actual relationships within the data rather than struggling with inconsistencies across datasets. It also leads to models which can generalise better on unseen data.

Feeding hungry algorithms

The public good is at the heart of AI driven approaches and indeed, the aim is to develop models with optimised predictive ability that can be generalised to many scenarios. For this to be achieved a large and diverse training source is required. This is often referred to as data hungry algorithms. To provide a large amount of enriched training data for optimised model development two main approaches have been explored: federated analytics and synthetic data.

Federation is when data from multiple sources is made available for training and analysis of models designed to run on data that is not held in a single place, nominally called distributed models. It provides the opportunity to test algorithms in different populations/settings to ensure generalisability. In the context of patient-level health data, the data is often held institutionally. Enabling federation and trustworthy sharing of these datasets requires extensive attention to governance models and a common model between multiple organisations is a known catalyst of this process ^{17 18}

Generating synthetic data ¹⁹ from original data sources is a resource intensive mechanism. It requires the development of models on the real data to learn existing patterns, formats, and statistical properties within the original data from which it is possible to generate further synthetic versions of these data. When working with sensitive data such as health records, ensuring patient data is safe and secure is covered by information governance. Depending on how close the synthetic data source is to the original data, the same governance level may still be applicable when trying to bring individual/patient data from multiple sources together. A suggested solution to overcome the governance challenges in the context of synthetic data is to use a low-fidelity version of the original data which means a level of bias has been added throughout the synthesisation process to ensure safety and security of individual level data. ²⁰ While the low fidelity data sources are generated based on real data, it is worth noting that the rise in generative AI also poses a concern for data pollution, particularly where AI tools such as Gretel.ai ²¹ are used to generate synthetic data which may also be used to train AI models – the problematic case of AI training AI!

When using sensitive health data of patients a further layer is in place to ensure security of access. These are called Trusted Research Environments (TREs), secure technology infrastructures which play a crucial role in consolidating disparate data collections into a centralised repository, facilitating researcher access to data for scientific exploration. However, integrating data from various sources into AI models poses challenges due to differences in data collection methods and formats, hindering computational analysis. In response, the FAIR (Findable, Accessible, Interoperable, Reusable) data principles were introduced in 2016 to enhance the reusability of scientific datasets by humans and machines. ²² Adoption of FAIR principles within TREs ensures well-documented, curated, and harmonised datasets, addressing issues raised above such as duplicated records and missing data. ²³ Additionally, preprocessing pipelines within TREs streamline data standardisation, creating “AI research-ready” datasets. ²⁴

Access to real-world healthcare data remains challenging, prompting the development of AI models on open-source or synthetic datasets. However, these models often exhibit performance discrepancies when applied to real world data ²⁵ It is therefore imperative to provide researchers with secure access to real-world healthcare data within TREs, bolstered by robust governance and support mechanisms. Initiatives like the GRAIMATTER study ²⁶ and AI risk evaluation workshops ²⁷ exemplify efforts to facilitate AI model development and translation from TREs to clinical settings. By establishing governance guidelines and promoting FAIR datasets, TREs aim to become important resources for the AI research community. Providing standardised and curated data rich repositories that AI models can be developed on is a top priority in UK-TREs. Given the well-defined and secure governance environments of TREs they may also provide the basis for federated data analysis allowing researchers to combine datasets across TREs/data environments. In this way they can provide the large numbers that data hungry algorithms require, while avoiding the wide-ranging and myriad ways that data for a specific dataset can be “wrong”.

Also in the AI series:

What is AI? Shedding light on the method and madness in these algorithms Generative AI models and the quest for human-level artificial intelligence