The Critical Issue of Algorithmic Bias in Credit Scoring Models

One of the most concerning aspects of algorithmic bias is the limited recourse available to those negatively impacted, leaving them vulnerable to opaque decision-making processes. This challenge underscores the need for increased transparency in how credit scoring models are developed and deployed. Ultimately, proactive mitigation strategies are essential to ensure fairness and equity in financial outcomes.

One of the biggest issues surrounding algorithmic bias in credit scoring is that adversely affected parties usually have little or no recourse for appealing unfavorable decisions. This problem happens because most widely used algorithms still can’t explain how they reached specific decisions, leaving people in the dark and forcing them to trust the technology, even as it potentially ruins lives.

A November 2025 academic review revealed numerous flaws in financial algorithms and confirmed various impacts. They included systematic disadvantages for minority groups and miscalibrated credit scores for individual borrowers. The researchers also discovered that these issues appeared despite the financial technology industry’s promises of superior efficiency.

One of the cited studies consistently showed that female applicants received credit scores six to eight points lower than their male counterparts. The researchers determined that the associated effects diminished economic welfare and that the ramifications continued for multiple borrowing cycles. Another investigation revealed persistent disparities across minority groups, despite the applicant’s chosen lender type.

Elsewhere, researchers examined the effects of using large language models to evaluate applicants’ loan data. This approach regularly recommended charging higher interest rates to Black applicants or denying their applications. It did not make the same suggestions for identical white applicants. These examples demonstrate why data science professionals must remain constantly aware of the potential for bias and uphold fairness by reducing the issue whenever possible.

Practical Tips for Bias Detection and Mitigation

Sources of bias in credit scoring data and algorithms are more common than you might think. They can include:

One straightforward way to identify bias in training data is to be aware of the most common types and build algorithms to be less reliant on them when possible. Regularly reviewing the training data is similarly effective because it can catch biases before they have real-life effects.

You can also perform a disparate impact analysis to mitigate bias, which compares aggregate measurements attributed to fewer privileges or less representation to their counterparts. Dividing the proportion of one group that received an adverse outcome and comparing it with that of the other group identifies bias.

Creating a set of fairness metrics is another practical mitigation approach, as it encourages data scientists to understand the impacts of various aspects of a person’s background that they can and cannot control. The examined attributes could include someone’s employment history, income and debt, but also the extent to which they experienced equal opportunities.

Improvements in explainable artificial intelligence will further both bias detection and mitigation. Developers then see the heavily weighted but irrelevant factors that could lead to unfair outcomes and correct issues early.

Considerations to Reduce Algorithmic Bias in Credit Scoring

Reducing algorithmic bias in credit scoring requires cooperation across roles and departments and includes those who will use the tools containing the algorithms. Committing to specific postprocessing steps empowers people to become more familiar with an algorithm’s functionality and potential shortcomings rather than automatically trusting the results. Those developing the algorithms should prioritize transparency by designing explainable models when possible and making them accessible enough for the expected audience.

Staying abreast of recent research shapes data scientists’ efforts by helping them understand the possibilities. In a 2024 case study, MIT researchers created a new technique that identifies and eliminates the specific attributes of training data that are the strongest contributors to a model’s biases about minority subgroups. This approach also preserves overall accuracy because it preserves more of the data compared to other options.

The developers confirmed that the technique can find hidden bias sources and training datasets with unlabeled information. This capability is significant because the data used by many applications lacks labels. They envision combining their technique with other approaches to improve fairness in high-stakes situations. This detail makes it well-suited for the financial industry because many of the associated decisions alter people’s lives and opportunities.

Maintaining responsible data science practices requires equipping professionals with the skills to detect and mitigate bias in an evolving technological landscape. A 2025 study involved a biased dataset that contained a disproportionately high number of white people with happy faces. This issue caused the AI algorithm to correlate race with emotional expressions.

The results of three experiments with human participants showed that most individuals did not notice the bias. This result shows why data scientists need ongoing education to spot less-obvious examples.

Stay Vigilant to Maintain Fairness

Your work on algorithms for credit scoring could adversely affect people’s lives and leave them with no way to contest unfavorable outcomes. Being a responsible data scientist means understanding the numerous risk factors and the controllable factors to minimize harm. Remaining aware of emerging AI applications in the financial industry and regularly meeting with colleagues to discuss ways forward increases fairness for everyone.

Three Reasons for Failure

There are three important reasons why so many projects fail that are often overlooked in the literature. First, the problem is often mis-specified: it is framed by technologists or vendors rather than co-owned by the domain experts who understand the decision and bear the consequences. Second, leadership expectations are frequently misaligned, short time horizons and demands for certainty collide with a technology that improves through iteration and organisational learning. Third, many deployments are brittle: they assume stability in a domain defined by rapid model change and rising user expectations, when what is needed is an engineered system designed to adapt.

This article draws on two decades of work building AI systems for medicine discovery at GSK (7) and at Tangram (8) to argue that success rests on three principles, in roughly this order:

• integrated subject matter expertise;
• patient and informed executive leadership;
• and building AI systems that learn with the organisation.

The sections that follow develop each element in the context of drug discovery and show how an AI platform can help move real projects into the small minority that deliver value.

Essential Contexts Outside Our Focus

Before turning to the three principles developed below, it is worth acknowledging four adjacent domains that I will not treat in detail here, each of which now has a substantial literature of its own. First, organisational scholars have long shown that new information systems reshape power, status and discretion, making the politics of implementation as important as the technology itself (9); (10). Second, governance, regulation and responsible AI practice which includes everything from model documentation and auditability to privacy, robustness and safety, have become central determinants of what can be deployed in practice, especially in regulated sectors (11); (12); (13). Third, there is an emerging body of work on workforce transformation: how AI complements or displaces skills, how hybrid human–AI roles are designed, and how training, trust and professional bodies mediate adoption (2); (3); (4); (5). Finally, the question of how to measure value and learn at portfolio scale with existing legacy IT systems (through experimentation, counterfactuals and disciplined comparisons between use cases) is itself a rich field that extends well beyond any single organisation. (3); (4); (5); (6); (14). Each of these strands is critical to understanding why AI succeeds, stalls or remains at a proof-of-concept level.

To maintain focus, I concentrate on three overlooked questions arising from direct experience: how to organise subject matter expertise such that the enterprise owns its AI; how to cultivate genuine leadership ownership; and how to engineer systems that learn and adapt rather than remain isolated demonstrations.

Principle 1: The importance of building with subject matter experts

The hardest part of building an AI platform is not the models or the engineers but assembling the subject matter experts (SMEs) who will frame and judge the work. Most commentary treats SMEs as validators, brought in at the end to bless a prototype. It rarely explains how to organise a molecular biologist, a clinician and a chemist so that they can state, in plain terms, what counts as an acceptable outcome.

Most commentators are not operators. They observe patterns across organisations but do not live with the consequences of poor SME integration. This distance between writing and practice shows up in the surveys. Foundry’s 2024 State of the CIO, summarised in MIT Sloan Management Review, reports that 85% of IT leaders see the CIO role as a driver of change, yet only 28% list leading transformation as their top priority (15)]. The people commenting on AI often sit with strategy decks rather than with the unglamorous work of managing technological change and cross-functional coordination.

Drug discovery starkly exposes the gap. The relevant team is wide and requires exceptional coordination. Business and portfolio leaders understand how projects absorb capital and create value whereas molecular biologists and geneticists judge whether a gene is plausibly causal for a disease. Clinicians think through trial design and patient risk. Chemists know what can be made and delivered. Statisticians, AI engineers and data scientists understand models, data pipelines, experimental design and evaluation. This diversity is a strength but requires a lot more from leaders of such teams. When these groups work as separate silos, the result is a generic set of tools whose outputs are not trusted by the users and whose inputs are irrelevant. When these experts can operate as a single team, the conversation starts with a simple set of questions. Which decisions are we trying to improve? How will we know if we have succeeded? What data and statistical methods count as acceptable evidence? Which risks are we prepared to take and which are not negotiable?

At Tangram, that joint framing often collapses into one critical choice: which disease do we want to target for drug development, and which gene is driving it? That decision already embeds genetics, hepatocyte biology, chemistry, clinical feasibility and commercial context. The role of AI and engineering is then precise. It is to help the group search the vast hypothesis space, structure the evidence and quantify uncertainty, while leaving the final judgement with experts who feel they own the AI platform, can see why the AI has come to the conclusions it has, and also own the consequences.

Principle 2: Patient and strategic executive leadership.

The second element is executive patience. Research from MIT Sloan shows that firms gaining value from AI tend to run more projects, over more years, with a sustained focus on learning how people and AI work together. (3) Leaders in these organisations accept that early returns are small and uneven. They invest in a pipeline of use cases rather than a single bet. They resist what researchers have called the “last mile problem”: AI projects that reach technical proof of concept but never change how work is done. (14) In life sciences this is acute. Discovery timelines are long, data are messy and early signals are faint. Leaders who expect quick, clean returns tend to cycle through pilots without ever building an asset that scientists trust.

Patience does not mean passivity; actually, the opposite is true. It means choosing a small number of important decisions, funding cross-functional teams to attack them, and holding the bar for quality high. It requires senior sponsorship to unblock data access, align incentives across discovery, clinical and commercial groups, and shield long term work from quarterly fashion cycles. When those conditions are in place, AI stops being a sequence of demonstrations and starts to become part of how the organisation thinks: the informed leader knows the difference.

Principle 3: Building AI Systems that learn with the organisation

The third element is the AI engineering. The MIT findings on the 95 percent figure are instructive: most generative AI projects fail not because the models are weak but because the systems around them are brittle. (1); (11) Foundation models are dropped into existing workflows with minimal adaptation. There is limited monitoring. Data quality is assumed rather than measured. When something breaks (as it inevitably will in non-deterministic systems) teams revert to manual work.

Modern AI engineering starts from the opposite assumption. Models and tools will change quickly. The surrounding stack must absorb that change without being rebuilt each time. The strategy must be built so that, when the product director hears a new technology is built or an LLM improved, this is always a good day.

Build only what you must.

The sensible principle is to only build the components where your domain expertise creates defensible value. Everything else should be bought. But buying is not effortless. Integration, monitoring, and vendor management drains teams unless you are staffed for it. Organisations that succeed at scale partner with vendors offering systems that learn and adapt; they focus on workflow integration; they deploy tools where process alignment is easiest.

So assuming you have the right staff who are working together, supportive leaders and good vendor relationships the next problem is how to create a platform itself.

A useful pattern for a modern AI platform has four parts:

• First, the data stack. Discovery teams need a small number of trusted stores for data with clear provenance and at least basic quality checks. For early target selection this means human genetics, expression data, interaction networks, preclinical phenotypes, clinical outcomes and internal experiments. Traceability and reproducibility is critical here. Any claim a model makes about a gene and disease pair should be traceable back to specific pieces of evidence.
• Second, the platform needs to be built on modular services rather than monoliths. Each service has a single responsibility and can be swapped when a better service appears. This keeps the cost of change low and allows teams to combine external tools with internal components in a controlled way.
• Third, the system needs to have continuous evaluation. Every component that answers questions is tested on held-out tasks, with simple metrics for accuracy, faithfulness, and recall, and monitored continually. There should be repeat measures and other tests of robustness. (16) There is no reason not to report error bars in AI and yet they are rarely part of AI publications. Where this matters most is at the interface with non-determinism, inherent in large language models. A good medical AI assistant should give consistent answers even when questions are phrased differently. It should also say it does not know when the information is unclear or incomplete. (16); (17); (18)
• Fourth, include memory and reinforcement learning so that the system learns. This is the most difficult component to implement and the one most often deferred. A system that cannot learn from use will make the same mistake repeatedly. Even the most patient users will lose trust and patience. But building memory into production systems, where the model retains context across sessions and improves from feedback, requires specialist expertise in reinforcement learning, retrieval-augmented generation with persistent stores, and the infrastructure to support online learning without catastrophic forgetting (19). These skills are in high demand and short supply. The alternative is a system that feels potentially useful in demonstrations but frustrates users in daily work.

For this to work, engineering teams need to stay in constant contact with biologists, geneticists, clinicians, chemists and portfolio managers. Together they decide what error rate is acceptable for a triage tool, what form of uncertainty estimate a portfolio board will respect and where human review is mandatory, for example before a new target enters serious preclinical work. Work on human–AI interaction design reinforces this point: systems should explain what they can and cannot do, expose their confidence and make it easy for users to correct them. The hardest part is that the first version is almost never right. Cross-functional teams need patience and ownership. They contribute real examples, refine prompts and evaluation sets, and expect the system to learn from its mistakes. The AI platform must be useful enough, early enough, that experts are willing to spend scarce attention improving it.

A worked example: target-indication pairing in siRNA

At Tangram we built LLibra OS, an internal system designed to surface and assess new small interfering RNA targets (8). siRNA refers to short double-stranded RNA molecules that can silence a specific gene via the RNA interference pathway. The purpose is narrow: the AI needs to help scientists identify medicines worth taking forward.

In early discovery, hypothesis generation often reduces to a single question: which target and disease pair should we move into the siRNA pipeline? The question sounds simple. It is not.

A purely data-driven approach can surface millions of candidates. Genome-wide association, expression atlases, and protein interaction networks will produce statistical associations at scale. But association is not mechanism. Two things can correlate because they share upstream causes, because they sit in the same pathway without being rate-limiting, or because of confounding in the data.

Plausibility requires a different kind of evidence. If we modulate this target, what functional change should we observe at the cellular or tissue level? Does that functional phenotype connect credibly to the disease we care about? For our purposes, the chain of reasoning must pass through liver biology: does knockdown of this gene alter a measurable secretory or metabolic function, and does that function relate to the clinical phenotype we wish to treat? Is there a real unmet need for your research for patients? (20)

The AI assists in answering these questions. It helps the team hold multiple threads of conditional evidence in view simultaneously. It retrieves, reasons and summarises over tens of millions of papers in the literature, joins and flags inconsistencies between thousands of data sources and quantifies uncertainty where the evidence is thin. Whilst the AI platform does the work of thousands of researchers and continually learns on the job, the expert judgement remains with the SMEs. The SME is central and is given all the reasons why and how the AI found a piece of evidence. The AI structures and expands the space in which that judgement operates. When it works well the AI platform uncovers the non-obvious connections that researchers may never have found.

Conclusions

Seen from this angle, the 95 percent figure is not a verdict on AI technology but a statistic about organisational design: how rarely good questions, high quality data, diverse experts and committed leaders are brought together at the same time. The systems described in this essay matter, but they are secondary. The primary determinant of value is whether biologists, clinicians, chemists, data scientists and portfolio leaders sit together, own the same objectives and are backed by executives willing to invest over years rather than quarters. Where that integrated team is absent, even elegant architectures will fail. Where they are present, imperfect tools still move the needle.

Much commentary on AI spends its energy on model choice, technical detail and tooling. This article has argued that the more important work is organisational: deciding which decisions to improve, agreeing what counts as acceptable evidence, and creating cross-functional teams that can live with the consequences. The few organisations that succeed treat AI as an experiment in decision making rather than a procurement exercise. They expect the stack to change and the vendors to turn over, but they hold fast to the team, the questions and the discipline.

For Real World Data Science readers, the implication is direct. AI projects fail when nobody owns the estimand, the counterfactual and the error bars. AI doesn’t need to be perfect, but it needs to be good enough. As the statistician George E. P. Box famously observed: “All models are wrong, but some are useful”. Usefulness here depends on design, discipline and humility as much as model choice. Statisticians, data scientists and methodologists can reclaim the narrative by insisting not only that every AI project begins with a clear question, a credible experiment and a plan to learn, but also that these are held collectively by an integrated team with visible executive backing. That is how more organisations move into the 5 percent.

1. Start With the Simplest Possible Use Case

One of Knowbot’s core design principles was minimal friction. MHF looked for a “gateway use case”: a low-risk, easy-to-understand tool that organisations could immediately see value in and adopt quickly.

Practitioner takeaway: Don’t begin with the most ambitious AI project your organisation can imagine. Begin with an easy, low risk project that still delivers value and treat it as a learning experience..

2. Culture Matters More Than Budget

Hudson notes that AI readiness among nonprofits varies widely and isn’t correlated with organisational size. Some large charities are slow to innovate due to bureaucracy; some small ones are enthusiastic but unlikely to benefit.

Practitioner takeaway: When planning an LLM deployment, assess cultural readiness, not just technical readiness. Ask:

• Who are the internal champions?

• How much AI literacy exists?

• How cautious is the organisation by default?

This will drive adoption far more than infrastructure.

3. Build for Trust First, Then Functionality

The biggest obstacle MHF faced wasn’t the model, the infrastructure, or the code. It was accessing the right decision-makers and establishing trust with LLMs in general and Knowbot in particular.

Practitioner takeaway: AI deployments in nonprofits are trust projects as much as technical ones. Practitioners should:

• Engage early with leadership.

• Be explicit about risks and mitigations.

• Provide clear, responsible documentation.

• Avoid overclaiming what the model can do.

The more transparent the process, the smoother the adoption.

4. Where Appropriate, Restrict the Model’s Knowledge Domain to Reduce Risk

Knowbot deliberately confines itself to the content on the host organisation’s website(s), plus general internal LLM knowledge. It doesn’t trawl the open internet. This dramatically limits opportunities for hallucinations, unsafe advice, or reputational risk.

Practitioner takeaway: Whenever possible, design LLM answer engine systems that operate on curated, organisation-owned content. Domain restriction is one of the most effective forms of practical AI safety.

5. Expect Surprising User Behaviour — And Design for It

One of the unexpected patterns in early usage: people asked Knowbot, “Who are you?” This required the team to add a new prompt component and require every partner to host a “What is Knowbot?” page.

Practitioner takeaway: Build processes for:

• Unexpected inputs

• Prompt evolution

• Iterative refinement

LLM deployment is never “set and forget.”

6. Technological Timing Matters — And Keeps Improving

Hudson emphasised that many capabilities now considered standard (e.g. long context length, ring fenced access to specific types of knowledge, server deployment ease) would have been impossible even a year earlier. The tools needed to fulfil a nonprofit’s evolving needs often appear in the LLM ecosystem soon after the nonprofit requests new functionality - making the decision whether to ‘build custom’ or ‘wait’ a tricky one.

Practitioner takeaway: Stay current. Model capabilities, guardrails, and hosting options evolve at high speed. What was impossible last quarter may be trivial today.

7. Value Impact Over Volume

Knowbot’s LLM processing costs MHF money, and soKnowbot’s team evaluates success not just by the number of questions answered but by the relevance of those questions to valuable decision-making. A tool that helps a policymaker or researcher retrieve something critical can have outsized impact.

Practitioner takeaway:When measuring impact, develop metrics that capture qualitative value, not just quantitative usage. For example you might consider:

• Complexity of queries

• Decision relevance

• Equity of access

• Whether the tool reduces burden on staff

8. In Fast-Moving Environments, Admit What You Don’t Know

Both Knowbot and TestRAMP were built in contexts where knowledge was changing daily. Hudson emphasises the importance of asking “naïve” questions, learning quickly, and not pretending expertise where there is none.

Practitioner takeaway: Cultivate humility. Curiosity and fast learning beats early certainty. Pair technical exploration with organisational openness about unknowns.

9. Relationships and Partnerships Are Everything

Across both initiatives, success depended less on algorithms and more on building new human relationships.

Practitioner takeaway: AI for public good is a team sport. Map stakeholders. Share progress transparently. Community buy-in creates technical resilience.

10. The Next Frontier: Agentic AI

Hudson argues we’re at a turning point where AI will expand from being “retrieval engines” to becoming “agentic systems that can do things.” With that shift comes both opportunity and new categories of risk.

Practitioner takeaway: Prepare now for agentic systems. Start with controlled automation, clear constraints, auditable logs, and robust governance. Retrieval is only the beginning.

Read our full conversation with Mike Hudson here.

Find out more about the Mike Hudson Foundation here.

Mike Hudson is an entrepreneur in technology & electronic markets. He now uses his expertise to help solve social problems. Mike founded TestRAMP, a pandemic nonprofit social market described as a “major contribution to Covid PCR testing & genomic sequencing” & donated its £2.4mn profits for charity. Mike is a Fellow of ZSL & adviser to its CEO. He is an honorary Research Fellow at City, University of London. Mike is a member of the Responsible AI Institute. He is a Foundation Fellow at St Antony’s College, University of Oxford.

2 What Agentic AI Looks Like in Practice

If you’re imagining robots in lab coats, that’s not quite what this is. It is more like releasing a highly motivated intern into a complex archive with partial instructions, limited supervision, and the freedom to decide which filing cabinets, databases, or tools to open next. It is messy. It is unpredictable. And it sometimes surprises you with just how resourceful or confused it can get. Agentic AI systems are purpose-built setups where a large language model is given a task and enough autonomy to decide how to approach it. That might mean choosing which tools to use, when to use them, and how to adapt when things go off-script. You are not just sending one prompt and getting an answer. You are watching a system reason, remember, call APIs, retry when things go wrong, and ideally, get to a useful result.

At Bayezian, we have explored this in several internal projects, including generating clinical codes from statistical analysis plans and study specifications, monitoring synthetic Electronic Health Records (EHRs) for rule violations, and running chained reasoning loops to validate document alignment. These efforts reflect the reality of building LLM agents into safety-critical and compliance-heavy workflows. Across these deployments, the question is never just “can it do the task” but “can it do the task reliably, interpretably, and safely in context”.

Broader research has followed similar directions. In clinical pharmacology and translational sciences, researchers have explored how AI agents can automate modelling and trial design while keeping a human in the loop, and offering blueprints for scalable, compliant agentic workflows link. In the context of patient-facing systems, agentic retrieval-augmented generation has improved the quality and safety of educational materials, with LLMs acting as both generators and validators of content link. Other teams have used multi-agent systems to simulate cross-disciplinary collaboration, where each AI agent brings a different scientific role to design and validate therapeutic molecules like SARS-CoV-2 nanobodies link.

Some of the systems we built used agent frameworks like LangChain or LlamaIndex. Others were bespoke combinations of APIs, function libraries, memory stores, and prompt stacks wired together to mimic workflow behavior. Regardless of the architecture, the core structure remained the same. The agent was given a task, a bit of autonomy, and access to tools, and then left to figure things out. Sometimes it worked. Sometimes it did not. That gap between intention and execution is where most of the interesting lessons sit.

In the next section, I describe one of those deployments in more detail: a multi-agent system used to monitor data flow in a simulated clinical trial setting.

3 Case Study: Monitoring Protocol Deviations with Agentic AI

Why We Built It

Clinical trials generate a stream of complex data, from scheduled lab results to adverse event logs. Hidden in that stream are subtle signs that something may be off: a visit occurred too late, a test was skipped, or a dose changed when it shouldn’t have. These are protocol deviations, and catching them quickly matters. They can affect safety, skew outcomes, and trigger regulatory scrutiny.

Traditionally, reviewing these events is a painstaking task. Study teams trawl through spreadsheets and timelines, cross-referencing against lengthy protocol documents. It is time-consuming, easy to miss context, and prone to delay. We wondered whether an AI-driven approach could act like a vigilant reviewer. Not to replace the team, but to help it focus on what truly needed attention.

Our motivation was twofold. First, to introduce earlier, more consistent detection without relying on rule-based systems that often buckle under real-world variability. Second, to test whether a group of coordinated language model agents, each with a clear focus, could carry out this work at scale while still being interpretable and auditable.

To do that, we built the system from the ground up. We designed a pipeline that could ingest clinical documents, extract key protocol elements, embed them for semantic search, and store them in structured form. That created the foundation for agents to work not just as readers of data, but as context-aware monitors. Understanding whether a missed Electrocardiogram (ECG) or a delayed Day 7 visit violated the protocol required more than lookup tables. It required reasoning. It required memory. It required agents built with intent.

What emerged was a system designed not just to scan data, but to think with constraints, assess context, and escalate issues when the boundaries of the trial were breached. The goal was not perfection, but partnership. A system that could flag what mattered, explain why, and stay open to human feedback.

How It Was Set Up

The system was built around a group of focused agents, each responsible for checking a specific type of protocol rule. Rather than relying on one large model to do everything, we broke the task into smaller parts. One agent reviewed visit timing. Another checked medication use. Others handled inclusion criteria, missed procedures, or serious adverse events. This made each agent easier to understand, easier to test, and less likely to be overwhelmed by conflicting information.

Before any agents could be activated, however, an early classifier was introduced to determine what type of document had arrived. Was it a screening form or a post-randomisation visit report? That initial decision shaped the downstream path. If it was a screening file, the system activated the inclusion and exclusion criteria checker. If it was a visit document, it was handed off to agents responsible for tracking timing, treatment exposure, scheduled procedures, and adverse events.

These agents did not operate in isolation. They worked on top of a pipeline that handled the messy reality of clinical data. Documents in different formats were extracted, cleaned, and converted into structured representations. Tables and free text were processed together. Key elements from study protocols were embedded and stored to allow flexible retrieval later. This gave the agents access to a searchable memory of what the trial actually required.

While many agentic systems today rely heavily on frameworks like LangChain or LlamaIndex, our system was built from the ground up to suit the demands of clinical oversight and regulatory traceability. We avoided packaged orchestration frameworks. Instead, we constructed a lightweight pipeline using well-tested Python tools, giving us more control over transparency and integration. For semantic memory and search, protocol content was indexed using FAISS, a vector store optimised for fast similarity-based retrieval. This allowed each agent to fetch relevant rules dynamically and reason through them with appropriate context.

When patient data flowed in, the classifier directed the document to the appropriate agents. If any agent spotted something unusual, it could escalate the case to a second agent responsible for suggesting possible actions. That might mean logging the issue, generating a report, or prompting a review from the study team. Throughout, a human remained involved to validate decisions and interpret edge cases that needed nuance.

We did not assume the agents would get everything right. The idea was to create a process where AI could handle the repetitive scanning and flagging, leaving people to focus on the work that demanded clinical judgement. The combination of structured memory, clear responsibilities, document classification, and human oversight formed the backbone of the system.

Figure 1 illustrates a two-phase agentic system architecture, where protocol documents are first parsed, structured, and embedded into a searchable memory (green), enabling real-time agents (orange) to classify incoming clinical data from the Clinical Trial Management System (CTMS), reason over protocol rules, detect deviations, and escalate issues with human oversight.

Where It Got Complicated

In early tests, the system did what it was built to do. It scanned incoming records, spotted missing data, flagged unexpected medication use, and pointed out deviations that might otherwise have slipped through. On structured examples, it handled the checks with speed and consistency.

But as we moved closer to real trial conditions, the gaps started to show. The agents were trained to recognise rules, but real-world data rarely plays by the book. Information arrived out of order. Visit dates overlapped. Exceptions buried in footnotes became critical. Suddenly, a task that looked simple in isolation became tangled in edge cases.

One of the most frequent problems was handover failure. A deviation might be correctly identified by the first agent, only to be lost or misunderstood by the next. A flagged issue would travel halfway through the chain and then disappear or be misclassified because the follow-up agent missed a piece of context. These were not coding errors. They were coordination breakdowns, small lapses in memory between steps that led to big differences in outcome.

We also found that decisions based on time windows were especially fragile. An agent could recognise that a visit was missing, but not always remember whether the protocol allowed a buffer. That kind of reasoning depended on holding specific details in working memory. Without it, the agents began to misfire, sometimes raising the alarm too early, other times not at all.

None of this was surprising. We had built the system to learn from its own limitations. But seeing those moments play out across agents, in ways that were subtle and sometimes difficult to trace, helped surface the exact places where autonomy met ambiguity and where structure gave way to noise.

A Glimpse Into the Details

One case brought the system’s limits into focus. A monitoring agent flagged a protocol deviation for a missing lab test on Day 14. On the surface, it looked like a valid call. The entry for that day was missing, and the protocol required a test at that visit. The alert was logged, and the case moved on to the next agent in the chain.

But there was a catch.

The protocol did call for a Day 14 lab, but it also allowed a two-day window either side. That detail had been extracted earlier and embedded in the system’s memory. However, at the moment of evaluation, that context was not carried through. The agent saw an empty cell for Day 14 and treated it as a breach. It did not recall that a test on Day 13, which had already been recorded, fulfilled the requirement.

This was not a failure of logic. It was a failure of coordination. The information the agent needed was available, but not in the right place at the right time. The memory had thinned just enough between steps to turn a routine variation into a false positive.

From a human perspective, the decision would have been easy. A reviewer would glance at the timeline, check the visit window, and move on. But for the agent, the absence of a test on the exact date triggered a response. It did not understand flexibility unless that flexibility was made explicit in the prompt it received.

That small oversight rippled through the process. It triggered an unnecessary escalation, pulled attention away from genuine issues, and reminded us that autonomy without memory is not the same as understanding.

How We Measured Success

To understand how well the system was performing, we needed something to compare it against. So we asked clinical reviewers to go through a set of patient records and mark any protocol deviations they spotted. This gave us a reference set, a gold standard, that we could use to test the agents.

We then ran the same data through the system and tracked how often it matched the human reviewers. When the agent flagged something that was also noted by a reviewer, we counted it as a hit. If it missed something important or raised a false alarm, we marked it accordingly. This gave us basic measures like sensitivity and specificity, in plain terms, how good the system was at picking up real issues and how well it avoided false ones.

But we also looked at the process itself. It was not just about whether a single agent made the right call, but whether the information made it through the chain. We tracked handovers between agents, how often a detected issue was correctly passed along, whether follow-up steps were triggered, and whether the right output was produced in the end.

This helped us see where the system worked as intended and where things broke down, even when the core detection was accurate. It was never just a question of getting the right answer. It was also about getting it to the right place.

What We Changed Along the Way

Once we understood where things were going wrong, we made a few targeted changes to steady the system.

First, we introduced structured memory snapshots. These acted like running notes that captured key protocol rules and exceptions at each stage. Rather than expecting every agent to remember what came before, we gave them a shared space to refer back to. This made it easier to hold onto details like visit windows or exemption clauses, even as the task moved between agents.

We also moved beyond rigid prompt templates. Early versions of the system leaned heavily on predefined phrasing, which limited the agents’ flexibility. Over time, we allowed the agents to generate their own sets of questions and reason through the answers independently. This gave them more space to interpret ambiguous situations and respond with a clearer sense of context, rather than relying on tightly scripted instructions. Alongside this, we rewrote prompts to be clearer and more grounded in the original trial language. Ambiguity in wording was often enough to derail performance, so small tweaks, phrasing things the way a study nurse might, made a noticeable difference. We then added stronger handoff signals. These were markers that told the next agent what had just happened, what context was essential, and what action was expected. It was a bit like writing a handover note for a colleague. Without that, agents sometimes acted without full context or missed the point altogether. Finally, we built in simple checks to track what happened after an alert was raised. Did the follow-up agent respond? Was the right report generated? If not, where did the thread break? These checks gave us better visibility into system behaviour and helped us spot patterns that weren’t obvious from the output alone.

None of these changes made the system perfect. But they helped close the loop. Errors became easier to trace. Fixes became faster to test. And confidence grew that when something went wrong, we would know where to look.

What It Taught Us

The system did not live up to the hype, and it was not flawless, but it proved genuinely useful. It spotted patterns early. It highlighted things we might have overlooked. And, just as importantly, it changed how people interacted with the data. Rather than spending hours checking every line, reviewers began focusing on the edge cases and thinking more critically about how to respond. The role shifted from manual detective work to something closer to intelligent triage.

What agentic AI brought to the table was not magic, but structure. It added pace to routine checks, consistency to decisions, and visibility into what had been flagged and why. Every alert came with a traceable rationale, every step with a record. That made it easier to explain what the system had done and why, which in turn made it easier to trust.

At the same time, it reminded us what agents still cannot do. They do not infer the way people do. They do not fill in blanks or read between the lines. But they do follow instructions. They do handle repetition. They do maintain logic across complex checks. And in clinical research, where consistency matters just as much as cleverness, that counts for a lot.

This experience did not make us think agentic systems were ready to run trials alone. But it did show us they could support the process in a way that was measurable, transparent, and worth building on.

What This Taught Us About Evaluation

Working with agentic systems made one thing especially clear. The way most people assess language models does not prepare you for what happens when those models are placed inside a real workflow.

It is easy enough to test for accuracy or coherence in response to a single prompt. But those surface checks do not reflect what it takes to complete a task that unfolds over time. When an agent is making decisions, juggling memory, switching between tools, and coordinating with others, a different kind of evaluation is needed.

We began paying attention to the sorts of things that rarely make it into research papers. Could the agent perform the same task consistently across repeated attempts? Did it remember what had just happened a few steps earlier? When one component passed information to another, did it land correctly? Did the agent use the right tool when the moment called for it, even without being told explicitly?

These were not academic concerns. They were practical indicators of whether the system would hold up under pressure. So we built simple ways to track them.

We looked at how stable the agent remained from one run to the next. We measured how often a person needed to step in. We checked whether the agent could retrieve details it had already encountered. And we monitored how information moved through the system, from one part to another, without being lost or altered along the way.

None of this required complex metrics. But each of these signals told us more about how the system behaved in real use than any benchmark ever did.

4 A Call for Practical Evaluation Standards

If we want reliable ways to judge these systems, we need to start from what happens when they are used in the real world. Much of the current thinking around evaluating agentic AI remains too abstract. It often focuses on what the system is supposed to do in principle, not what it manages to do in practice. But the most useful insights emerge when things fall apart. When an agent loses track of its task, forgets what just happened, or takes an unexpected turn under pressure.

A recent assessment of Sakana.ai’s AI Scientist made this point sharply. The system promised end-to-end research automation, from forming hypotheses to writing up results. It was an ambitious step forward. But when tested, it fell short in important ways. It skimmed literature without depth, misunderstood experimental methods, and stitched together reports that looked complete but were riddled with basic errors. One reviewer said it read like something written in a hurry by a student who had not done the reading. The outcome was not a failure of intent, but a reminder that sophisticated language does not always reflect sound reasoning.

Instead of designing evaluation methods in isolation, we should begin with real scenarios. That means observing where agents stumble, how they recover, and whether they can carry through when steps are long and outcomes matter. It means showing the messy bits, not just polished results. Tools that help us retrace decisions, inspect memory, and understand what went wrong are just as important as the outputs themselves.

Only by starting from lived use with its uncertainty, complexity, and human oversight, can we build evaluation methods that truly reflect what it means for these systems to be useful.

5 Closing Thoughts from the Field

Agentic AI carries genuine promise, but even a single deployment can reveal how much distance there is between ambition and execution. These systems can be impressively capable in some moments and surprisingly brittle in others. And in domains where decisions must be precise and timelines matter, that brittleness is more than an inconvenience; it introduces real risk.

The lessons from our experience were not abstract. They came from watching one system try to handle a demanding, high-context task and seeing where it stumbled. It was not a matter of poor design or unrealistic expectations. The complexity was built in, the kind that only becomes visible once a system moves beyond isolated prompts and into continuous workflows.

That is why evaluation needs to begin with real use. With lived attempts, not controlled tests. With unexpected behaviours, not just benchmark scores. As practitioners, we have a front-row seat to what breaks, what improves with small tweaks, and what truly helps. That view should help shape how the field evolves.

If agentic systems are to mature, the stories of where they struggled and how we adapted cannot sit on the sidelines. They are part of how progress happens. And they may be the clearest indicators of what needs to change next.

About the authors

Francis Osei is the Lead Clinical AI Scientist and Researcher at Bayezian Limited, where he designs and builds intelligent systems to support clinical trial automation, regulatory compliance, and the safe, transparent use of AI in healthcare. His work brings together data science, statistical modelling, and real-world clinical insight to help organisations adopt AI they can understand, trust, and act on.

::: :::

1 Summing it up

We end where we began in the first article of our series. Through this four-part series, we introduced a Curated Data Enterprise (CDE) Framework (see Figure 1) that can guide the development and dissemination of statistics broadly applicable to addressing social and economic issues while ensuring replicability and reusability. The CDE provides the scaffold for scaling the statistical product development of interest to the US Census Bureau and broadly applies to official statistics agencies (Keller et al. 2022). We illustrated this through a use case on climate resiliency of skilled nursing facilities, highlighting the replicability and reusability of the capabilities that would benefit inclusion in a CDE.

As noted in the first three articles, the process begins with articulating purposes and uses through stakeholder engagement and continues by leveraging that engagement, including subject matter expertise, to inform statistical product development. Eliciting purposes and uses from stakeholders and data users is facilitated by asking questions such as:  

1. What questions keep you awake at night because you don’t have data insights to address them? What are those purposes and uses that you need statistical products to support?

2. How do we collaborate and engage with you to better understand your needs and help you identify gaps in understanding regarding purpose and use?

3. How do we prioritize what statistical products to develop first?

Examples of purposes and uses that drive new statistical products include accurately measuring gig employment (Salvo et al. 2022a), migration due to extreme climate events (Salvo et al. 2022b), the various dimensions of housing affordability (Wu et al. 2023), and addressing the undercount of young children (Salvo et al. 2023). Other topics that require multiple sources and types of data include creating a household living budget based on the minimum necessary to ensure an adequate standard of living (Lancaster et al. 2023) and using this budget as a starting point for measuring insecurity across components such as food or housing (Montalvo et al. 2023).

2 Developing an end-to-end (E2E) curation system

Purposes and uses defined in use cases are important to support the rapid development of statistical products. These use cases will capture the imagination of those working to address today’s critical issues and advance public understanding and trust in federal statistics. The above paragraph provides examples of purposes and uses for which we have developed use cases.

Use cases are a powerful mechanism to promote methodological research to develop and implement capabilities needed in a CDE. The objectives are to undertake research projects that have the potential to create statistical products with explicit purposes and uses that will exercise the end-to-end (E2E) curation components.

When implemented, these proposed use cases will demonstrate a sequence of capabilities needed to build the CDE, such as agile data discovery, reusing modules and data (including synthetic data), tracking the provenance of collected and generated data, reusing synthetic data and methods to integrate many types of data, conducting statistical analysis involving heterogeneous data integration, and reviewing data and statistical results with an equity and ethics lens. These steps will be captured in an end-to-end curation system.

1. Criteria for developing and evaluating use cases that will uncover the capabilities and research necessary to develop the CDE

Criteria are needed to evaluate, and partner with researchers and stakeholders in developing and implementing the capabilities to capture in the CDE. The choice of use cases, when curated, needs to provide unique insight into CDE capabilities and statistical product development. The capabilities to be developed include addressing some purpose and use that no single source of information can resolve, generating practical diagnostics to improve existing methods, creating pilot software, and validating new and improved statistical products. These criteria, developed through listening sessions and discussions with experts, guide the prioritization and selection of use cases and their evaluation after curation (see Table 2) (Keller et al. 2022).

2. An end-to-end curation process

Curation is an end-to-end process defined by the context of the purposes and uses that document the decisions and trade-offs at each step in the CDE Framework. The following curation definition will be used as it serves the CDE’s vision.

Curation involves documenting, for each statistical product, the inputs from which the product is derived, the wrangling used to transform the information into product, and the statistical product itself. Purposes and uses provide the context for each statistic and statistical product.

This definition has evolved from numerous stakeholder discussions via listening sessions and discussions with Census Bureau staff. (Nusser et al. forthcoming; Faniel et al. 2019; NASEM 2022).

As use cases are curated, the CDE capabilities will evolve to quickly develop statistical products. These curated use cases are integral to developing an E2E curation process for the CDE.  

3. Invitation to contribute purpose and use ideas for developing new statistical products

The CDE development aims to curate a significant number of use cases that address social and economic issues that have the potential to define capabilities to be built in the CDE. Initially, they are seeking ideas for purposes and uses to define these use cases and statistical products.

The skilled nursing facility use case included code, data, and documentation to calculate the probability of workers getting to work during a weather event, resilience indicators at the county or sub-county level, alternative skilled nursing home deficiency measures, and other capabilities.

Incorporating capabilities in the CDE

To accelerate the development of statistical products, the Census Bureau will develop use cases to articulate and create CDE capabilities. This requires identifying those valuable nuggets for learning and quickly translating and incorporating this information into the CDE. Examples of critical capabilities of interest are learning about the utility of synthetic data, the ability to aggregate data into custom geographies, and combining different units of analysis. The expected outcome is the creation of an innovative 21^st Century Census Curated Data Enterprise focused on purposes and uses that overcome the limitations and challenges of today’s survey-alone model.  

The 21^st Century Census Curated Data Enterprise development presents an opportunity for researchers to help drive the development of the CDE as the foundation for creating new statistical products. The US Census Bureau is seeking ideas for purposes and uses that will define new statistical products. They are interested in research projects (use cases) that are guided by the CDE framework as potential new statistical products. They want to learn from and understand your experiences in using the CDE framework, for example, what worked well, what challenges you faced, how each step in the framework was curated, and what capabilities are replicable and reusable for developing and enhancing statistical products.

1 Introduction

Here, we demonstrate how the CDE Framework can be implemented for a research use case related to skilled nursing facilities. The framework provides the guiding principles for ethical, transparent, and reproducible research and dissemination and the research process for developing the statistical product.

Across the US, federally regulated skilled nursing facilities (SNFs) provide essential care, rehabilitation, and related health services to about 1.3 million people. An SNF is a facility that meets specific federal regulatory certification requirements that enable it to provide short-term inpatient care and services to patients who require medical, nursing, or rehabilitative services. Their patients can be among the most vulnerable members of our society, and yet, historically, SNFs have not been incorporated into existing emergency response systems. For example, during the 2004 Florida hurricane season, SNFs were given the same priority as day spas for restoring electricity, telephones, water, and other essential services (Hyer et al. 2006). Even worse are the deaths of SNF residents in Louisiana following Hurricanes Katrina and Rita in 2005 (Dosa et al. 2008). This was still an issue in 2021. In Louisiana, 15 SNF residents died when evacuated to a warehouse during Hurricane Ida (2021), and 12 died in Florida as a result of Hurricane Irma (2017). In both instances, the deaths were attributed to extreme heat and lack of electricity (Skarha et al. 2021).

These events prompted the (The White House 2022) initiative, Protecting Seniors by Improving Safety and Quality of Care in the Nation’s Nursing Homes, stating, ‘All people deserve to be treated with dignity and respect and to have access to quality medical care.’

However, there are questions that need to be addressed to best protect SNFs and their residents. For example, how resilient are SNFs in extreme climate events? This use case demonstration shows how we built a new statistical product to address this question using the CDE Framework (Lancaster et al. 2023).

2 Purposes and uses

A skilled nursing facility (SNF) is a federally regulated nursing facility with the staff and equipment to provide skilled nursing care, skilled rehabilitation services, and other related health services (Medicare & Medicaid Services 2023). The context of this use case is to create a baseline picture of SNFs in Virginia and then integrate information on the risk of extreme flood events to assess facility and community preparedness – for example, how likely are the nursing staff¹ to make it to the facility in the event of a flood?

This use case has two parts. The first creates a baseline data picture of SNFs, bringing together data about the residents, nursing staff, and SNF characteristics. The second addresses two issues raised in the (The White House 2022) initiative: emergency preparedness and nurse staffing. We frame these issues into three purpose and use questions with the ultimate goal of creating statistical products that address these questions:

1. Can SNF workers get to work during an extreme flood event?

2. Are SNFs prepared for a flood emergency?

3. Can communities support SNFs during an emergency?

3 Statistical product development stages

Subject matter input and literature review

The subject matter experts consulted included nursing facility administrators, SNF resident advocates, demographers, and researchers. Our discussions and literature review informed us of the many federal policies governing SNFs regarding inspections and data reporting requirements (procedural data). In addition, we were told about non-public data sources on residents and SNF staff that were aggregated to the SNF level and provided to the public under a grant from the National Institute on Aging. This information was important since we had yet to come across this source in our data discovery process. The dialogue with experts and our literature review helped us generate a ‘wish list’ of variables we used to inform our data discovery process that we visualized into a conceptual data map (see Figure 1).

Data discovery

Data discovery focused on identifying data sources to address the purpose and use questions and was informed by the conceptual data map.

For the first question – Can SNF workers get to work during an extreme flood event? – we discovered and used proprietary synthetic population, transportation routes, building data sources, and publicly available flood data. The HERE Premium Streets proprietary data includes information about roads, such as type of road, speed limits, number of lanes, etc. The proprietary synthetic population data, Building Knowledge Base (BKB), are used to identify where SNF workers live and work to map transportation routes from home to work (Mortveit et al. 2023). Publicly available data from the Federal Emergency Management Administration (FEMA) provided flooding risk estimates along the routes from nursing staff homes to the SNF.

For the second question – Are SNFs prepared for a flood emergency? – we used Center for Medicare and Medicaid (CMS) SNF inspection and deficiency data as a proxy for preparedness. We also examined SNF residents’ physical and mental health to assess SNF emergency preparedness. For example, if most residents faced mobility challenges, the SNF would need more resources available during an emergency to move residents to a safer facility. We used data about residents from the Long Term Care Focus (LTCFocus 2022) Public Use Data sponsored by the National Institute on Aging (Brown University 2022).

We used data to measure community resilience, assets, and risks by geography at the county, city, and census tract levels to address the third question, Can communities support SNFs during an emergency? These data included:

• Health professional shortages area (HRSA 2022)
• Shelter facilities and emergency service providers data (Homeland Security: Geospatial Management Office 2022)
• Community Resilience Indicator Analysis and National Risk Index for Natural Hazards (FEMA 2022).

All data are provided in a GitHub repository along with their metadata, except for the three proprietary data sources. Articles about how the synthetic estimates are constructed are provided for two of these proprietary data sources. The third data source was obtained from a private-sector vendor whose data and documentation are proprietary; a link is provided to their website.

Data ingest and governance

All the public data, metadata, code, statistical products, data processes, and relevant literature on SNF policies and regulations are stored in a GitHub repository.

In our experience, data wrangling is the most time-consuming and challenging part of product development. This speaks directly to the benefit of the CDE; once a researcher has wrangled together multiple data sources, it can be made available to other researchers.

The two predominant issues with data wrangling for this Use Case included reconciling data sources that contain data on the same topic and creating linkages between data sources. For example, we reviewed three hospital data sources:

1. Homeland Security Infrastructure Foundation-Level Data (HIFLD) (DHS 2022)
2. HealthData.gov - COVID-19 Reported Patient Impact and Hospital Capacity by State (HHS 2022)
3. Map of VHHA Hospital and Health System Members (Virginia Hospital & Healthcare Association 2022)

We observed inconsistences and omissions across the three data sources including: 

• non-standard hospital names and hospital classification types
• inconsistent availability of hospital IDs (such as Medicare Provider Number)  
• conflicting geographic information, including address, latitude, and longitude.

We did not attempt to reconcile these inconsistencies for the demonstration but decided to use a single source for shelter facility and emergency service provider data. We used HIFLD data since they provided the most current data (DHS 2022). The use of these data reinforces the purpose of the use case – to illuminate the challenges in creating statistical products and what the Census Bureau would need to consider.

Similar inconsistencies made it difficult to link data sources using geographic variables. For example, we used shelter facility and emergency service provider data sources from the HIFLD – including hospitals, Red Cross chapter facilities, National Shelter System Facilities, emergency medical service stations, fire stations, and urgent care facilities – to calculate a metric for potential community support. The goal was to place each facility in a Virginia county or independent city. Virginia is divided into 95 counties, and 38 independent cities considered county-equivalents for census purposes, and in some cases, there is a county and a city with the same name (eg, Richmond County and Richmond City, each in different locations in Virginia). It was necessary to canonicalize the county and city names (when available), which meant aligning upper and lower cases, removing unnecessary characters, and distinguishing between county and city.²

The challenge with locating shelter facilities and emergency service providers within a county or independent city was using different variables to identify their location (latitude and longitude, address, ZIP code³, Federal Information and Processing Standard (FIPS) code, and county/city name). In cases where the data source only had a ZIP or FIPS code, a Department of Housing and Urban Development crosswalk was used to link the two codes; in other cases, a crosswalk that linked non-independent cities and towns to counties was used; and in others, a crosswalk that linked FIP codes to counties and independent cities. Researchers would benefit from exhaustive crosswalks between all variables on the same topic, such as location variables, facility names, and identification numbers, to reduce the time spent on data wrangling.

Regarding data products related to popular indices, such as climate disaster risks and community resilience, they are operationalized differently across the various departments and agencies within the federal and state governments and private and non-profit sectors. It is an enormous task to review the methodology and technology reports (if available) to understand their differences and decide which versions are most relevant (fitness-for-purpose) for a particular use case. Again, after reviewing the options for this use case, we determined that the National Risk Index for riverine and coastal floods from FEMA was the best option for climate risk estimates. The detailed technical report, National Risk Index Technical Document (FEMA 2021), provides a clear assessment of the assumptions and limitations of the data and a description of how the risk estimates were derived. Researchers would benefit from guidance on the numerous constructions of indices on the same topic. A use case on a specific index topic could be used to highlight differences and similarities among indices, which would help with data wrangling and fitness-for-use. Ideally, the use case could benchmark the various constructions and provide a statistical assessment.

3.1 Question 1: Can SNF workers get to work during an extreme flooding event?

Sufficient nursing staff is of significant concern to assure resident safety and quality of care.

Since proprietary synthetic population data and commercial sector digitized mapping data were used to construct the routes SNF nursing staff are likely to take from home to work, only an outline of the computational process used to identify the routes is provided. Publicly available data from FEMA were used to estimate flooding risk along a particular route. Below is a general description of the modeling steps and the proprietary data used to assess SNF vulnerability as a function of the nursing staff’s inability to report to work due to the transportation infrastructure (Choupani and Mamdoohi 2016).

Computational modules

Here is the basic outline of the process that uses proprietary data that starts at network construction and ends with routes. For more details, see the GitHub repository: Vulnerability of SNFs concerning Commuting.

1. Extract network data from HERE (2021 Q1 in this use case).
2. Process the extracted data to form a network suitable for routing. This includes inference of speed limits for road links where such data is missing.
3. Prepare origin-destination pairs. In this case, the list of locations pairs a worker’s home and work locations. The person is constructed in the synthetic population pipeline, and residences and workplaces are derived through the data fusion process used to construct the NSSAC building database.
4. Construct routes using the Quest router.

Once the routes to an SNF were established, the expected number of nursing staff at an SNF during a flood event could be calculated as the sum of the probabilities of each worker being able to commute to work during a flood event. A computational model was developed using the following data:

• SNF locations in Virginia from the Centers for Medicare & Medicaid Services (CMS);
• Home locations of workers at each SNF assigned from the synthetic population and Building Knowledge Base (Beckman et al. 1996; Mortveit et al. 2023);
• Virginia road networks; and
• FEMA census tract-level riverine and coastal flood risks.

Using router software, the Virginia road network was used from the HERE map data to compute each nursing staff’s likely route to their SNF. Routers are commonly used within transportation and traffic simulators. The router software used for this demonstration is a highly parallelizable router previously developed in BI NSSAC, known as the Simba router (Barrett et al. 2013).

The FEMA risk data provide the riverine and coastal flood risks for each census tract in Virginia. Given the routes, the FEMA riverine and coastal flood risks were used to estimate the probability of the nursing staff making it to work. The FEMA technical document National Risk Index Technical Document (FEMA 2021) provides information on how natural hazard risks are calculated. We use these risk estimates ranging from 0 to 100 as a proxy for the probability a worker can reach the SNF by dividing by 100. For example, we assume a risk is zero if there is zero probability of being unable to reach the SNF due to an extreme flood event.

In contrast, a risk of 100 indicates the roads are underwater, and the probability of being unable to reach the SNF is one. The maximum risks along transportation routes leading to an SNF range from 0 to 47 for riverine flooding and 0 to 40 for coastal flooding. We assume the combined value of the maximum riverine and coastal flood risks along a worker’s transportation routes, divided by 100, is the worker’s probability of not getting to work during a flooding event.

Since we do not have data on the exact home locations of the nursing staff, we estimated how many could reach the facility by taking a random sample (whose size is the CMS average daily nursing staff⁴ for an SNF) from the possible routes identified using the HERE Virginia road network. We calculated the average with a 95% nonparametric confidence interval. The 283 SNFs used in our research have an average daily nursing staff of 12,609. Using the above approach, we estimated that 10,005 (95% CI: 9,013, 10,700) or 79% can get work during an extreme flood event. The individual SNF nursing staff percentage who can make it to work ranges from 48% to 93%.

Figure 2 visualizes this analysis for the 283 SNFs ordered by the observed average daily nursing staff numbers at the facility from smallest to largest, displayed using the orange line. The black line indicates the expected number in an extreme flood event and the 95% nonparametric confidence interval (grey band). The code for Figure 2 is provided in the GitHub repository.

For example, in King George County, the SNF is Heritage Hall King George (Federal Provider Number 495300 in Figure 3), located near the Potomac River, which opens to the Chesapeake Bay. According to CMS, the Heritage Hall King George facility has an average daily skilled nursing staff of 41. Using the HERE Virginia road network, we identified 101 routes the staff could use to reach the facility. The combined maximum coastal and riverine flood risks along these routes ranged from 5.6 to 66.7; a random sample of 41 from the 101 routes gives an average probability of reaching the facility of 0.74 with a 95% nonparametric confidence interval of [0.65, 0.80]. These were used to estimate the average number of nursing staff at the facility, 30, during a flood event, along with a 95% nonparametric confidence interval [14, 38]. Publicly available data from the Federal Emergency Management Administration (FEMA) provided flooding risk estimates along the routes from the nursing staff home to the SNF along with proprietary road and building information.

3.2 Question 2. Are SNFs prepared for emergencies?

To address this question, we examined how prepared SNFs are for emergencies using annual inspection and deficiency data as a proxy for preparedness. CMS issues deficiencies to SNFs that fail to meet federal Medicare and Medicaid preparedness standards. Every deficiency is classified into one of 12 categories based on the scope and severity of the deficiency. There are two broad types of non-health-related deficiencies:

• Emergency Preparedness Deficiencies – There are four elements of emergency preparedness. They cover an emergency plan, policies and procedures, a communication plan, and training and testing.

• Fire Life Safety Code – The set of fire protection requirements are designed to provide a reasonable degree of safety from fire. They cover construction, protection, and operational features designed to provide safety from fire, smoke, and panic.

We calculated separate Emergency Preparedness and Fire Life Safety Code deficiency indices to combine them to create a single index to measure SNF preparedness and distinguish between high and low performing SNFs. The computation of the indices has four steps.

1. Number of deficiencies: For each SNF, the total number of deficiencies during the past four years, 2018-2022, was divided by the number of SNF inspections over the same period to estimate the average number of deficiencies per inspection.

2. Time to resolve deficiencies: We next computed the average number of days it took to resolve each deficiency.

3. Scope and severity of deficiencies: We then transformed the deficiency letter inspection rating for scope and severity to a numerical weight using the CMS technical guide, Care Compare Nursing Home Five-Star Quality Rating System (Medicare & Medicaid Services 2022),and averaged the ratings.

4. The estimates from these three steps were summed to compute separate Emergency Preparedness and Fire Life Safety Code deficiency indices (see Figure 4) and are provided for reuse in a .csv file on GitHub.

Figure 4 displays the results of an exploratory data analysis for each index. These analyses assessed fitness-for-use; we wanted to construct an indicator with sufficient variability to discriminate between high and low-performing SNFs. It is evident we accomplished this in Figure 4 there are SNFs with indices outside the main body of the data. We summed the Emergency Preparedness and Fire Life Safety Code indices and categorized them into high, medium, low, and no deficiencies.

3.3 Question 3: Can communities support SNFs during emergencies?

To answer this question, we computed a community resiliency index using the US Census American Community Survey and the guidance provided by the Homeland Security document Community Resilience Indicator Analysis: County-Level Analysis of Commonly Used Indicators from Peer-Reviewed Research (Edgemon et al. 2018). The index was constructed by summing the county (census tract) level percentages for the following variables:

• fraction employed
• fraction with no disability
• fraction with a high school diploma or greater
• fraction of households with at least one vehicle
• reverse GINI Index – so all indicators are in a positive direction.

Figure 5 displays the combined deficiency indices, Emergency Preparedness + Fire Life Safety Code, for each SNF with the choropleth map for the community resilience index at the census tract level. We also examined the number of shelter facilities and emergency service providers and the availability of medical staff per 10,000 residents. We constructed isochrones to establish the distance from the SNF to these potential sources of support. Working on this component of the use case highlighted the need for cross-agency data, pointing to the utility of future strategic partnering between the US Census Bureau, CMS, and FEMA.

In addition to describing the population using a resilience index, we also developed a measure to present the number of shelter facilities and emergency service providers (data from Homeland Security / Homeland Infrastructure Foundation Level Data) and the availability of medical doctors (MDs) and Doctor of Osteopathic Medicine (ODs) who provide direct patient care (HRSA 2022) (Figure 6). 

The number of MDs and ODs is described as a primary care health professional shortage area. HRSA defines these contiguous areas where primary medical care professionals are overutilized, excessively distant, or inaccessible to the population of the area under consideration. Figure 6 (bottom) shows that approximately one-third of the counties and independent cities have health professional shortage areas across their entire boundary, and another 40 percent have shortages within parts of their boundaries.

4 Guiding principles for ethical, transparent, reproducible statistical product development and dissemination.

Communication

We communicated results throughout the Demonstration Use Case research with our Census CDE Working Group (composed of former Census Bureau Directors and Communication Director, and academic and industry census experts), with the Census Bureau, at conferences such as the annual Federal Statistical Committee on Methodology, and sharing drafts to seek input and ideas. The discussions and presentations helped to shape ideas and advance our thinking about how best to address the purpose and use questions.

Stakeholder engagement

We engaged stakeholders by sharing our research and results through conference presentations at the American Community Survey Data Users Conference and the Applied Public Data Users Conference. We also shared this demonstration project at Listening Sessions with stakeholders as an example of statistical product development. The Listening Sessions bring together 7 to 12 stakeholders by topic (e.g., children’s health) or function (e.g., state demographers) to seek their ideas for new statistical products.

Equity and ethics

As described in the Introduction, there are ethics and equity issues that drew us to develop this Use Case. Here we focus on equity and ethics vis-a-vis the data choices and analyses. With regard to ethical considerations with our data discovery process, fitness-for-purpose evaluation, and analyses, two questions arose:

1. What role does synthetic data have to play, and how do you benchmark it to evaluate fitness-for-purpose?

2. How do you construct and evaluate an index with the goal of identifying vulnerable populations?

Realizing the importance of nursing staff levels, we discussed and questioned whether the synthetic data had biases and were not representative of SNF residents and employees. We benchmarked the synthetic SNF nursing staff numbers against those submitted quarterly to CMS and observed they were biased low, so we decided to use the CMS data. These data were used to estimate the average number of nursing staff that could reach the facility during an extreme flood event (Figure 2).

In this use case, we were fortunate to have the “truth” to benchmark the synthetic data for the average daily nursing staff at each SNF. But this was not the case for the home locations of the nursing staff, therefore, the synthetic locations were not used since we had no way to benchmark them. Ideally, we would use the actual addresses of SNF employees. Instead, we used a simulation to estimate the average risks over routes leading to the SNF. This approach could be replaced with (or benchmarked against) the Census commuting data sets (eg, Commuting Flows or the LEHD Origin-Destination Employment Statistics) and the home census tract used as the starting point for each worker. For the number of nursing staff and their home locations, it is impossible to identify potential biases that would result in the inequitable allocation of emergency rescue resources without a thorough understanding of how the synthetic data were generated.

How one evaluates the equity of an index is a more challenging task. Questions that need to be addressed include:

1. How do you select the variables used to construct an indicator to guide an equitable allocation of technical assistance?

2. What relationship between these variables is important?

3. What are the differences across the numerous publicly available resilience estimators? Do some lead to a more equitable allocation of technical assistance in the event of an extreme clime event?

4. How do you validate a resilience estimator?

The technical document Community Resilience Indicator Analysis: County-Level Analysis of Commonly Used Indicators from Peer-Reviewed Research (Edgemon et al. 2018) identified the 20 most commonly selected variables for constructing resilience estimators from peer-reviewed research. Future research will need to validate these indices against past extreme climate events.

Privacy and confidentiality

We did not do a full disclosure review. However, some data are proprietary, and we could not release those data. We discuss how we used these data.

Dissemination

We disseminated the final version of the use case in the University of Virginia Libra Open repository (Lancaster et al. 2023).

Curation

Curation involves documenting all steps of the process so that they can be repeated, validated, reused, or extended. The final report explains the process in words. Curation must also provide the data, metadata, source code, and products. This led us to construct a GitHub repository. A README file guides the reader through the material and provides instructions for replicating the research results. Note that the README file must be downloaded for the hyperlinks to work.

5 Using the SNF statistical product

This potential statistical product has many uses. Federal policymakers and administrators regulate SNFs; however, they only sometimes realize the impacts on costs and the need for increased resources to meet these regulations. For example, by reviewing the aggregate inspection deficiency metrics, policymakers can target resources where they are most needed. Providing additional funding to pay workers more, improve their facilities, and address inspection deficiencies would improve the quality of SNFs. 

The media and advocacy groups play a role in highlighting good and bad cases of SNF care or where communities do not have adequate assets to support SNFs during an emergency event. For example, a New Yorker article (Rafiei 2022) highlighted how nursing homes decline dramatically when bought by private equity owners. The GAO (September 22, 2023) recently identified the need for more information about private equity ownership in CMS data – a gap that CMS needs to address. And, of course, researchers and analysts are essential for conducting research that leads to creating and improving statistical products around SNFs. By releasing a regularly scheduled SNF statistical product, the changes in SNFs over time can be monitored.

6 What CDE capabilities have this use case demonstrated?

As demonstrated by this use case, the CDE Framework is a powerful process for guiding and curating the development of statistics to address complex purposes and uses. Additionally, use cases help illuminate technical capabilities that should be present in the data enterprise to facilitate and accelerate the reuse of data and methods in the development and dissemination of new statistical products.

This CDE demonstration is the first of many use cases needed to define and develop CDE capabilities. Underlying each use case is the curation process. Curation documents each step, including decisions that may involve trade-offs. Curation preserves and adds value to the data. This includes organizing to facilitate data discovery and easy access; providing metadata to enable the reuse in scientific and programmatic research; enhancing the value of the data enterprise through linkages between datasets; and mapping the network of interconnections between datasets, research outputs, researchers, and institutions. Over time, a searchable curation system will be needed as a foundation for creating statistical products in the CDE.

The types of products from a use case that can benefit the larger community are only limited by the creativity of the researchers and stakeholders carrying out the use case. The products from this use case are re-useable code; integrated data sets across diverse topics for each SNF; maps and other visualizations; statistical products such as SNF deficiency indices and various indices that measure community and SNF resilience; the probability of a worker reaching an SNF in the event of extreme flooding; and a GitHub repo that provides easy access to all these products plus relevant metadata, literature, and government documents and regulations.

Conducting this use case has been an eye-opening experience as to the amount and quality of publicly available data to address our research questions. The statistical capabilities and products flowing from diverse use cases can only be identified as the program progresses.

Introduction

Today, official statistics – tables, reports and microdata – are produced using data from a single survey. These surveys are foundational for researchers and policymakers. However, many issues cannot be answered by surveys alone. For example, creating a picture of how prepared skilled nursing facilities (SNFs) are for climate emergencies requires wrangling all types of data about the facilities and their communities.(Note: A skilled nursing facility is a facility that meets specific federal regulatory certification requirements that enable it to provide short-term inpatient care and services to patients who require medical, nursing, or rehabilitative services.) This includes SNF data on the number and dates of inspections, deficiencies, residents’ mental and physical health, the number of nursing staff and where they live, community assets data on the number of shelter facilities, health professionals and emergency service providers, and community risks data on the probability of an extreme climate event. How can we create new statistical products useful to policymakers, emergency responders, skilled nursing facility staff, and others to inform their decisions?

With the explosion of available data, there is an opportunity to combine all types of information to create statistical products that address cross-cutting topics for a wide range of purposes and uses. The US Census Bureau is modernizing and transforming its enterprise system to accommodate a new way to produce statistical products that take advantage of all data types: designed surveys and censuses, public and private administrative data, opportunity data scraped from the internet, and procedural data (Keller et al. 2022).

‘We are moving towards a single enterprise, data-centric operation that enables us to funnel data from many sources in a single data lake using common collection and ingestion platforms… This is the essence of a curated data approach — assemble, assess, and fill in the gaps to create quality statistical data.’

Robert Santos, Director, US Census Bureau

This curated approach is embodied in the Curated Data Enterprise (CDE). The Curated Data Enterprise Framework in Figure 1 provides a guide for creating statistical products that enable the full integration of data from many sources (Keller et al. 2020). At the heart of the framework are the purposes and uses that provide the context and driving force for developing the statistical product. The outer rectangle in Figure 1 identifies the guiding principles for ethical, transparent and reproducible product development and dissemination. The inner rectangle identifies the steps in the statistical product development, including integrating primary and secondary data sources. The arrows convey that this process may only sometimes be linear. Instead, the process is iterative, where new information may be discovered at any point, requiring reevaluating and updating prior steps. Our Social and Decision Analytics research group in the Biocomplexity Institute developed, tested, and refined the CDE (data science) Framework in our research since 2013 (Keller et al. 2017, 2020). The proposed use of the CDE to develop statistical products at the US Census Bureau is in its early stages.

The next article in this series will put the CDE Framework into practice by demonstrating the use case on skilled nursing facilities’ preparedness for emergencies during extreme climate events. As a prelude to that article, we have created a visual for the statistical product development component of how that process works in action in Figure 2.

The CDE Framework’s guiding principles and research steps are described below. To find out more click on a cross reference.

Guiding principles:

• Purposes and uses
• Stakeholders
• Curation
• Equity and ethics
• Privacy and confidentiality
• Communications and dissemination

Research steps:

• Subject matter input
• Data discovery
• Data ingestion & Governance
• Data wrangling
• Fitness-for-purpose
• Statistics development

Guiding principles

Research steps

Introduction

Two centuries ago, when the Framers of the US Constitution laid the cornerstone for the federal statistical system, they could not have imagined the complexity of questions future generations would want to ask or the variety of data sources available to address them. Back in 1787, counting the population and apportioning state seats in the House of Representatives were the most urgent tasks before the young nation, and so a requirement for a decennial census was written into the Constitution. Now, 233 years later, the census continues to serve its original purpose – but purposes and uses for census data have exploded.

Questions we now seek to answer go beyond what the census (or surveys) alone can hope to address. Even with the multitude of other surveys commissioned by today’s US Census Bureau, researchers and policymakers find themselves looking to novel sources of data – from structured numeric data in traditional databases to unstructured text documents scraped from the internet – to explore issues such as understanding how prepared nursing homes and communities are for extreme climate events,eg, hurricanes, wildfires, or floods. Wrangling these sources with traditionally designed data, such as censuses and surveys, can fill data gaps, improve the quality and usefulness of statistical products, speed up their dissemination, and inspire the creation of new types of statistical products.

That is the impetus for developing the Curated Data Enterprise (CDE), an innovation in data science aimed at creating statistical products from all data types and building the infrastructure to support them. The Curated Data Enterprise, as the name implies, includes an end-to-end curation model to capture the complete statistical product development process. The CDE is designed to enable data discovery and retrieval, data quality assessment across multiple and diverse sources of information, and the reuse of data and models over time to accelerate statistical product development. The US Census Bureau has partnered with the University of Virginia, a working group of former Census Bureau Directors, a Communication Director, and university, non-profit and industry experts to develop this approach.

A new approach

To realize the CDE vision, the development of statistical products will address stakeholder questions using all data types – designed surveys and censuses, public and private administrative data, opportunity data scraped from the internet and procedural data (Keller et al. 2022). This new approach aligns with the US Census Bureau’s modernization and transformation (Thieme 2022) while maintaining the fundamental responsibilities of statistical agencies (OMB 2023). It is also consistent with a conclusion by the NASEM Panel on the Implications of Using Multiple Data Sources for Major Survey Programs: ‘The quality of statistics produced from multiple data sources depends on properties of the individual sources as well as the methods used to combine them. A new framework of quality standards and guidelines is needed to evaluate such data sources’ fitness for use’ (NASEM 2023, 192).

The CDE approach provides such a framework to address many of the challenges that official statistics face today, as well as demonstrate that they are poised to adopt a new approach to producing official statistics. For example:

• The timeliness and frequency of our official statistics are insufficient when there are shocks to the economy, such as the Covid-19 pandemic, when retrospective survey data were of limited usefulness. Federal agencies responded during the pandemic with relevance and agility by creating and launching fast-response Household Pulse Surveys that met immediate needs for data, trading off timeliness for quality (Groshen 2021). Public engagement and support for these new relevant and timely data products at a time of crisis were essential to the success of this new statistical product.

• The policy environment has responded to technological, social, and survey changes by encouraging efficient use of existing data, reuse, sharing and furthering open data principles. Researchers are now creating innovative statistical products using multiple data sources to better address the US’s needs and interests. The Commission on Evidence-Based Policymaking (Abraham et al. 2018) and the Federal Data Strategy (“Federal Data Strategy, Leveraging Data as a Strategic Asset” 2021) recommendations encourage agencies to permit access to data to undertake evaluation and research studies.

• Techniques such as rapid scanning, text recognition, user-friendly uploads, and new devices, sensors, and systems can now record and transcribe data in real time. Using these techniques, governments and corporations now routinely and instantaneously collect and store data on behaviors and states as varied as purchase transactions, climate and road conditions, healthcare plan utilization, and land use and zoning. Extensive digitization and recording, better system connectedness and interactivity, and increased human-computer interaction can result in faster data accumulation, enhancing the usability of private and public administrative data while maintaining privacy and confidentiality (Brady 2019; Jarmin 2019).  

• New techniques and data sources can transform statistical agencies ‘from the 20th-century survey-centric model to a 21st-century model that blends structured survey data with administrative and unstructured alternative digital data sources’, leading to better measures of the gig economy, retail sales, healthcare, workforce, and tools and methods to integrate multiple data sources while maintaining privacy and confidentiality (Jarmin 2019).

The next three articles in this series will:

• provide an overview of the CDE and its corresponding framework

• put the CDE Framework into practice through a demonstration use case on the resilience of skilled nursing facilities

• describe our next steps for developing the CDE through a use case research program.

Background

Decisions around medium and long-term allocation of healthcare resources are fraught with challenges and uncertainties, which explains the use of blunt resource allocations based on across-the-board annual percentage uplifts.

The Bristol, North Somerset, South Gloucestershire Integrated Care Board (BNSSG ICB - we love elaborate acronyms in the National Health Service!), in the south west of England, is part of the local NHS apparatus responsible for planning the current and future health needs of the one million resident population.

Population Segmentation

Before tackling the complex problem of forecasting healthcare resources into the future, we first need to understand the current situation regarding the distribution of health needs.

While every individual has a unique set of circumstances, population segmentation is an approach used to help understand overall need by combining individuals into different groups, based on certain criteria.

We use the Cambridge Multimorbidity Score which is a metric designed to summarise the presence of multiple health conditions, known as multimorbidity. Using that score, which applies different weights to different health conditions, we previously found a way of splitting the adult (17+) population into five Core Segments, with Core Segment 1 patients having the lowest score and being the least ill and Core Segment 5 being those with the most multimorbidity.

Applied to the BNSSG adult population (of around 750K individuals), the following interesting properties were found:

1. Halving: Going up one segment results in roughly half the number of people in that segment
2. Doubling: Going up one segment results in roughly twice the NHS monetary spend per person per year

We can see this in Figure 2.

Creating The Model

To forecast health needs of the population, in terms of how many people will be in which Core Segment in what future year, the Dynamic Population Model (DPM) takes information from two different sources:

1. The Office for National Statistics projections for our area. From this, we get yearly projections for not just the total 17+ population, but also the predicted number of people turning 17 (and so entering our model), deaths, and in- and out-ward migration.

2. NHS patient attribute and activity data, stored in the System Wide Dataset (SWD). This gives us: past and current information on the adult population’s NHS healthcare usage; the Core Segment breakdown of our current and past populations; the proportion of those turning 17, migrating, and dying that are in each Core Segment. From this, we estimate the historical rates of transition within Core Segments, which is essentially the yearly number of people getting sicker or healthier.

By synthesising these pieces of data, we create our DPM forecast. Starting from the most up to date Core Segment population breakdown, the model takes yearly time steps into the future, at each time step using the inputs to estimate how many people are to be in each Core Segment. This modelling approach of having discrete time steps and different movements between states can be set up as a Markov chain, although here we have formulated it as a set of difference equations - through which the outflow of each Core Segment population at each time step is deterministic. The design was led by Zehra and Christos, through a collaboration between the NHS and the Centre for Healthcare Innovation and Improvement (CHI2) at the University of Bath.

The model can be thought of as having the following inputs:

From these inputs, it deterministically outputs the yearly forecasts for the number of people in each Core Segment. From these yearly Core Segment population figures, we can also forecast use by point of delivery by taking historic SWD information on the activity used by current Core Segment breakdown, under the assumption that stays the same into the future.

We combine these population health segment projections – i.e., how many people will be in which Core Segment in what future year – with recent NHS healthcare usage data to yield forecasted changes for various delivery points, like Emergency Department (ED) visits or maternity service appointments.

Findings

The first output of the model is the population forecast for each Core Segment, as plotted in Figure 3. The visualisation is a type of sankey diagram called an alluvial plot, which shows the proportion of people moving between the Core Segments each year. As it is to be expected, the majority of individuals stay in the same Core Segment year-on-year as the process of acquiring conditions and developing multimorbidity takes places over many years and decades.

The concerning insight shown in Figure 3 is that all Core Segments apart from (the most healthy) Core Segment 1 are due to increase in size, with Core Segment 5 having the largest percentage increase over the next 20 years. While, at first glance, this could be attributed to the effect an ageing population, in which people are staying alive for longer we will see in the next set of results that this itself does not wholly explain the forecasted Core Segment changes.

In applying the typical NHS healthcare usage per Core Segment to the projections of Figure 3, we derive the expected future healthcare usage for various healthcare settings (Figure 4). In overlaying to these the equivalent projections due solely to demographic factors (both for total population size and capturing the effect of Age and Sex), we see that the DPM projections for increased resource use are not solely attributable to an ageing and growing population, but also to a population becoming gradually less healthy over time.

Specifically, from Figure 4 we can glean the following insights:

1. In all areas except Maternity, the DPM forecasts an increased use beyond just the growing, aging population. The reason that Maternity can be explained as the exception is due to it closely following the demographic changes forecast, specifically for numbers of women of child bearing age.

2. For Community contacts, with the highest proportion of use from Core Segment 5 patients, the DPM forecasts the highest increase into the future. This is because, relative to current size, the number of Core Segment 5 patients is set to increase the largest and so that has the largest impact on Community contacts, which include home visits to patients to support rehabilitation and services to manage long-term mobility issues such as physiotherapy.

3. Whilst Secondary Elective and Non-Elective activity is forecast to grow at similar rates, the Carbon and Cost values are forecast to grow more for Secondary Non-Elective due to the average Carbon and Cost usage per person in Core Segment 5 being higher. In this context ‘Secondary’ is a hospital stay, with ‘Elective’ being planned and ‘Non-Elective’ being unplanned. For example, a hip replacement is elective whereas an admission following a road traffic accident is non-elective.

Limitations

It’s difficult to make predictions, especially about the future.

– Danish Proverb

As with any modelling / forecasting method, there are limitations to be mindful of.

1. The cost and activity usage estimates are made under the assumption that we will continue to deliver services as they are currently being delivered. We know this isn’t going to be true, as healthcare-seeking behaviour evolves over time, with younger people accessing healthcare in different ways to previous generations. On top of that, healthcare advances can result in significant changes in healthcare provision, in ways unaccounted for within this model.

2. The model is tied to ONS forecasts for population change, and robust forecasting is hard. It is difficult to estimate what the population will look like in 20 years’ time, and the influence of uncertain and unknown future local development and housing plans. Having said this, population forecasts tend to be robust, one way to consider this is that everyone who will be an adult by the end of the forecast in 20 years’ time has already been born.

3. The DPM does not explicitly account for the interaction of demand and capacity: it simply predicts future healthcare resource requirement assuming that health needs of a given Core Segment patient are met in the same way they are met now. This is an essential assumption to help ensure legitimate use of the empirically derived Core Segment transition rates. However, it inevitably limits practical use, as flexing demand and capacity assumptions is of importance to planners and service managers.

4. It is not possible to validate the model on historic data, firstly because of point 3. above but also because we only have good quality SWD information for the past two years, so cannot reliably look further back into the past and create a forecast that we can check against what actually happened.

5. Whilst it is possible to use the model in other healthcare systems and geographic areas, the underlying data required to generate the Core Segments is non-trivial, so significant data pipelining may be required to get to create local model inputs, as explained above in Section 3.

What Next

We have already generated local use cases for the DPM in forecasting different geographical areas or specific hospital trusts. We envisage the DPM becoming a standard tool in most forward planning initiatives and will continue to refine the model as more information becomes available both for calibration and validation.

Outside of BNSSG, we are keen to disseminate our modelling approach to others who may be interested, as well as expanding our collaboration. There are also other innovative approaches in this space, such as the Health in 2040 report by the Health Foundation which looks at England-level and uses the same ONS forecasts, but using a different ‘micro simulation’ modelling approach.

If long-term forecasting in the NHS is of interest to you and your work, we’d love to chat! Please get in touch at bnssg.analytics@nhs.net

Summary

Reliably forecasting longer-term population health needs and healthcare resource requirements is essential if the NHS is to effectively plan for tomorrow’s problems today.

While this is undoubtedly a difficult problem – both conceptually and statistically – our modelling, undertaken through an academic-NHS collaboration, demonstrates that there are alternatives beyond the commonly-used but simplistic approaches based only on demographic factors.

The problem

Data duplication is a ubiquitous problem affecting data quality. Organisations often have multiple records that refer to the same entity but no unique identifier that ties these entities together. Data entry errors and other issues mean that variations usually exist, so the records belonging to a single entity aren’t necessarily identical.

For example, in a company, customer data may have been entered multiple times in multiple different databases, with different spellings of names, different addresses, and other typos. The inability to identify which records belong to each customer presents a data quality problem at all stages of data analysis – from basic questions such as counting the number of unique customers, through to advanced statistical analysis.

With the growing size of datasets held by many organisations, any solution must be able to work on very large datasets of tens of millions of records or more.

Approach

In collaboration with academic experts, the team started with desk research into data linking theory and practice, and a review of existing open source software implementations.

One of the most common theoretical approaches described in the literature is the Fellegi-Sunter model. This statistical model has a long history of application for high profile, important record linking tasks such as in the US Census Bureau and the UK Office for National Statistics (ONS).

The model takes pairwise comparisons of records as an input, and outputs a match score between 0 and 1, which (loosely) can be interpreted as the probability of the two records being a match. Since the record comparison can be either two records from the same dataset, or records from different datasets, this is applicable to both deduplication and linkage problems.

An important benefit of the model is explainability. The model uses a number of parameters, each of which has an intuitive explanation that can be understood by a non-technical audience. The relative simplicity of the model also means it is easier to understand and explain how biases in linkage may occur, such as varying levels of accuracy for different ethnic groups.

Implementation

Through our desk research and open source software review, an existing software package called fastLink was identified which implements the Fellegi-Sunter model, but unfortunately the software is not able to handle very large datasets of more than a few hundred thousand records.

Inspired by the popularity of fastLink, the team quickly realised that the methodology it was developing was generally applicable and could be valuable to a wide range of users if published as a software package.

As we spoke to colleagues across government and beyond, we found record linkage and deduplication problems are pervasive, and crop up in many different guises, meaning that any software needed to be very general and flexible.

The result is Splink – which is a Python package that implements the Fellegi-Sunter model, and enables parameters to be estimated using the Expectation Maximisation algorithm.

The package is free to use, and open source. It is accompanied by detailed documentation, including a tutorial and a set of examples.

Splink makes no assumptions about the type of entity being linked, so it is very flexible. We are aware of its use to match data on a variety of entity types including persons, companies, financial transactions and court cases.

The package closely follows the statistical approach described in fastLink. In particular it implements the same mathematical model and likelihood functions described in the fastLink paper (see pages 354 to 357), with a comprehensive suite of tests to ensure correctness of the implementation.

In addition, Splink introduces a number of innovations:

• Able to work at massive scale – with proven examples of its use on over 100 million records.
• Extremely fast – capable of linking 1 million records on a laptop in around a minute.
• Comprehensive graphical output showing parameter estimates and iteration history make it easier to understand the model and diagnose statistical issues.
• A waterfall chart which can be generated for any record pair, which explains how the estimated match probability is derived.
• Support for deduplication, linking, and a combination of both, including support for deduplicating and linking multiple datasets.
• Greater customisability of record comparisons, including the ability to specify custom, user defined comparison functions.
• Term frequency adjustments on any number of columns.
• It’s possible to save a model once it’s been estimated – enabling a model to be estimated, quality assured, and then reused as new data becomes available.
• A companion website provides a complete description of the various configuration options, and examples of how to achieve different linking objectives.

Using Splink

Full documentation and a tutorial are available for Splink, but the following snippet gives a simple example of Splink in action:

from splink.datasets import splink_datasets
from splink.duckdb.blocking_rule_library import block_on
from splink.duckdb.comparison_library import (
    exact_match,
    jaro_winkler_at_thresholds,
    levenshtein_at_thresholds,
)
from splink.duckdb.linker import DuckDBLinker

df = splink_datasets.fake_1000

# Specify a data linkage model
settings = {
    "link_type": "dedupe_only",
    "blocking_rules_to_generate_predictions": [
      block_on("first_name"),
      block_on("surname"),
    ],
    "comparisons": [
        jaro_winkler_at_thresholds("first_name", 2),
        jaro_winkler_at_thresholds("surname"),
        levenshtein_at_thresholds("dob"),
        exact_match("city", term_frequency_adjustments=True),
        exact_match("email"),
    ],
}

linker = DuckDBLinker(df, settings)

# Estimate model parameters

# Direct estimation using random sampling can be used for the u probabilities
linker.estimate_u_using_random_sampling(target_rows=1e6)

# Expectation maximisation is used to train the m values
br_training = block_on(["first_name", "surname"])
linker.estimate_parameters_using_expectation_maximisation(br_training)

br_training = block_on("dob")
linker.estimate_parameters_using_expectation_maximisation(br_training)

# Use the model to compute pairwise match scores
pairwise_predictions = linker.predict()

# Cluster the match scores into groups to produce a synthetic unique person id
clusters = linker.cluster_pairwise_predictions_at_threshold(
  pairwise_predictions, 0.95
)
clusters.as_pandas_dataframe(limit=5)

The example shows the flexibility of Splink, and how various types of configuration can be used:

• How should different data fields be compared? In this example, the Jaro-Winkler distance is used for names, whereas Levenshtein is used for date of birth since Jaro-Winkler is not appropriate for numeric data.
• What blocking rules should be used? Blocking rules are the primary determinants of how fast Splink will run, but there is a trade off between speed and accuracy. In this case, the input data is small, so the blocking rules are loose.
• How should the model parameters be estimated? In this case, the user has no labels for supervised training, and so uses the unsupervised Expectation Maximisation approach.
• Is clustering needed? In this case, each person may potentially have many duplicates, so clustering is used. This creates an estimated (synthetic) unique identifier for each entity (person) in the input dataset.

Outcomes

Splink has been used to link some of the largest datasets held by the Ministry of Justice as part of the Data First programme, and researchers are now able to apply for secure access to these datasets. Research using this data won the ONS Linked Administrative Data Award at the 2022 Research Excellence Awards.

More widely, the demand for Splink has been higher than we expected – with over 7 million downloads. It has been used in other government departments including the Office for National Statistics and internationally, the private sector, and published academic research from top international universities.

Splink has also had external contributions from over 30 people, including staff at the Australian Bureau of Statistics, DataBricks, other government departments, academics, and various private sector consultancies.

Background

From the outset of the rollout of body-worn cameras, TJJD faced a major issue with implementation: in 2019, body worn cameras were an established tool for law enforcement, but there was very little literature or best practice to draw from for their use in a correctional environment. Unlike police officers, juvenile correctional officers (JCOs) deal directly with their charges for virtually their entire shift. In an eight-hour shift, a police officer might record a few calls and traffic stops. A juvenile correctional officer, on the other hand, would record for almost eight consecutive hours. And, because TJJD recorded round-the-clock for hundreds of employees at a time, this added up very quickly to a lot of footage.

For example, a typical dorm in a correctional center might have four JCOs assigned to it. Across a single week, these four JCOs would be expected to record at least 160 hours of footage.

This was replicated across every dorm. Three dorms, for example, would produce nearly 500 hours of footage, as seen below.

Finally, we had more than one facility. Four facilities with three dorms each would produce nearly 2,000 hours of footage every week.

In actuality, we had a total of five facilities each with over a dozen dorms producing an anticipated 17,000 hours of footage every week – an impossible amount to monitor manually.

As a result, footage review had to be done in a limited, reactive manner. If our monitoring team received an incident report, they could easily zero in on the cameras of the officers involved and review the incident accordingly. But our executive team had hoped to be able to use the footage proactively, looking for “red flags” in order to prevent potential abuses instead of only responding to allegations.

Because the agency had no way of automating the monitoring of footage, any proactive analysis had to be metadata-based. But what to look for in the metadata? Once again, the lack of best-practice literature left us in the lurch. So, we brainstormed ideas for “red flags” and came up with the following that could be screened for using camera metadata:

1. Minimal quantity of footage – our camera policy required correctional officers to have their cameras on at all times in the presence of youth. No footage meant they weren’t using their cameras.

2. Frequently turning the camera on and off – a correctional officer working a dorm should have their cameras always on when around youth and not be turning them on and off repeatedly.

3. Large gaps between clips – it defeats the purpose of having cameras if they’re not turned on.

In addition, we came up with a fourth red flag, which could be screened for by comparing camera metadata with shift-tracking metadata:

4. Mismatch between clips recorded and shifts worked – the agency had very recently rolled out a new shift tracking software. We should expect to see the hours logged by the body cameras roughly match the shift hours worked.

Analysis, part 1: Quality control and footage analysis

For this analysis, I gathered the most recent three weeks of body-worn camera data – which, at the time, covered April 1–21, 2019. I also pulled data from Shifthound (our shift management software) covering the same time period. Finally, I gathered HR data from CAPPS, the system that most of the State of Texas used at the time for personnel management and finance.¹ I then performed some quality control work, summarized in the dropdown box below.

In order to operationalize the “red flags” from our brainstorming session, I needed to see what exactly the cameras captured in their metadata. The variables most relevant to our purposes were:

• Clip start
• Clip end
• Camera used
• Who was assigned to the camera at the time
• The role of the person assigned to the camera

Using these fields, I first created the following aggregations per employee ID:

I used these aggregations to devise the following staff metrics:

Once I established these metrics for each employee I looked at their respective distributions. Standard staff shift lengths at the time were eight hours. If staff were using their cameras appropriately, we would expect to see distributions centered around clip lengths of about an hour, eight or fewer clips per day, and 8-12 footage hours per day. We would also expect to see 0 large gaps.

```{r}
library(tidyverse)

Footage_Metrics_by_Employee <- read_csv("Output/Footage Metrics by Employee.csv")

Footage_Metrics_by_Employee %>% 
  select(-Clips, -Days_With_Footage, -Footage_Hours, -Gaps) %>% 
  pivot_longer(-Employee_ID, names_to = "Metric", values_to = "Value") %>% 
  ggplot(aes(x = Value)) +
  geom_histogram() +
  facet_wrap(~Metric, scales = "free")
```

By eyeballing the distributions I could tell most staff were recording fewer than 10 clips per day, shooting about 0.5–2 hours for each clip, for a total of 2–10 hours of daily footage, with the majority of employees having less than one significant gap per day. Superficially, this appeared to provide evidence of widespread attempts at complying with the body-worn camera policy and no systemic rejection or resistance. If this were indeed the case, then we could turn our attention to individual outliers.

First, though, we thought we would attempt to validate this initial impression by testing another assumption. If each employee works on average 40 hours per week – a substantial underestimate given how common overtime was – we should expect, over a three-week period, to see about 120 hours of footage per employee in the dataset. This is not what we found.

Average footage per employee was 70.2 hours over the three-week period, meaning that the average employee was recording less than 60% of shift hours worked. With so many hours going unrecorded for unknown reasons, we needed to investigate further.

Surely the shift data would clarify this…

Analysis, part 2: Footage and shift comparison

With the data on shifts worked from our timekeeping system, I could theoretically compare actual shifts worked to the amount of footage recorded. If there were patterns in where the gaps in footage fell, that comparison might help to explain why.

In order to join the shift data to the camera data, I needed a common unit of analysis beyond “Employee ID.” Using only this value would produce a nonsensical table that joined up every clip of footage to every shift worked.

For example, let’s take employee #9001005 at Facility Epsilon between April 1–3. This employee has the following clips recorded during that time period:

We can join this to a similar table of shifts logged. This particular employee had the following shifts scheduled from April 1–3:

The table shows two eight-hour morning shifts from 6:00 am to 2:00 pm. We can join the two tables together by ID on a messy many-to-many join, but that tells us nothing about how much they overlap (or fail to overlap) without extensive additional work. For example, we have a unique identifier for employee clip (Clip_ID) and employee shift (Shift_ID), but what we need is a unique identifier that can be used to join the two. Fortunately, for this particular data we can create a unique identifier since both clips and shifts are fundamentally measures of time. While Employee_ID is not in itself unique (i.e., one employee can have multiple clips attached to that ID), Employee_ID combined with time of day is unique. A person can only be in one place at a time, after all!

To reshape the data for joining, I created a function that takes any data frame with a start and end column and unfolds it into discrete units of time. Using the code below to create the “Interval_Convert” function, the shift data above for employee 9001005 converts into one entry per hour of the day per shift. As a result, two eight-hour shifts get turned into 16 employee hours (a sample of which is shown below).

```{r}
library(sqldf)
library(lubridate)

Interval_Convert <- function(DF, Start_Col, End_Col, Int_Unit, Int_Length = 1) {
browser()
  Start_Col2 <- enquo(Start_Col)
  End_Col2 <- enquo(End_Col)
  
  Start_End <- DF %>%
    ungroup() %>%
    summarize(Min_Start = min(!!Start_Col2),
              Max_End = max(!!End_Col2)) %>%
    mutate(Start = floor_date(Min_Start, Int_Unit),
           End = ceiling_date(Max_End, Int_Unit))
  
  DF <- DF %>%
    mutate(Single = !!Start_Col2 == !!End_Col2)
  
  Interval_Table <- data.frame(Interval_Start = seq.POSIXt(Start_End$Start[1], Start_End$End[1], by = str_c(Int_Length, " ", Int_Unit))) %>%
    mutate(Interval_End = lead(Interval_Start)) %>%
    filter(!is.na(Interval_End))
  
  by <- join_by(Interval_Start <= !!End_Col2, Interval_End >= !!Start_Col2)  
  
  Interval_Data_Table <- Interval_Table %>% 
    left_join(DF, by) %>% 
    mutate(Seconds_Duration_Within_Interval = if_else(!!End_Col2 > Interval_End, Interval_End, !!End_Col2) -
             if_else(!!Start_Col2 < Interval_Start, Interval_Start, !!Start_Col2)) %>%
    filter(!(Single & Interval_End == !!Start_Col2),
           as.numeric(Seconds_Duration_Within_Interval) > 0)
  
  return(Interval_Data_Table)
}
```

The footage could be converted in a similar manner, and in this way I could break down both the shift data and the clip data into an hour-by-hour view and compare them to one another. Using this new format, I joined together the full tables of footage and shifts to determine how much footage was recorded with no corresponding shift in the timekeeping system.

To summarize what the table is telling us: Almost every employee has footage hours that do not match with logged shifts, totaling nearly 14,000 hours when you add up the Footage_Hours_No_Shift column. But what about the opposite case? How many shift hours were logged with no corresponding footage?

Oh dear. Again, almost every employee has logged shift hours with no footage: 47,000 hours in total. To put it another way, that’s an entire work week per employee not showing up in camera footage.

At this point, we could probably rule out deliberate noncompliance. The clip data already implied that most employees were following the policy, and our facility leadership would surely have noticed a mass refusal large enough to show up this clearly in the data.

One way to check for deliberate noncompliance would be to first exclude shifts that contain zero footage whatsoever. This would rule out total mismatches, where – for whatever reason – the logged shifts had totally failed to overlap with recorded clips. For the remaining shifts that do contain footage, we could look at the proportion of the shift covered by footage. So, if an eight-hour shift had four hours of recorded footage associated with it, then we could say that 50% of the shift had been recorded. The following histogram is a distribution of the number of employees organized by the percent of their shift-hours they recorded (but only shifts that had a nonzero amount of footage).

As it turned out, most employees recorded the majority of their matching shifts, a finding that roughly aligns with the initial clip analysis. So, what explains the 14,000 hours of footage with no shifts, and the 47,000 hours of shifts with no footage?

Causes of failure

Here, I believed, we had reached the end of what I could do with data alone, and so I presented these findings (or lack thereof) to executive leadership. The failure to gather reliable data from linking the clip data to the shift data prompted follow-ups into what exactly was going wrong. As it turned out, many things were going wrong.

First, a number of technical problems plagued the early rollout of the cameras:

• All of our facilities suffered from high turnover, and camera ownership was not consistently updated. Employees who no longer worked at the agency could therefore appear in the clip data – somebody else had taken over their camera but had not put their name and ID on it.

• We had no way of telling if a camera was not recording due to being docked and recharging or not recording due to being switched off.

• In the early days of the rollout, footage got assigned to an employee based on the owner of the dock, not the camera. In other words, if Employee A had recorded their shift with their camera but uploaded the footage using a dock assigned to Employee B then the footage would show up in the system as belonging to Employee B.

The shift data was, unsurprisingly, even worse, and it was here we came across our most important finding. While the evidence showed that there wasn’t any widespread non-compliance with the use of the cameras, there was widespread non-compliance with the use of our shift management software. Details are included in the dropdown box below.

What we learned from failure

Whatever means we used to monitor compliance with the camera policy, we learned that it couldn’t be fully automated. The agency followed up this analysis with a random sampling approach, in which monitors would randomly select times of day they knew a given staff member would have to have their cameras turned on and actually watch the associated clips. This review process confirmed the first impressions from the statistical review above: most employees were making good faith attempts at complying with the policy despite technical glitches, short-staffing, and administrative confusion. It also confirmed that proactive monitoring of correctional officers was a human process which had to come from supervisors and staff.

The one piece of the analysis we did use going forward was the clip analysis (converted into a Power BI dashboard and included in the GitHub repository for this article), but only as a supplement for already-launched investigations, not a prompt for one. Body-worn camera footage remained immensely useful for investigations after-the-fact, but inconsistencies in clip data were not, in and of themselves, particularly noteworthy “red flags.” At the end of the day, analytics can contextualize and enhance human judgment, but it cannot replace it.

In academia, the bias in favor of positive findings is well-documented. The failure to find something, or a lack of statistical significance, does not lend itself to publication in the same way that a novel discovery does. But, in an applied setting, where results matter more than publication criteria, negative findings can be highly insightful. They can falsify erroneous assumptions, bring unknown problems to light, and prompt the creation of new processes and tools. In this context, a failure is only truly a failure if nothing is learned from it.

Our perspective on the challenge

The goal of this challenge is to use machine learning and natural language processing (NLP) to link language-based entries in the IRI and FNDDS databases. Our proposed approach is based on our prior work using deep learning models to map users’ natural language meal descriptions to the FNDDS database (Korpusik et al. 2017b) to retrieve nutrition information in a spoken diet tracking system. In the past, we found a trade-off between accuracy and cost, leading us to select convolutional neural networks over recurrent long short-term memory (LSTM) networks – with nearly 10x as many parameters and 2x the training time required, LSTMs achieved slightly lower performance on semantic tagging and food database mapping on meals in the breakfast category. Here, we propose to investigate state-of-the-art transformers, specifically the contextual embedding model (i.e., the entire sentence is used as context to generate the embedding) known as BERT (Bidirectional Encoder Representations from Transformers, Devlin et al. 2018).

Our approach

Our approach is to fine-tune a large pre-trained BERT language model on the food data. BERT was originally trained on a massive amount of text for a language modelling task (i.e., predicting which word should come next in a sentence). It relies on a transformer model, which uses an “attention” mechanism to identify which words the model should pay the most “attention” to. We are specifically using BERT for binary sequence classification, which refers to predicting a label (i.e., classification) for a sequence of words. In our case, during fine-tuning (i.e., training the model further on our own dataset) we will feed the model pairs of sentences (where one sentence is the UPC description of a food item and the other is the EC description of another food item), and the model will perform binary classification, predicting whether the sentences are a match (i.e., 1) or not (i.e., 0). We start with the 2015–2016 ground truth PPC data for positive examples, and five randomly sampled negative examples per positive example.

Our results

After training, the 48K model (so-called because it was trained on 48,000 data samples) was used at test time via ranking all possible 2017–18 EC descriptions given an unseen UPC description. The rankings were obtained through the model’s output value – the higher the output (or confidence), the more highly we ranked that EC description. To speed up the ranking process, we used blocking (i.e., only ranking a subset of all possible matches), specifically with exact word matches (using only the first six words in the UPC description, which appeared to be the most important), and fed all possible matches through the model in one batch per UPC description. Since we still did not have sufficient time to complete evaluation on the full set of test UPC descriptions, we implemented an expedited evaluation that only considered the first 10 matching EC descriptions in the BERT ranking process (which we call BERT-FAST). We also report results for the slower evaluation method that considers all EC descriptions that match at least one of the first six words in a given UPC description, but note that these results are based on just a small subset of the total test set. See Table 1 below for our results, where the (5?) indicates how often the correct match was ranked among the top-5. See Table 2 for an estimate of how long it takes to train and test the model on a CPU.

Future work/refinement

In the future, with more time available, we would train on all data, not just our limited dataset of 48,000 pairs, as well as perform evaluation on the held-out test set with the full set of possible EC matches that have one or more words in common with the UPC description. We would compare against baseline word embedding methods such as word2vec (Mikolov et al. 2017) and Glove (Pennington et al. 2014), and we would explore hierarchical prediction methods for improving efficiency and accuracy. Specifically, we would first train a classifier to predict the generic food category, and then train finer-grained models to predict specific foods within a general food category. Finally, we are exploring multi-modal transformer-based approaches that allow two input modalities (i.e., food images and text descriptions of a meal) for predicting the best UPC match.

Lessons learned

We recommend that future challenges provide every team with both a CPU and a GPU in their workspace, to avoid transitioning from one to the other midway through the challenge. In addition, if possible, it would be very helpful to provide a mechanism for running jobs in the background. Finally, it may be useful for teams to submit snippets of code along with library package names, in order for the installations to be tested properly beforehand.

Perspective on the challenge

Text matching is an essential task in natural language processing (NLP, Pang et al. 2016), while record linkage across different sources is an essential task in data science. Machine learning techniques allow people to combine data faster and cheaper than using manual linkage. However, in the context of the Food for Thought challenge, existing methods for matching universal product codes (UPCs) to ensemble codes (ECs) require every UPC to be compared with every EC code (Figure 1a). Such approaches can be computationally expensive in the training process when data is noisy. Here, we propose an ensemble model with a category-based adapter to tackle this problem, drawing on the category information included in UPC and EC data. The category-based adapter allows UPCs to be first matched with only a small and reliable set of ECs (Figure 1b). Then, an ensemble model will be deployed to make predictions for UPC-EC matching. Our proposed approach can achieve competitive performance compared with state-of-the-art models.

Our approach

We propose a two-step framework to address this problem. To begin with, we use a category-based adapter to get reliable candidate ECs for each UPC. Then, an ensemble model (Dietterich 2000) is deployed to make a prediction for each UPC-EC pair.

Our results

We randomly selected 500 samples from the 2017–2018 UPC-EC data to train the ensembled weight for each model. Two functions were adapted to make a fusion of base-string and BERT models:

denotes the final confidence score. and represent base_string_similarity_score and BERT_similarity_score, respectively. and are corresponding model weights for base_string and BERT models.

A better Success@5 is achieved with function (1). The ensembled weights for the base-string model and BERT model are 0.738 and 0.262, respectively. The experiment result indicates that the base_string model contributes more than the BERT model when the ensemble model makes predictions. The prediction result for the 2017–2018 data is:

• Success@5: 0.727
• NDCG@5: 0.528

Computation time is 6 hours.

Future work

Our next step will focus on adding the newly generated EC data into our knowledge base, which allows the model to be more stable to make predictions for UPC-EC matching. Our model is an unsupervised method, which does not need labels for each instance. We use cosine similarity to rank the matches, so no labels are needed in the training process. However, our future work will try to label some instances to handle the UPC-EC matching task in a supervised manner.

Lessons learned

1. If the data is not complex, simple models may outperform complex models. For example, in our experiment, we found that the base-string model outperforms single RoBERTa (Liu et al. 2019) or BERT models. However, our ensemble model can outperform each individual model since model fusion allows information aggregation from multiple models.

2. Multi-label models may not work well on UPC-EC data. In our early work, we tried to consider the UPC-EC matching task as a multi-label problem, e.g., we labeled each EC as a binary label which indicated whether the EC was an appropriate match or not. We mapped UPC and EC pairs into a multi-label table. However, we find that the UPC and EC keeps a one-to-one relation for most UPCs. The model performance of a multi-label model, i.e., Label-Specific Attention Network (LSAN, Xiao et al. 2019), is lower than base-string model on both Success@5 and NDCG@5 metrics.

Our perspective on the challenge

At the start of this competition, we decided to test three general approaches, in the order listed:

1. A heuristic approach, where we use only the data and a defined similarity metric to predict which FNDDS label a given IRI item should have.

2. A simpler modeling approach, where we train a simple statistical classifier, like a random forest (Parmar et al. 2019), logistic regression, etc., to predict the FNDDS label for a given IRI item. For this method, we opted to use a random forest as our statistical model, as it was a simpler model to use as a baseline, having shown decent performance in a wide range of classification tasks. As it turned out, this approach was quite robust and accurate, so we kept it as our main model for this approach.

3. A large language modeling approach, where we train a model like BERT (Devlin et al. 2018) to map the descriptions for given IRI and FNDDS items to the FNDDS category the supplied IRI item belongs to.

Our approach

As we explored the data provided, we opted to use the given 2017–2018 PPC dataset as our primary dataset for both training and testing. To ensure a fair evaluation of the model, we randomly split the dataset into 60% training samples and 40% testing samples, making sure our training process never sees the testing dataset. For evaluating our models, we adopted the competition’s metrics: Success@5 and NDCG@5. After months of testing, our statistical classifier (approach #2) proved itself to be the model that both processes the data fastest and achieves the highest performance on our testing metrics.

This approach, at a high level, takes in the provided data (among other configuration parameters), formats the data in a computer-readable format – converting the IRI and FNDDS descriptions to a numerical representation with word embeddings (2018; Mikolov et al. 2013; Pennington et al. 2014) and then using that numerical representation to calculate the distances between each description – and then trains a classification model (random forest (2019)/neural network (Schmidhuber 2015)) that can predict an FNDDS label for a given IRI item.

In terms of data, our approach uses the FNDDS/IRI descriptions, combining them into a single “description” field, and the IRI item’s categorical items – department, aisle, category, product, brand, manufacturer, and parent company – to further discern between items.

While most industrial methods require use of a graphics processing unit (graphics card, or GPU) to perform this kind of processing, our primary method only requires the computer’s internal processor (CPU) to function properly. With that in mind, to achieve the best possible performance on our test metrics, the most time-consuming operations are run in parallel. The time taken to train our primary model can likely be further improved if we parallelize these operations across a GPU, with the only downside being the imposition of a GPU requirement for systems aiming to run this method.

In addition to our primary method, our team has worked with alternate approaches on the GPU (using BERT (2018), neural networks (2015), etc.) to either: 1) speed up the time it takes to process and make inferences for the data, achieving similar performance on our test metrics, or 2) achieve higher performance, likely at a cost to the time it takes to process everything. Our reasoning behind doing so is that if a simple statistical model performs well, then a larger language model should be able to demonstrate a higher performance on our test metrics without much of an increase in training time. At the current time, these methods are still unable to match the performance/efficiency tradeoff of our primary method.

After exploring alternate methods to no avail, our team then decided to focus again on our primary method, the random forest (2019), and a secondary method, feed-forward neural network mapping our input features (X) to the FNDDS labels (Y) (2015), to optimize their training hyperparameters for the dataset. Our aim in this is to see which of our already-implemented, easier-to-run downstream methods would better optimize the performance/efficiency tradeoff after having its training parameters optimized to the fullest. This has resulted in a marginal increase in training time (+20-30 minutes) and a roughly 5% increase in performance for our still-highest performing model, the random forest.

Overall, our primary method – the random forest – gave us an approximate training time (including data pre-processing) of 4 hours 30 minutes for our ~38,000 IRI item training set, and an approximate inference time of 15 minutes on our testing set of ~15,000 IRI items. Furthermore, our method gave us a Success@5 score of .789 and an NDCG@5 score of .705 on our testing set.

Our results

Future work/refinement

As mentioned previously, we only used the given 2017–2018 PPC dataset as our primary dataset for both training and testing. Going forward, we would like to include datasets from previous years as well, which we believe would further increase our model performance. Additionally, the datasets generated from this research have the potential to inform and support additional studies from a variety of perspectives, including nutrition, consumer research, and public health. Further research utilizing these datasets has the potential to make significant contributions to our understanding of consumer behavior and the role of food and nutrient consumption in overall health and well-being.

Lessons learned

It was interesting that the random forest model performed better than the vanilla neural network model. This shows that a simple solution can work better, depending on the application. This observation is in line with the well-established principle in machine learning that the choice of model should be guided by the nature of the problem and the characteristics of the data. In this case, the random forest model, being a simpler and more interpretable model, was better suited to the problem at hand and was able to outperform the more complex neural network model. These results underscore the importance of careful model selection and the need to consider both the complexity of the model and the specific requirements of the problem when choosing an algorithm for a particular application.

Methods used to date

On the surface, the linking process may appear simple: both the FNDDS and retail food scanner data are databases of food. But the scanner data are produced for market research, and the FNDDS for dietary studies. The scanner data include about 350,000 items with sales each year, while the FNDDS has only 10,000–15,000 items. Scanner data relates to specific products, while FNDDS items are often more general. Both datasets have different hierarchical structures – the FNDDS hierarchy is based around major food groups: dairy; meat, poultry and seafood; eggs; nuts and legumes; grains; fruits; vegetables; fats and oils; and sugars, sweets, and beverages. Items fall into the groups regardless of preparation method or form. That is, broccoli prepared from frozen and from fresh both appear in the vegetable group, and for some fruits and vegetables, the fresh, frozen, canned and dried form are the same FNDDS item. Vegetable-based mixed dishes, such as broccoli and carrot stir-fry or soup, are also classified in the vegetable group. On the other hand, the scanner data classifies foods by grocery aisle. That is, the fresh and frozen broccoli are classified in different areas: produce and frozen vegetables. Similarly, when sold as a prepared food, the broccoli and carrot stir-fry may be found in the frozen entries, as a kit in either the frozen or produce section, refrigerated foods, or all of these.

To allow researchers to import the FNDDS nutrient data into the scanner data, a one-to-many match between FNDDS and scanner data items was needed. The food descriptions in the scanner data include brand names and package sizes and are written as a consumer would pronounce them – e.g., fresh and crisp broccoli florets, ready-cut, 10 oz – versus a more general FNDDS description such as “Broccoli, raw”. (Also linked to the “Broccoli, raw” code would be broccoli sold with stems attached, broccoli spears, and any other way raw broccoli is sold.) In the scanner data, the Universal Product Code (UPC) and the European Article Number (EAN) can link items between tables within the scanner data, as well as between datasets of grocery items, such as the USDA Global Branded Foods Product Database, a component of USDA’s Food Data Central. However, these codes are not related to the FNDDS codes, or any other column within the FNDDS. In other words, before development of the PPC, there were no established linking identifiers.

Figure 1 shows the process USDA uses to develop matches between scanner data and FNDDS.

We start the linking process by categorizing the scanner data items into homogeneous groups to make the first round of automated matching more efficient. To save time, we use the second lowest hierarchical category in the scanner data which generally divides items within a grocery aisle into homogenous groups such as produce, canned beans, baking mixes, and bread. Once the linking categories for scanner data are established, we select appropriate items from the FNDDS. Since the FNDDS is highly structured, this selection is usually straightforward.

Our next step is to use semantic matching to create a search table that aligns similar terms within the IRI product dictionary and FNDDS. This first requires that we extract attributes from the FNDDS descriptions into fields similar to those in the scanner data product dictionary. The FNDDS descriptions are found across multiple columns because they are added as the need arises to provide examples of brand names or alternative descriptions of foods which help code the foods WWEIA participants report eating. We manually create matching tables that link terms used in FNDDS to those used in the scanner data, organized by the fields defined in the restructured FNDDS. We then use this table as the basis of a probabilistic matching process. For example, when linking the produce group, “fresh” in the scanner data would be aligned with “raw” and “prepared from fresh” and NOT “prepared from frozen” in the FNDDS, and “broccoli florets” would also be aligned with “raw” and “broccoli”. Since the FNDDS is designed to code the foods individuals report eating, many of the foods in the FNDDS are already prepared and result in descriptions such as “broccoli, steamed, prepared from fresh” or “broccoli, boiled, prepared from frozen”.

Once the linking table is established, the probabilistic match process returns the single best possible match for each item in the scanner data. For example, a match between fresh broccoli florets and frozen broccoli would have a lower probability score than “broccoli, raw”. Because these matches form the basis of major USDA policies, we cannot accept an error rate of more than 5 percent, and lower is preferred. To reach that goal, nutritionists review every match to make sure the probabilistic match did not return a match between cauliflower florets and fresh broccoli, say, or that a broccoli and carrot stir-fry is not matched to a dish with broccoli, carrots, and chicken. The correct matches, such as the one between fresh broccoli florets and raw broccoli, are set aside while the items with an incorrect match, such as cauliflower florets and the broccoli and carrot stir-fry, are used to revise the search table. Revisions might include adding (NOT chicken) to the broccoli and carrot stir-fry dish. Mixed dishes — such as the broccoli and carrot stir-fry — pose particular challenges because there are a wide variety of similar products available in the grocery store. After a few rounds of revising the search table and running the probabilistic match process, it is more efficient to use a manual match, established by one nutritionist and reviewed by another, after which the match is assumed to be correct.

The process improved with each new wave of FNDDS and IRI data. Our first creation of the PPC linked the FNDDS 2011/12 to the 2013 IRI retail scanner data. Subsequent waves started with the previous search table and resulting matches were reviewed by nutritionists. We also used more fields in the IRI product dictionary to create the homogeneous linking groups and made modifications to these groups with each wave. During each wave we experimented with the number of rounds of probabilistic matching that was the most cost effective. For some linking groups it took less human time to manually match from the start, while for other groups it was more efficient to do multiple rounds of improvements to the search table. Starting with the most recent wave (matching FNDDS 2017/18 to the 2017 and 2018 retail scanner data), we assumed previous matches appearing in the newer data were correct. Although this assumption was good for most matches, a review demonstrated the need to review previous matches prior to removing the item from the list of scanner data items needing FNDDS matches. In the future we intend to explore methods developed by the participants of the Food for Thought competition.

Linking challenges

An ongoing challenge to the linking problem is that both the scanner data and the FNDDS undergo substantive changes each year, meaning that both the previous matches and search tables need to be reviewed and revised with each new effort, as tables that work with one cycle of FNDDS and scanner data will need revisions to use with the next cycle. Changes to the scanner data that impact our current method include dropped and added items, data corrections, and revisions to the categories that form the basis of the homogeneous linking groups. In addition, there are errors such as incorrect food descriptions, conflicting package size information, and changes in the item description from year to year. Since the FNDDS is designed to support dietary recall studies, revisions reflect both changes to available foods and the level of detail respondents can provide. These revisions result in dropped/added food codes, changes to food descriptions that impact which scanner data items match to the FNDDS items, and revisions to recipes used in the nutrient coding which impacts the number of retail ingredients available in the FNDDS.

Of the four parts of the PPS, establishing the matches is the most time-consuming task and constitutes at least 60 percent of the total budget. In the most recent round, we had 168 categories and each one went through 2-3 automated matching rounds; after each round, nutritionists spent an average of two hours reviewing the matches. This adds up to somewhere between 670 and 1,000 hours of review time. After the automated review, manual matching requires an additional 300 hours. Reducing the amount of time required to establish matches and link the FNDDS and retail scanner datasets may lead to significant time savings, resulting in faster data availability. That, in turn, could allow more timely policy-based research, and the mandated revision of the Thrifty Food Plan can continue with the most recent food price data.

Acknowledgements

The research presented in this compendium supports the Purchase to Plate Suite of data products. Carlson has been privileged to both develop and lead this project over the course of her career, but it is not a solo project. Many thanks to the Linkages Team from USDA’s Economic Research Service (Christopher Lowe, Mark Denbaly Elina Page, and Catherine Cullinane Thomas) the Center for Nutrition Policy and Promotion (Kristin Koegel, Kevin Kuczynski, Kevin Meyers Mathieu, TusaRebecca Pannucci), and our contractor Westat, Inc. (Thea Palmer Zimmerman, Carina E. Tornow, Amber Brown McFadden, Caitlin Carter, Viji Narayanaswamy, Lindsay McDougal, Elisha Lubar, Lynnea Brumby, Raquel Brown, and Maria Tamburri). Many others have supported this project over the years.

Competition structure

The competition ran over 10 months and consisted of three separate challenges: two interim, one final. Applications opened in September 2021, and the competition started in January 2022. Submission deadlines for the first and second interim challenges were in July and September 2022, respectively. For these rounds, participants submitted preliminary solutions for evaluation based solely on quantitative metrics, and two awards of $10,000 were given to the highest-scoring teams. The deadline for the final challenge was in October 2022. Here, solutions were evaluated by the scientific review board based on three judging criteria: quantitative metrics, transferability, and innovation. First, second, and third place winners received awards of $30,000, $1,500, and $1,000 respectively. Final presentations were given at the Food for Thought symposium in December 2022.

The competition was run entirely within the Coleridge Initiative’s Administrative Data Research Facility (ADRF), which was established by the United States Census Bureau to inform the decision-making of the Commission on Evidence-Based Policy under the Evidence Act. ADRF follows the Five Safes Framework: safe projects, safe people, safe data, safe settings, and safe outputs.

In keeping with this framework, participants were provided with ADRF login credentials after signing the relevant data use agreements during the onboarding process. All participants were required to agree to the ADRF terms of use, to complete security training, and to pass a security training assessment prior to accessing the challenge data. Participants’ access within ADRF was limited to the challenge environment and data only. There was no internet access, so Coleridge Initiative ensured that any packages requested by teams were available for use within the environment after passing security review. All codes and documentation were only allowed to be exported outside ADRF after export reviews from both Coleridge Initiative and USDA staff. At the end of each challenge, the teams submitted write-ups and supporting files by placing all the necessary submission files in their ADRF team folder. Detailed submission instructions are available via the Real World Data Science GitHub repository.

Metrics

Submissions were evaluated by Coleridge Initiative and technical review and subject review boards based on the following criteria:

• Quantitative metrics were used to measure the predictive accuracy and runtime of the model.
  
• Transferability measured the quality of documentation and code, and the ability of individuals who are not involved in model development to replicate and implement the team’s approach.
  
• Innovation measured novelty and creativity of the model in addressing the linkage problem.

Technical review was overseen by faculty members from computer science and engineering departments of top US universities. Subject review was handled by subject matter experts from USDA and Westat.

From a quantitative perspective, the most common way to evaluate machine learning competition submissions is to use model predictive accuracy. However, single metrics are typically incomplete descriptions of real-world tasks, and they can easily hide significant differences between models which simple predictive accuracy cannot capture. To select the most appropriate official challenge metrics, Coleridge Initiative reviewed the literature on the use of evaluation measures in both classification and ranking task machine learning competitions. Success at 5 (S@5) and Normalized Discounted Cumulative Gain at 5 (NDCG@5) scores were ultimately used as the quantitative metrics.

The metrics were applied as follows: models proposed by each team were tasked with outputting five potential FNDDS matches for each IRI code, with potential FNDDS matches ordered from most likely to least likely. S@5 and NDCG@5 scores are broadly similar – both measure whether a correct match is present in the five proposed matches that participants were asked to identify. However, S@5 does not take rank position into account and only considers whether the five proposed FNDDS matches contain the correct FNDDS response. NDCG@5 does take rank into account and also measures how highly the correct FNDDS response is ranked among the five proposed matches. Both measures range from 0 to 1 (or 0% to 100%). Models get a “full credit” for S@5 as long as they contain the correct FNDDS option. NDCG@5 penalizes models when the correct match is ranked lower on the list of 5 proposed matches.

Technical description

Results

The next few articles in this collection walk readers through the solutions proposed by competition finalists. Figure 3 provides a brief summary.

Lessons learned

It was undoubtedly challenging for teams to work with highly secured data in a private data enclave for this data challenge. We solicited feedback from teams and summarized the issues that we experienced throughout the competitions, together with the solutions to resolve those issues. Below are our main lessons learned and we hope this summary can serve to inform future competitions.

• Environmental factors: The installation and setup of packages, libraries, and resources, as well as the configuration of GPUs, system dependencies, and workspace design were expected to take a long time as each team had their own needs. To accelerate the process, we requested a list of specific package and environment requirements from the teams in advance. However, due to the complexity of the system configuration required by the teams, environment setup took longer than expected. Thus, the challenge deadlines had to be postponed a few times to accommodate this.

• Time commitment: Twelve teams were selected to participate in the challenge, but only three teams remained in the final challenge. Other than one team that was disqualified for violating the ADRF terms of use agreement, eight dropped out because of other commitments and insufficient time to meaningfully participate. To ensure security, ADRF does not allow jobs to run in the backend, which also adds to the time commitment of teams. To encourage teams to participate in the final challenge, we gave out additional awards for second and third places.

• Computing resource limit: One issue encountered in evaluating submitted models was computing environment resource limits due to the secured nature of the data enclave. The original private test dataset is four times larger than the public test dataset, making it unfeasible to evaluate. To overcome this issue, given the fixed resource constraints, we decided to reduce the private test set to 40% of its original size. It would have been helpful, though, if the competition had set a model running time limit at the outset, so that participants could build simpler yet effective models.

• Supporting code: Although the initial baseline model we provided was extremely simple, we found this helped participants a lot in the initial phase – yet there is space to improve. To be specific, supporting codes should be constructed so that all relevant data tables are used and specify the main function to run the code, especially how the model should be tested. The teams only used the main table, which was the only table that was used in the baseline model, for training and did not touch the other supporting table. If we included the other table in the baseline model, it could help participants to have a better use of this data as well. In addition, a baseline model should be intuitive for the participants to follow, allowing evaluators to easily replace the public test set with the private test set without any programming modifications.

Davit’s journey towards reproducibility

```{python}
# Hardcoded path
file_path = "/user/notebooks/toydata.csv"
try:
    with open(file_path) as file:
        data = file.read()
        print(data)
except FileNotFoundError:
    print("File not found")
```

```{python}
# Relative path
import os

file_path = os.path.join(os.getcwd(), "toydata.csv")
try:
    with open(file_path) as file:
        data = file.read()
        print(data)
except FileNotFoundError:
    print("File not found")
```

```{python}
# Set an arbitrary predictive performance value of a benchmark model
# and accept/reject models if the results are above/below the value.
benchmark = 50
# Set the model details in one place for a better overview
model = {
    "model1": {"name": "model1", "type": "simple"}, 
    "model2": {"name": "model2", "type": "complex"}
}
# Set the current model to "model1" to use it for training and check its results
current_model = model["model1"]
# Train a simple model for "model1" and a complex model for "model2"
# Training result of the "model1" is 30 and for "model2" is 70
model_structure = train(current_model["type"])
# Save the model and its result in a .csv file
model_structure.to_csv('/all/notebooks/results-of-model1.csv', index=False)
```

```{python}
# Load the model result and compare with benchmark
print("Model name: {}".format(current_model["name"]))
print("Model type: {}".format(current_model["type"]))
# Load the result of the current model
result = pd.read_csv('/all/notebooks/results-of-model2.csv').iloc[0, 0]
print("Result: {}".format(result))

if result > benchmark:
    print("\033[3;32m>>> Result is better than the benchmark -> Accept the model and use it for calculations")
else:
    print("\033[3;31m>>> Result is NOT better than the benchmark -> Reject the model as it is not optimal")
```

```{python}
import os
# Set an arbitrary predictive performance value of a benchmark model
# and accept/reject models if the results are above/below the value.
benchmark = 50
# Set the model details (INCLUDING PATHS) in one place for a better overview
model = {
    "model1": {"name": "model1", "type": "simple", "path": os.path.join(os.getcwd(), "results-of-model1.csv")}, 
    "model2": {"name": "model2", "type": "complex", "path": os.path.join(os.getcwd(), "results-of-model2.csv")}
}
# Set the current model to "model1" to use it for training and check its results
current_model = model["model1"]
# Train a simple model for "model1" and a complex model for "model2"
# Training result of the "model1" is 30 and for "model2" is 70
model_structure = train(current_model["type"])
# Save the model and its result in a .csv file
model_structure.to_csv(current_model["path"], index=False)
```

```{python}
# Get the model result and compare with the benchmark
print("Model name: {}".format(current_model["name"]))
print("Model type: {}".format(current_model["type"]))
# Load the result of the current model WITH a VARIABLE PATH
result = pd.read_csv(current_model["path"]).iloc[0, 0]
print("Result: {}".format(result))

if result > benchmark:
    print("\033[3;32m>>> Result is better than the benchmark -> Accept the model and use it for calculations")
else:
    print("\033[3;31m>>> Result is NOT better than the benchmark -> Reject the model as it is not optimal")
```

```{python}
# Path works on macOS/Linux
with open("../../all/notebooks/toydata.csv", "r") as f:
    print(f.read())

# Path works only on Windows    
with open(r"..\..\all\notebooks\toydata.csv", "r") as f:
   print(f.read())
```

```{python}
from pathlib import Path

# Path works on every OS: macOS/Linux/Windows
# It will automatically replace the path to "..\..\all\notebooks\toydata.csv" when it runs on Windows
with open(Path("../../all/notebooks/toydata.csv"), "r") as f:
    print(f.read())
```

The deep learning challenge

So far, this article has dealt with barriers to reproducibility – and ways around them – that will apply to most, if not all, modern research projects. While I’d encourage any scientist to adopt these practices in their own work, it is important to stress that these alone cannot guarantee reproducibility. In cases where standard statistical procedures are used within statistical software packages, reproducibility is often achievable. However, in reality, even when following the same procedures, differences in outputs can occur, and identifying the reasons for this may be challenging. Cooking offers a simple analogy: subtle changes in room temperature or ingredient quality from one day to the next can impact the final product.

One of the challenges for research projects employing machine learning and deep learning algorithms is that outputs can be influenced by the randomness that is inherent in these approaches. Consider the four portraits below, generated by the Midjourney bot.

Each portrait looks broadly similar at first glance. However, upon closer inspection, critical differences emerge. These differences arise because deep learning models rely on numerous interconnected layers to learn intricate patterns and representations. Slight random perturbations, such as initial parameter values or changes in data samples, can propagate through the network, leading to different decisions during the learning process. As a result, even seemingly negligible randomness can amplify and manifest as considerable differences in the final output, as with the distinct features of the portraits.

Randomness is not necessarily a bad thing – it mitigates overfitting and helps predictions to be generalised. However, it does present an additional barrier to reproducibility. If you cannot get the same results using the same raw materials – data, code, packages and computing environment – then you might have good reasons to doubt the validity of the findings.

There are many elements of an analysis in which randomness may be present and lead to different results. For example, in a classification (where your dependent variable is binary, e.g., success/failure with 1 and 0) or a regression (where your dependent variable is continuous, e.g., temperature measurements of 10.1°C, 2.8°C, etc.), you might need to split your data into training and testing sets. The training set is used to estimate the model (hyper)parameters and the testing set is used to compute the performance of the model. The way the split is usually operationalised is as a random selection of rows of your data. So, in principle, each time you split your data into training and testing sets, you may end up with different rows in each set. Differences in the training set may therefore lead to different values of the model (hyper)parameters and affect the predictive performance that is measured from the testing set. Also, differences in the testing set may lead to variations in the predictive performance scores, which in turn lead to potentially different interpretations and, ultimately, decisions if the results are used for that purpose.

This aspect of randomness in the training of models is relatively well known. But randomness may hide in other parts of code. One such example is illustrated below. Here, using Python, we set the seed number to 0 using np.random.seed(seed value). The random.seed() function from the package numpy (abbreviated np) saves the state of a random function so that it can create identical random numbers independently of the machine you use, and this is for any number of executions. A seed value is an initial input or starting point used by a pseudorandom number generator to generate a sequence of random numbers. It is often an integer or a timestamp. The number generator takes this seed value and uses it to produce a deterministic series of random numbers that appear to be random but can be recreated by using the same seed value. Without providing this seed value, the first execution of the function typically uses the current system time. The animation below generates two random arrays arr1 and arr2 using np.random.rand(3,2). Note that the values 3,2 indicate that we want random values for an array that has 3 rows and 2 columns.

```{python}
import numpy as np

#Set the seed number e.g. to 0
np.random.seed(0)
# Generate random array
arr1 = np.random.rand(3,2)
## print("Array 1:")
## print(arr1)

#Set the seed number as before to get the same results
np.random.seed(0)
# Generate another random array
arr2 = np.random.rand(3,2)
## print("\nArray 2:")
## print(arr2)
```

If you run the code yourself multiple times, the values of arr1 and arr2 should remain identical. If this is not the case, check that the seed value is set to 0 in lines 4 and 11. These identical results are possible because we set the seed value to 0, which ensures that the random number generator produces the same sequence of numbers each time the code is run. Now, let’s look at what happens if we remove the line np.random.seed(0):

```{python}
#Generate random array
arr1 = np.random.rand(3,2)
## print("Array 1:")
## print(arr1)

#Generate another random array
arr2 = np.random.rand(3,2)
## print("\nArray 2:")
## print(arr2)
```

Here, the values of arr1 and arr2 will be different each time we run the code since the seed value was not set and is therefore changing over time.

This short code demonstrates how randomness that can be controlled by the seed value may affect your code. Therefore, unless randomness is required, e.g., to get some uncertainty in the results, setting the seed value will contribute to making your work reproducible. I also find it helpful to document the seed number I use in my code so that I can easily reproduce my findings in the future. If you are currently working on some code that involves random number generators, it might be worth checking your code and making all necessary changes. In our work (see code chunk 9 in the Jupyter notebook) we set the seed value in a general way, using a framework (config) so that our code always uses the same seed to train our algorithm.

Conclusion

We hope you have enjoyed learning more about our quest for reproducibility. We have explained why reproducibility matters and provided tips for how to achieve it – or, at least, work towards it. We have introduced a few important issues that you are likely to encounter on your own path to reproducibility. In sum, we have mentioned:

• The importance of having relative instead of hard-coded paths in code.
• Operating system compatibility issues, which can be solved by using Docker containers for a consistent computing environment.
• The convenience of Jupyter notebooks for code editing – particularly useful for data science projects and work using deep learning because of the ability to include text and code in the same document and make the work accessible to everyone (so long as they have an internet connection).
• The need for version control using, for example, Git and GitHub, which allows you to keep track of changes in your code and collaborate with others efficiently.
• The importance of setting the seed values in random number generators.

The graphic below provides a visual overview of the different components of our study and shows how each component works with the others to support reproducibility.

We use (A) the version control system, Git, and its hosting service, GitHub, which enables a team to share code with peers, efficiently track and synchronise code changes between local and server machines, and reset the project to a working state in case something breaks. Docker containers (B) include all necessary objects (engine, data, and scripts). Docker needs to be installed (plain-line arrows) by all users (project leader, collaborator(s), reviewer(s), and public user(s)) on their local machines (C); and (D) we use a user-friendly interface (JupyterLab) deployed from a local machine to facilitate the operations required to reproduce the work. The project leader and collaborators can edit (upload/download) the project files stored on the GitHub server (plain-line arrows) while reviewers and public users can only read the files (dotted-line arrows).

Now, it is over to you. Our Jupyter notebook provides a walkthrough of our research. Our GitHub repository has all the data, code and other files you need to reproduce our work, and this README file will help you get started.

And with that, we wish you all the best on the road to reproducibility!