Medical AI Needs a Safe Failure Plan, Not Just a Successful Demo
By Michael T. Neely
Credits: marionbrun (Pixabay)
In 2018 and 2019, a deep learning system that screens retinal photographs for diabetic eye disease was put to work in clinics in Thailand’s Pathum Thani province, where nurses run eye screening because specialists are scarce. In the first six months, the system rejected 393 of 1,838 images (21%) as ungradable, often because photos were taken in rooms that could not be darkened. Slow connections meant some images took 60 to 90 seconds to upload; when one clinic’s internet went down for two hours, the number of patients screened fell from 200 to 100.
Under the study protocol, a patient with an ungradable image was to be referred to a provincial hospital an hour’s drive away, a heavy cost for people who could not miss work or had no way to travel (Beede et al., 2020). None of this was about whether the model could read a good image. It was about what happened when it could not.
Global health discussions about artificial intelligence often begin with performance: Can a model detect disease, predict deterioration, classify an image, or support a treatment decision? Those questions matter, but they are not enough. In real health systems, safety depends just as much on what happens when the technology is unavailable, wrong, poorly matched to the local population, or updated after deployment.
From global guidelines to local governance
This operational layer matters more in resource-constrained health systems. A model validated in a well-resourced academic centre may be used in a setting with different patient populations, staffing patterns, connectivity, equipment, languages, clinical pathways, and access to specialist review. The governance question is whether the institution using it can tell when the model stops working as intended and whether care stays safe when the tool should not be trusted.
Equity, then, is about who receives the capacity to govern AI safely, not only who receives AI.
International guidance already calls for governance across the AI lifecycle, not one-time approval. The World Health Organization’s 2021 guidance on the ethics and governance of AI for health sets out six principles for designing and using these tools. WHO’s 2023 regulatory considerations on AI for health call for documentation across the whole product lifecycle. The FUTURE-AI guideline, a consensus of 117 experts from 50 countries published in The BMJ in 2025, recommends showing local clinical validity, auditing and updating tools periodically, and logging how each tool is used. The five parts below overlap closely with FUTURE-AI’s 30 recommendations, which also cover intended use and human oversight. The difference is the audience: FUTURE-AI is addressed mainly to the teams that build AI tools, while this framework is for the hospitals, clinics, and health programs that deploy them.
Establishing safeguards against medical AI
The first is a clear boundary around intended use. Before deployment, a health organization should be able to state what the system is designed to do, in which populations and settings it has been validated, and what decisions remain outside its scope. If a tool was validated in one clinical environment, extending it to another should require a deliberate review. It should not spread to new wards or patient groups simply because it is already installed.
Second, local validation should be proportional to risk and local conditions. Smaller systems cannot always match the validation programs of large health networks, but they can still ask essential questions:
Does performance differ by age, sex, language, local prevalence of the condition, or other locally relevant characteristics? Prevalence matters even when sensitivity and specificity hold steady: where a disease is common, a negative result reassures less; where it is rare, a positive result is more often wrong. Do local images, laboratory values, and records resemble the data the model was built on? Are clinicians seeing false reassurance or unnecessary escalation?
A smaller validation process that asks the right local questions can be more useful than a sophisticated evaluation conducted elsewhere. The FUTURE-AI panel reached a similar view, setting aside mandatory third-party clinical evaluation as impractical at scale, especially in resource-limited settings.
Third, every AI-dependent workflow needs a fallback. Current guidance says least about this: FUTURE-AI mentions an alternative procedure only in passing, as one way to mitigate risk. Hospitals routinely plan for power loss, network downtime, staffing shortages, and equipment failure; AI should be treated the same way. Teams should know when to switch to a manual or non-AI process, who has authority to make that decision, and how the fallback is tested. In the Thai clinics, the researchers amended the protocol so that eye specialists reviewed ungradable images alongside patients’ records instead of referring every such patient automatically, which reduced unnecessary travel and missed work (Beede, 2020). That fallback was designed after deployment; it should be designed before.
If care cannot safely continue without the model, that dependency itself becomes a governance risk.
Fourth, institutions need version awareness. AI systems can change through vendor updates, new model releases, altered thresholds, revised data pipelines, or changes in surrounding software. A decision made with version 3 of a tool may not be reproducible with version 5. Tuberculosis screening shows why. WHO’s 2021 screening guidelines allow computer-aided detection software to read chest X-rays in place of a human reader when screening people aged 15 and older for tuberculosis. A companion WHO/TDR toolkit asks programs to set the software’s score threshold from local data, noting that thresholds differ across products and even across updated versions of the same software. Health systems therefore need enough documentation to know which version influenced a significant decision, and should recheck local thresholds after each update.
Fifth, there must be a pathway for escalation and redress. Clinicians and patients need a way to question an AI-influenced outcome without being told that the system’s answer is final. That starts with telling patients when an AI tool shaped a decision about their care, because no one can question a result they do not know a tool produced. The route must also be realistic: where a facility has no formal complaints process, it can be the right to ask the clinician in charge for a human review, with unresolved cases going to the facility’s quality committee.
Meaningful human review requires authority to reconsider the result, access to relevant information, and a way to correct records or decisions when an error is found. Transparency without a correction pathway offers little protection.
Who should supply this capacity?
A resource-constrained hospital that is given an advanced model without local validation support, fallback procedures, documentation, and escalation authority has received technology without the infrastructure required to control its risks. Donors, vendors, and ministries of health often decide which tools reach a district hospital; they should share that responsibility. The cost of local validation and fallback planning should be built into every grant or purchase that brings an AI tool to a hospital like this.
The most useful measure of responsible medical AI may therefore be operational rather than promotional: Can the health system explain when the tool should be used, recognize when it should not be used, continue care when it fails, document what changed, and give patients and clinicians a meaningful way to challenge harmful outcomes? If any answer is no, the implementation is not finished, regardless of how impressive the demonstration looked.
AI-use statement: The author used OpenAI’s ChatGPT to help organize sources, propose an outline and the order of sections (the “structural drafting” in the earlier version of this statement), edit language for clarity, and format the document. In revising the essay after editorial review, the author used Tab, an AI assistant, to locate and check the cited sources and to draft revised passages, including the case examples. The author directed the argument, reviewed and verified every source and claim, and is responsible for the final text.
is the publisher and owner of XS10 Magazine, a board member of the Global Music Conference, and an independent AI policy commentator. He writes on responsible AI, human oversight, accountability, and practical governance in high-stakes systems. For more than two years, he has provided free AI education and mentoring to creative-community learners. He is also the recipient of an honorary doctorate in music.