Clinical artificial intelligence is no longer a speculative layer around health care. Algorithms flag suspected strokes on scans, guide image acquisition, estimate cardiac risk, summarize visits, and rank patients for follow-up. The U.S. Food and Drug Administration’s public list spans radiology, cardiovascular care, neurology, pathology, and other specialties and includes multiple final decisions dated June 29, 2026 (FDA).

But authorization, clinical usefulness, and public trust are different tests. A model can meet a premarket standard and still perform poorly at a new hospital, for a demographic group underrepresented in its data, or after workflows and disease patterns change. The central policy question is no longer whether AI can assist care. It is whether health systems can continuously show that a particular tool helps the intended patients under actual conditions.

The evidence gap behind impressive accuracy

Many clinical AI claims rest on retrospective datasets: the model is tested against records or images collected in the past. Such studies are useful, but they do not reproduce the pressures of live care, where missing data, unusual cases, time constraints, and clinician behavior affect outcomes.

A 2025 cross-sectional study of 903 FDA-authorized AI-enabled devices found that clinical performance studies were reported for 505, or 55.9%. Of those 505, only 41 studies were prospective and 12 were randomized. Sex-specific results were available for 28.7%, and age-specific results for 23.2% (JAMA Network Open). Those figures do not prove that the devices are unsafe. They show how often public evidence leaves unanswered questions about generalizability and patient benefit.

Clinical performance is also relational. A warning system may be statistically accurate yet cause harm if it produces too many alerts, arrives after a decision has been made, or encourages clinicians to defer reflexively to a score. The World Health Organization warns that large multimodal models can produce false, biased, or incomplete statements and can foster automation bias, in which users overlook errors they might otherwise catch (WHO).

Regulation is becoming a lifecycle process

The FDA’s position is usefully direct: “The FDA does not regulate AI as such”. It regulates products according to intended use, risk, and legal category. Some clinical decision-support functions are medical devices; others may fall outside the device definition if a health professional can independently review the basis for a recommendation.

The agency’s January 2026 final guidance on clinical decision-support software clarified that boundary and supplied examples of device and non-device functions (FDA). That distinction matters as general-purpose models are embedded in note-taking, triage, and patient communications. A broad model may perform hundreds of functions, but regulatory treatment turns on the specific claim and context.

Clinical AI Tools Face a Test of Trust, Safety, and Regulation — supporting image
Clinical AI Tools Face a Test of Trust, Safety, and Regulation — supporting image

Adaptive systems add another problem: the version evaluated before deployment may not be the version used months later. FDA guidance now allows a predetermined change control plan to describe anticipated modifications and how they will be validated. International regulators have also converged on ten Good Machine Learning Practice principles, including representative datasets, independent test sets, clinically relevant testing, clear information for users, and monitoring after deployment (FDA, Health Canada, and MHRA).

Trust must be engineered into the workflow

Hospitals should treat AI procurement more like the adoption of a clinical service than the purchase of ordinary software. Before launch, they need local validation against their patient population and equipment. During use, they need subgroup measures, calibration checks, documented overrides, incident reporting, and a plan for model or data drift. A dashboard showing average accuracy is not enough if performance differs by age, skin tone, language, sex, disability, or site.

Responsibility must also be legible. Clinicians need to know what data shaped an output, its intended use, important limitations, and whom to contact when it fails. Patients need plain-language notice when AI materially influences care and a meaningful route to human review. Developers and hospitals should define who investigates an error, who can suspend the system, and how affected patients are informed.

The WHO’s governance framework places autonomy, safety, transparency, accountability, inclusion, and sustainability at the center of health AI (WHO). Those principles become real only through operating rules: representative trials, monitored deployment, accessible explanations, and enforceable redress.

Clinical AI will earn trust neither through novelty nor through a regulatory clearance alone. Trust will come from evidence that survives contact with diverse patients and changing clinics—and from institutions prepared to stop or modify a tool when that evidence weakens.

Sources