When Accuracy Does Not Transfer: A Deployment-Grounded Evaluation Framework for Clinical Artificial Intelligence in Resource-Variable Health Systems

Background. Clinical artificial intelligence (AI) models increasingly inform patient care, yet performance measured during development frequently fails to transfer to the settings that adopt them, and the failure is often silent. This risk is greatest for under-resourced health systems, which rarely hold training-representative data or the infrastructure to detect degradation after deployment. Objective. To characterise the mechanisms and magnitude of the performance-transfer gap in clinical AI and to specify a reproducible, deployment-grounded evaluation protocol that reduces the risk of undetected failure. Methods. We conducted a structured synthesis of peer-reviewed external-validation and human-centred field evidence, graded each source with an explicit evidence-tier scheme, and derived a five-component evaluation framework aligned with consensus AI reporting standards (TRIPOD+AI, DECIDE-AI, SPIRIT-AI, and CONSORT-AI). Results. External validation of a widely deployed proprietary sepsis model showed the area under the curve fall from a developer-reported 0.76–0.83 to 0.63, with 33% sensitivity, a 12% positive predictive value, and roughly two-thirds of septic patients missed at the operational alerting threshold. Field studies of diabetic-retinopathy screening demonstrated that laboratory-calibrated input thresholds and socio-environmental workflow factors, rather than model accuracy alone, governed real-world performance. Three mechanisms — population shift, data-and-workflow shift, and the human-factors gap — account for most observed degradation. Conclusion. A reported accuracy figure is a property of a model in a specific setting, not of the model in general. Local pre-deployment validation, sample-size-honest subgroup reporting, silent-mode monitoring, and continuous drift surveillance should be treated as core requirements rather than optional additions. The framework is presented as an evaluation commitment and evidence review, not as outcome data from a completed deployment. Keywords: clinical decision support; external validation; distribution shift; model calibration; algorithmic fairness; post-deployment monitoring; healthcare AI governance; low-resource settings.