Interpretability of an FDA-authorized AI/ML sepsis diagnostic tool improved by SHAP values.

Authors

• Gregory L Watson
• Ramona Berghea, Corewell Health East
• Francisco Davila, Corewell Health East
• Hugo Davila, Corewell Health East
• Carmen DeMarco, Corewell Health East
• Aimee Espinosa, Corewell Health East
• Alexandra Halalau, Corewell Health East
• Nicholas Maddens, Corewell Health East
• Scott Smith, Corewell Health East
• Matthew D. Sims, Corewell Health East

Document Type

Article

Publication Date

2-25-2026

Publication Title

JAMIA Open

Abstract

OBJECTIVES: To assess the interpretability and acceptance of Shapley values for making artificial intelligence/machine learning (AI/ML) tools more transparent, interpretable, and useful to clinicians.

MATERIALS AND METHODS: Structured assessments were conducted with 30 clinicians (15 providers; 15 nurses; 8 assessments per clinician) to evaluate their ability to understand interventional Shapley Additive exPlanations (SHAP) values, a type of Shapley value that provides individualized variable importance scores and ascertain their perspective on SHAP value utility for the use of an AI/ML sepsis diagnostic. Participants were shown the diagnostic interface for real clinical scenarios with de-identified patient data with and without SHAP values. The primary outcomes were clinician ability to correctly interpret SHAP values and clinician self-reported improvement in their understanding of how the AI/ML algorithm produced its result.

RESULTS: Participants correctly interpreted SHAP values in 235 of 240 assessments (98%; CI, 95%-99%) and reported SHAP values improved their understanding of how the algorithm produced its result in every case (240/240; 100%; CI, 99%-100%). Participants were unanimous (30/30) in preferring the interface with SHAP values over the interface without.

DISCUSSION: Clinician participants strongly preferred the device interface with SHAP values, were unanimous in reporting SHAP values improved their understanding of the AI/ML diagnostic, and scored nearly perfectly when asked to interpret SHAP values.

CONCLUSION: These results suggest health care providers value transparency into AI/ML algorithms designed for clinical use, and that Shapley values are a useful approach to providing that transparency, which in turn may improve tool adoption and clinical utility.

Volume

9

Issue

1

First Page

ooag020

Recommended Citation

Watson GL, Staples G, Carver R, Bhargava A, López-Espina C, Schmalz L, et al Ali F, [Berghea R, Davila F, Davila H, DeMarco C, Espinosa A, Halalau A, Maddens N, Smith S, Sims MD]. Interpretability of an FDA-authorized AI/ML sepsis diagnostic tool improved by SHAP values. JAMIA Open. 2026 Feb 25;9(1):ooag020. doi: 10.1093/jamiaopen/ooag020. PMID: 41767180

DOI

10.1093/jamiaopen/ooag020

ISSN

2574-2531

PubMed ID

41767180