FolioStart free

Computer Science · Literature

Research papers on Explainable AI in healthcare

Recent and highly-cited academic work on explainable ai in healthcare, gathered from Semantic Scholar, CrossRef and OpenAlex.

Search all 200M+ papers on this topic, free →Or track new explainable ai in healthcare papers automatically as they publish

  3. A survey of methods for explaining black box models

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, et al. · 2019 · ISTI Open Portal · 4,994 citations

    In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provi

    Save this paper

  4. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

    Professor Gary S. Collins, Karel G.M. Moons, Paula Dhiman, et al. · 2024 · BMJ · 3,185 citations

    The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement was published in 2015 to provide the minimum reporting recommendations for studies developing or evaluating the performance of a prediction model. Methodological advances in the field of prediction have since included the widespread use of artificial intelligence (AI) powered by machine learning methods to develop prediction models. An update to the TRIPOD statement is thus needed. TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used. The new checklist supersedes

    Save this paper

  5. An overview of clinical decision support systems: benefits, risks, and strategies for success

    Reed T. Sutton, David Pincock, Daniel C. Baumgart, et al. · 2020 · npj Digital Medicine · 3,044 citations

    Computerized clinical decision support systems, or CDSS, represent a paradigm shift in healthcare today. CDSS are used to augment clinicians in their complex decision-making processes. Since their first use in the 1980s, CDSS have seen a rapid evolution. They are now commonly administered through electronic medical records and other computerized clinical workflows, which has been facilitated by increasing global adoption of electronic medical records with advanced capabilities. Despite these advances, there remain unknowns regarding the effect CDSS have on the providers who use them, patient outcomes, and costs. There have been numerous published examples in the past decade(s) of CDSS succes

    Save this paper

  6. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective

    Julia Amann, Alessandro Blasimme, Effy Vayena, et al. · 2020 · BMC Medical Informatics and Decision Making · 2,108 citations

    BACKGROUND: Explainability is one of the most heavily debated topics when it comes to the application of artificial intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evaluation of what explainability means for the adoption of AI-driven tools into clinical practice. METHODS: Taking A

    Save this paper

  7. The false hope of current approaches to explainable artificial intelligence in health care

    Marzyeh Ghassemi, Luke Oakden‐Rayner, Andrew L. Beam · 2021 · The Lancet Digital Health · 1,575 citations

    The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for deci

    Save this paper

  8. Designing Theory-Driven User-Centric Explainable AI

    Danding Wang, Qian Yang, Ashraf Abdul, et al. · 2019 · 874 citations

    From healthcare to criminal justice, artificial intelligence (AI) is increasingly supporting high-consequence human decisions. This has spurred the field of explainable AI (XAI). This paper seeks to strengthen empirical application-specific investigations of XAI by exploring theoretical underpinnings of human decision making, drawing from the fields of philosophy and psychology. In this paper, we propose a conceptual framework for building human-centered, decision-theory-driven XAI based on an extensive review across these fields. Drawing on this framework, we identify pathways along which human cognitive patterns drives needs for building XAI and how XAI can mitigate common cognitive biases

    Save this paper

  9. Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review

    Anna Markella Antoniadi, Yuhan Du, Yasmine Guendouz, et al. · 2021 · Applied Sciences · 615 citations

    Machine Learning and Artificial Intelligence (AI) more broadly have great immediate and future potential for transforming almost all aspects of medicine. However, in many applications, even outside medicine, a lack of transparency in AI applications has become increasingly problematic. This is particularly pronounced where users need to interpret the output of AI systems. Explainable AI (XAI) provides a rationale that allows users to understand why a system has produced a given output. The output can then be interpreted within a given context. One area that is in great need of XAI is that of Clinical Decision Support Systems (CDSSs). These systems support medical practitioners in their clini

    Save this paper

  10. Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges

    Qaiser Abbas, Woonyoung Jeong, S. Lee · 2025 · Healthcare · 223 citations

    Background: Theintegration of artificial intelligence (AI) into clinical decision support systems (CDSSs) has significantly enhanced diagnostic precision, risk stratification, and treatment planning. AI models remain a barrier to clinical adoption, emphasizing the critical role of explainable AI (XAI). Methods: This systematic meta-analysis synthesizes findings from 62 peer-reviewed studies published between 2018 and 2025, examining the use of XAI methods within CDSSs across various clinical domains, including radiology, oncology, neurology, and critical care. Model-agnostic techniques such as visualization models like Gradient-weighted Class Activation Mapping (Grad-CAM) and attention mecha

    Save this paper

  11. Co-design of Human-centered, Explainable AI for Clinical Decision Support

    Cecilia Panigutti, Andrea Beretta, D. Fadda, et al. · 2023 · ACM Transactions on Interactive Intelligent Systems · 117 citations

    eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (D

    Save this paper

  13. Explainability in medicine in an era of AI-based clinical decision support systems

    Robin Pierce, Wim Van Biesen, Daan Van Cauwenberge, et al. · 2022 · Frontiers in Genetics · 92 citations

    The combination of "Big Data" and Artificial Intelligence (AI) is frequently promoted as having the potential to deliver valuable health benefits when applied to medical decision-making. However, the responsible adoption of AI-based clinical decision support systems faces several challenges at both the individual and societal level. One of the features that has given rise to particular concern is the issue of explainability, since, if the way an algorithm arrived at a particular output is not known (or knowable) to a physician, this may lead to multiple challenges, including an inability to evaluate the merits of the output. This "opacity" problem has led to questions about whether physician

    Save this paper

  14. Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis

    Mozhgan Salimparsa, K. Sedig, Dan Lizotte, et al. · 2025 · Informatics · 39 citations

    While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-cent

    Save this paper

  15. A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

    Alessandro Gambetti, Qiwei Han, Hong Shen, et al. · 2025 · ArXiv · 20 citations

    Explainable Artificial Intelligence (XAI) is essential for the transparency and clinical adoption of Clinical Decision Support Systems (CDSS). However, the real-world effectiveness of existing XAI methods remains limited and is inconsistently evaluated. This study conducts a systematic PRISMA-guided survey of 31 human-centered evaluations (HCE) of XAI applied to CDSS, classifying them by XAI methodology, evaluation design, and adoption barrier. Our findings reveal that most existing studies employ post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, typically assessed through small-scale clinician studies. The results show that over 80% of the studies adopt post-hoc, model-agnostic

    Save this paper

  16. Explainable AI in Healthcare: Systematic Review of Clinical Decision Support Systems

    N. A. Aziz, Awais Manzoor, Muhammad Deedahwar, et al. · 2024 · 17 citations

    This systematic review examines the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in Clinical Decision Support Systems (CDSS), highlighting significant advancements and identifying persistent challenges. Utilising the PRISMA protocol, we searched major indexed databases such as Scopus, Web of Science, PubMed, and the Cochrane Library, to analyse publications from January 2000 to April 2024. This timeframe captures the progressive integration of XAI in CDSS, offering a historical and technological overview. The review covers the datasets, application areas, machine learning models, explainable AI methods, and evaluation strategies for multiple XAI methods. Analy

    Save this paper

  17. Explainable AI for Transparent MRI Segmentation: Deep Learning and Visual Attribution in Clinical Decision Support

    V. M, Jayapradha V, A. K., et al. · 2024 · International Journal of Computational and Experimental Science and Engineering · 14 citations

    For medical diagnosis and therapy planning, the importance of accurate MRI segmentation cannot be overemphasized. Conversely, the inscrutability of deep learning models remains obstacles to their application in therapeutic contexts. In this article, an interpretability artificial intelligence framework is introduced. It combines an MRI segmentation model based on deep learning, visual attribution algorithms and natural language explanations. EXPERIMENT The dataset is consisting of plenty of different types of brain MRI scans, and used to test the architecture. The average of Dice score of our method is 88.7% and 92.3% for segmentation of tumor and categorization of tissues, respectively. Bot

    Save this paper

  18. Multi-task reinforcement learning and explainable AI-Driven platform for personalized planning and clinical decision support in orthodontic-orthognathic treatment

    Zhiyuan Li, Liwei Wang · 2025 · Scientific Reports · 10 citations

    This study presents a novel clinical decision support platform for orthodontic-orthognathic treatment that integrates multi-task reinforcement learning with explainable artificial intelligence. The platform addresses the challenges of personalized treatment planning in complex dentofacial deformities by formulating treatment as a sequential decision-making process optimizing multiple clinical objectives simultaneously. We developed a comprehensive framework comprising: (1) a multi-task reinforcement learning core with specialized state-action representations for craniofacial structures; (2) complementary explainable AI components that render complex model decisions interpretable within clini

    Save this paper

  19. Explainable AI for Trustworthy Clinical Decision Support: A Case-Based Reasoning System for Nursing Assistants

    Yen-Ku Liu, Yun-Cheng Tsai · 2024 · 2024 IEEE International Conference on Big Data (BigData) · 8 citations

    The increasing complexity of patient data and the need for reliable, transparent decision-making in healthcare drive the demand for advanced AI systems. This paper introduces a Clinical Decision Support System (CDSS) integrating Explainable AI (XAI) with Retrieval-Augmented Generation (RAG), designed to offer transparent, real-time, and evidence-based recommendations. HIPAA-compliant and grounded in historical case studies, the system facilitates trustworthy decision-making in high-pressure environments for nursing assistants. Although preliminary, simulations indicate the system’s potential to reduce cognitive load and enhance decision accuracy, with future clinical trials planned for compr

    Save this paper

  20. Current state and promise of user-centered design to harness explainable AI in clinical decision-support systems for patients with CNS tumors

    Eric W. Prince, D. M. Mirsky, T. Hankinson, et al. · 2025 · Frontiers in Radiology · 7 citations

    In neuro-oncology, MR imaging is crucial for obtaining detailed brain images to identify neoplasms, plan treatment, guide surgical intervention, and monitor the tumor's response. Recent AI advances in neuroimaging have promising applications in neuro-oncology, including guiding clinical decisions and improving patient management. However, the lack of clarity on how AI arrives at predictions has hindered its clinical translation. Explainable AI (XAI) methods aim to improve trustworthiness and informativeness, but their success depends on considering end-users’ (clinicians') specific context and preferences. User-Centered Design (UCD) prioritizes user needs in an iterative design process, invo

    Save this paper

  21. Explainable AI for Diabetes Risk Assessment: Enhancing Clinical Decision Support with Interpretable Machine Learning Models

    C. S. Reddy, Mohan Annamalai · 2025 · 2025 5th International Conference on Soft Computing for Security Applications (ICSCSA) · 7 citations

    This paper presents a comprehensive study on the application of Explainable Artificial Intelligence (XAI) for diabetes risk assessment, focusing on the interpretability of machine learning models in clinical decision support systems. While machine learning has demonstrated high accuracy in predicting diabetes, the lack of transparency in decision-making processes limits its adoption in healthcare. We employ interpretable models and model-agnostic explanation techniques such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) to enhance the understanding of predictive outcomes. Using real-world datasets including the PIMA Indians Diabetes Dataset

    Save this paper

  22. Leveraging ChatGPT and explainable AI for enhancing clinical decision support

    Radwa El Shawi, Leila Jamel · 2025 · Scientific Reports · 6 citations

    Large language models (LLMs) excel in many natural language processing tasks. However, their direct application to tabular, domain-specific clinical data remains challenging, as they lack innate mechanisms for reasoning over structured numerical features. This paper presents HealthAI-Prompt, a novel framework that systematically adapts LLMs for tabular clinical decision-making-specifically, predicting diabetes risk-through contextual prompts that combine detailed task descriptions with domain knowledge. Our domain knowledge integration leverages insights from high-performing machine learning models optimized via automated machine learning (AutoML) technique, together with local explanations

    Save this paper

  23. Enhancing AI Clinical Decision Support Trust: Design Workshop Insights from General Practitioners

    R. A. D. L. M. K. Ranwala, Andre Q. Andrade · 2025 · Studies in health technology and informatics · 5 citations

    Artificial Intelligence (AI) predictive models are increasingly integrated into Clinical Decision Support Systems (CDSS). However, real-world implementation is lagging due to a lack of trust and acceptance by healthcare providers. To investigate factors related to trust and acceptance of AI-based CDSS, we conducted a workshop with General Practitioners to explore user, model, and organizational factors that could affect trust and recommendation acceptance. The workshop discussions revealed that while explainability is crucial for understanding AI decisions, current explainable AI (XAI) visualizations appeared to provide limited value to clinicians without technical backgrounds. Participants

    Save this paper

Write your paper with these sources

Folio is the integrity-first research workspace: search 200M+ papers, save sources, and write with citations that format themselves. Free for students and researchers.

Start writing free →