Explainable AI for Mental Health and Biomedical Decision Systems: A Comprehensive Review
DOI:
https://doi.org/10.59247/jahir.v4i2.376Keywords:
Explainable AI, External Validation, Mental Health, Clinical Translation, Human-Centered EvaluationAbstract
The application of Artificial Intelligence (AI) in mental health is experiencing rapid development, while algorithmic transparency and clinical translation readiness still face fundamental obstacles. This review synthesizes empirical findings related to the use of Explainable Artificial Intelligence (XAI) in mental health and biomedical decision systems, focusing on three evaluative aspects, namely explainability architecture, validation strength, and clinical integration. The literature search followed the PRISMA 2020 guidelines across five major databases for publications from 2020 to 2026 and yielded nine studies that met the inclusion criteria. The synthesis results show the dominance of post-hoc approaches, particularly SHAP, which are commonly applied to ensemble and boosting models, while intrinsic models and counterfactual approaches are still rarely used. The majority of studies rely on internal validation, while independent external validation and prospective application in real clinical workflows are relatively limited. User-based evaluation of explainability has also been understudied, with algorithmic transparency more often understood as technical feature attribution rather than as a verified mechanism in clinical decision-making. These findings indicate a persistent gap between methodological advances and the level of clinical translation maturity. Explainability has not been systematically integrated with robust validation designs or user-oriented evaluations. This review proposes a translation evaluation framework that combines technical and clinical dimensions to assess the readiness for XAI implementation more comprehensively. The development of XAI in mental health requires evaluation standardization, strengthened external validation, and prospective testing focused on clinical impact and user trust.
References
N. Koutsouleris, T. U. Hauser, V. Skvortsova, and M. De Choudhury, “From promise to practice: towards the realisation of AI-informed mental health care,” Lancet Digit. Health, vol. 4, no. 11, pp. e829–e840, Nov. 2022, doi: 10.1016/S2589-7500(22)00153-4.
G. Antoniou, E. Papadakis, and G. Baryannis, “Mental Health Diagnosis: A Case for Explainable Artificial Intelligence,” International Journal on Artificial Intelligence Tools, vol. 31, no. 03, May 2022, doi: 10.1142/S0218213022410032.
Y. Hong and Z. Xia, “AI-Driven Innovations in Psychological Assessment: Multimodal Data, Intelligent Analytics, and Ethical Challenges,” in Proceedings of the 2025 International Conference on Artificial Intelligence and Smart Manufacturing, New York, NY, USA: ACM, May 2025, pp. 854–859. doi: 10.1145/3756423.3756564.
D. W. Joyce, A. Kormilitzin, K. A. Smith, and A. Cipriani, “Explainable artificial intelligence for mental health through transparency and interpretability for understandability,” NPJ Digit. Med., vol. 6, no. 1, p. 6, Jan. 2023, doi: 10.1038/s41746-023-00751-9.
I. Levkovich, S. Shinan-Altman, and Z. Elyoseph, “Can large language models be sensitive to culture suicide risk assessment?,” J. Cult. Cogn. Sci., vol. 8, no. 3, pp. 275–287, Dec. 2024, doi: 10.1007/s41809-024-00151-9.
I. Ahmed, A. Brahmacharimayum, R. H. Ali, T. A. Khan, and M. O. Ahmad, “Explainable AI for Depression Detection and Severity Classification From Activity Data: Development and Evaluation Study of an Interpretable Framework,” JMIR Ment. Health, vol. 12, pp. e72038–e72038, Sep. 2025, doi: 10.2196/72038.
X. Li and M. Mahmoud, “Unlocking the Black Box: Concept-Based Modeling for Interpretable Affective Computing Applications,” in 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), IEEE, May 2024, pp. 1–10. doi: 10.1109/FG59268.2024.10581918.
A. Malhotra and R. Jindal, “XAI Transformer based Approach for Interpreting Depressed and Suicidal User Behavior on Online Social Networks,” Cogn. Syst. Res., vol. 84, p. 101186, Mar. 2024, doi: 10.1016/j.cogsys.2023.101186.
C.-C. Tsai et al., “Effect of Artificial Intelligence Helpfulness and Uncertainty on Cognitive Interactions with Pharmacists: Randomized Controlled Trial,” J. Med. Internet Res., vol. 27, p. e59946, Jan. 2025, doi: 10.2196/59946.
A. Gerdes, “The role of explainability in AI-supported medical decision-making,” Discover Artificial Intelligence, vol. 4, no. 1, p. 29, Apr. 2024, doi: 10.1007/s44163-024-00119-2.
H. Liu, “Interpretable Artificial Intelligence (XAI) in Biomedicine: Status, Challenges and Interdisciplinary Solutions,” Theoretical and Natural Science, vol. 117, no. 1, pp. 95–101, Jul. 2025, doi: 10.54254/2753-8818/2025.LD25347.
Suneel Pappala, M. Malyadri, K Venkata Naganjaneyulu, A. Prakashini, and Pasuladi Santosh, “Explainable AI for healthcare professionals: Advancing risk assessment, diagnostic precision, and ethical clinical interventions,” World Journal of Biology Pharmacy and Health Sciences, vol. 22, no. 1, pp. 446–453, Apr. 2025, doi: 10.30574/wjbphs.2025.22.1.0425.
A. F. Markus, J. A. Kors, and P. R. Rijnbeek, “The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies,” J. Biomed. Inform., vol. 113, p. 103655, Jan. 2021, doi: 10.1016/j.jbi.2020.103655.
H. W. Loh, C. P. Ooi, S. Seoni, P. D. Barua, F. Molinari, and U. R. Acharya, “Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011–2022),” Comput. Methods Programs Biomed., vol. 226, p. 107161, Nov. 2022, doi: 10.1016/j.cmpb.2022.107161.
J. Jung, H. Lee, H. Jung, and H. Kim, “Essential properties and explanation effectiveness of explainable artificial intelligence in healthcare: A systematic review,” Heliyon, vol. 9, no. 5, p. e16110, May 2023, doi: 10.1016/j.heliyon.2023.e16110.
T. Greenhalgh et al., “Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies,” J. Med. Internet Res., vol. 19, no. 11, p. e367, Nov. 2017, doi: 10.2196/jmir.8775.
X. Liu et al., “Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension,” Nat. Med., vol. 26, no. 9, pp. 1364–1374, Sep. 2020, doi: 10.1038/s41591-020-1034-x.
B. Vasey et al., “Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI,” Nat. Med., vol. 28, no. 5, pp. 924–933, May 2022, doi: 10.1038/s41591-022-01772-9.
G. S. Collins et al., “TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, p. e078378, Apr. 2024, doi: 10.1136/bmj-2023-078378.
K. Lekadir et al., “FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare,” BMJ, vol. 388, p. e081554, Feb. 2025, doi: 10.1136/bmj-2024-081554.
A. H. van der Vegt, I. A. Scott, K. Dermawan, R. J. Schnetler, V. R. Kalke, and P. J. Lane, “Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework,” Journal of the American Medical Informatics Association, vol. 30, no. 9, pp. 1503–1515, Aug. 2023, doi: 10.1093/jamia/ocad088.
N. R. Haddaway, M. J. Page, C. C. Pritchard, and L. A. McGuinness, “PRISMA2020: An R package and Shiny app for producingPRISMA 2020‐compliant flow diagrams, with interactivityfor optimised digital transparency and Open Synthesis,” Campbell Systematic Reviews, vol. 18, no. 2, Jun. 2022, doi: 10.1002/cl2.1230.
S. Cruz Rivera et al., “Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension,” Nat. Med., vol. 26, no. 9, pp. 1351–1363, Sep. 2020, doi: 10.1038/s41591-020-1037-7.
X. Liu et al., “Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension,” Nat. Med., vol. 26, no. 9, pp. 1364–1374, Sep. 2020, doi: 10.1038/s41591-020-1034-x.
B. Vasey et al., “Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI,” Nat. Med., vol. 28, no. 5, pp. 924–933, May 2022, doi: 10.1038/s41591-022-01772-9.
R. F. Wolff et al., “PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies,” Ann. Intern. Med., vol. 170, no. 1, pp. 51–58, Jan. 2019, doi: 10.7326/M18-1376.
S. Banerjee, P. Lio, P. B. Jones, and R. N. Cardinal, “A class-contrastive human-interpretable machine learning approach to predict mortality in severe mental illness,” NPJ Schizophr., vol. 7, no. 1, Dec. 2021, doi: 10.1038/s41537-021-00191-y.
M. Y. Chun et al., “Prediction of conversion to dementia using interpretable machine learning in patients with amnestic mild cognitive impairment,” Front. Aging Neurosci., vol. 14, Aug. 2022, doi: 10.3389/fnagi.2022.898940.
A. Caetano et al., “Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-based System for Suicide Prevention”, doi: 10.1109/ACCESS.2024.0429000.
F. de Arriba-Pérez and S. García-Méndez, “Leveraging large language models through natural language processing to provide interpretable machine learning predictions of mental deterioration in real time,” Arab. J. Sci. Eng., vol. 50, no. 15, pp. 11577–11591, Aug. 2025, doi: 10.1007/s13369-024-09508-2.
G. Wang et al., “Investigating Protective and Risk Factors and Predictive Insights for Aboriginal Perinatal Mental Health: Explainable Artificial Intelligence Approach,” J. Med. Internet Res., vol. 27, no. 1, 2025, doi: 10.2196/68030.
W. Zuo and X. Yang, “Network-based predictive models for artificial intelligence: an interpretable application of machine learning techniques in the assessment of depression in stroke patients,” BMC Geriatr., vol. 25, no. 1, Dec. 2025, doi: 10.1186/s12877-025-05837-5.
J. Chen, Y. Lin, R. Hu, and C. Hu, “Prediction model for depression risk in middle-aged and elderly patients with metabolic syndrome: a nomogram and interpretable machine learning approach based on CHARLS,” BMC Psychiatry, vol. 25, no. 1, Dec. 2025, doi: 10.1186/s12888-025-07434-7.
R. Zhang et al., “Identification and validation of an explainable machine learning model for vascular depression diagnosis in the older adults: a multicenter cohort study,” BMC Med., vol. 23, no. 1, Dec. 2025, doi: 10.1186/s12916-025-04283-9.
A. Ebrahimi et al., “Explainable AI models for identifying anxiety and distress in cardiac patients with ICDs,” BMC Med. Inform. Decis. Mak., Dec. 2025, doi: 10.1186/s12911-025-03315-x.
M. Alkan, I. Zakariyya, S. Leighton, K. B. Sivangi, C. Anagnostopoulos, and F. Deligianni, “Artificial Intelligence-Driven Clinical Decision Support Systems,” Feb. 2025.
M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?,’” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA: ACM, Aug. 2016, pp. 1135–1144. doi: 10.1145/2939672.2939778.
C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat. Mach. Intell., vol. 1, no. 5, pp. 206–215, May 2019, doi: 10.1038/s42256-019-0048-x.
S. Reddy, “Explainability and artificial intelligence in medicine,” Lancet Digit. Health, vol. 4, no. 4, pp. e214–e215, Apr. 2022, doi: 10.1016/S2589-7500(22)00029-2.
J. M. Bauer and M. Michalowski, “Human-centered explainability evaluation in clinical decision-making: a critical review of the literature,” Journal of the American Medical Informatics Association, vol. 32, no. 9, pp. 1477–1484, Sep. 2025, doi: 10.1093/jamia/ocaf110.
W. Alharbi and A. A. Alfayez, “Explainable artificial intelligence in pancreatic cancer prediction: from transparency to clinical decision-making,” Front. Oncol., vol. 15, Dec. 2025, doi: 10.3389/fonc.2025.1720039.
H. Eshkiki, F. Tanhaei, F. Caraffini, and B. Mora, “A Survey of the Application of Explainable Artificial Intelligence in Biomedical Informatics,” Applied Sciences, vol. 15, no. 24, p. 12934, Dec. 2025, doi: 10.3390/app152412934.
P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, “AI in health and medicine,” Nat. Med., vol. 28, no. 1, pp. 31–38, Jan. 2022, doi: 10.1038/s41591-021-01614-0.
A. Nicolson, E. Bradburn, Y. Gal, A. T. Papageorghiou, and J. A. Noble, “The human factor in explainable artificial intelligence: clinician variability in trust, reliance, and performance,” NPJ Digit. Med., vol. 8, no. 1, p. 658, Nov. 2025, doi: 10.1038/s41746-025-02023-0.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Pramesti Dewi, Purwono Purwono, Annastasya Nabila Elsa Wulandari, Indah Trivilia
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All articles published in the JAHIR Journal are licensed under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. This license grants the following permissions and obligations:
1. Permitted Uses:
- Sharing – You may copy and redistribute the material in any medium or format.
- Adaptation – You may remix, transform, and build upon the material for any purpose, including commercial use.
2. Conditions of Use:
- Attribution – You must give appropriate credit to the original author(s), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in a way that suggests the licensor endorses you or your use.
- ShareAlike – If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original (CC BY-SA 4.0).
- No Additional Restrictions – You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
3. Disclaimer:
- The JAHIR Journal and the authors are not responsible for any modifications, interpretations, or derivative works made by third parties using the published content.
- This license does not affect the ownership of copyrights, and authors retain full rights to their work.
For further details, please refer to the official Creative Commons Attribution-ShareAlike 4.0 International License.
