Critical Engagement: The Value of Transparency of AI in Healthcare

Why is transparency important for the use of AI in healthcare? Responses to this question typically claim that transparency is something owed to the patient – because it is a condition for informed consent, legitimacy, accountability to the patient, etc. In this paper, we draw attention to why transparency can be valuable for medical practitioners. We claim that transparent AI models facilitate critical engagement by medical practitioners with AI models that they use. That is, they enable practitioners to assess why AI models make the recommendations they do, think about how those reasons affect their own beliefs and judgments, and make reasoned decisions about whether to maintain or change their own judgments. Via this process, AI models can help medical practitioners to improve their practice in a distinctly valuable way. In turn, this benefits both medical practitioners and their patients. This conclusion has important implications for AI design in healthcare: if AI models are to be used in healthcare, they should be designed in ways which allow medical practitioners to understand how the models arrive at their recommendations, and engage with them critically.