TY - JOUR
T1 - ChatGPT and the clinical informatics board examination
T2 - The end of unproctored maintenance of certification-
AU - Kumah-Crystal, Yaa
AU - Mankowitz, Scott
AU - Embi, Peter
AU - Lehmann, Christoph U.
N1 - Publisher Copyright:
© 2023 The Author(s). Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved.
PY - 2023/9/1
Y1 - 2023/9/1
N2 - We aimed to assess ChatGPT's performance on the Clinical Informatics Board Examination and to discuss the implications of large language models (LLMs) for board certification and maintenance. We tested ChatGPT using 260 multiple-choice questions from Mankowitz's Clinical Informatics Board Review book, omitting 6 image-dependent questions. ChatGPT answered 190 (74%) of 254 eligible questions correctly. While performance varied across the Clinical Informatics Core Content Areas, differences were not statistically significant. ChatGPT's performance raises concerns about the potential misuse in medical certification and the validity of knowledge assessment exams. Since ChatGPT is able to answer multiple-choice questions accurately, permitting candidates to use artificial intelligence (AI) systems for exams will compromise the credibility and validity of at-home assessments and undermine public trust. The advent of AI and LLMs threatens to upend existing processes of board certification and maintenance and necessitates new approaches to the evaluation of proficiency in medical education.
AB - We aimed to assess ChatGPT's performance on the Clinical Informatics Board Examination and to discuss the implications of large language models (LLMs) for board certification and maintenance. We tested ChatGPT using 260 multiple-choice questions from Mankowitz's Clinical Informatics Board Review book, omitting 6 image-dependent questions. ChatGPT answered 190 (74%) of 254 eligible questions correctly. While performance varied across the Clinical Informatics Core Content Areas, differences were not statistically significant. ChatGPT's performance raises concerns about the potential misuse in medical certification and the validity of knowledge assessment exams. Since ChatGPT is able to answer multiple-choice questions accurately, permitting candidates to use artificial intelligence (AI) systems for exams will compromise the credibility and validity of at-home assessments and undermine public trust. The advent of AI and LLMs threatens to upend existing processes of board certification and maintenance and necessitates new approaches to the evaluation of proficiency in medical education.
KW - ChatGPT
KW - Clinical Informatics Board Examination
KW - artificial intelligence
KW - large language models
KW - medical education
UR - http://www.scopus.com/inward/record.url?scp=85165993930&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85165993930&partnerID=8YFLogxK
U2 - 10.1093/jamia/ocad104
DO - 10.1093/jamia/ocad104
M3 - Article
C2 - 37335851
AN - SCOPUS:85165993930
SN - 1067-5027
VL - 30
SP - 1558
EP - 1560
JO - Journal of the American Medical Informatics Association
JF - Journal of the American Medical Informatics Association
IS - 9
ER -