A Redesign of the ACGME Resident/Fellow Survey Through Expert Reviews and Cognitive Interviews

Background Since 2003, the Accreditation Council for Graduate Medical Education (ACGME) has surveyed residents and fellows in its accredited programs. The Resident/Fellow Survey is a screening and compliance tool that programs can also use for continuous quality improvement. However, stakeholders have reported potential problems with the survey’s overall quality and credibility. Objective To redesign the 2006 Resident/Fellow Survey using expert reviews and cognitive interviews. Methods In 2018-2019, the ACGME redesigned the Resident/Fellow Survey using an iterative validation process: expert reviews (evidence based on content) and cognitive interviews (evidence based on response processes). Expert reviews were conducted by a survey design firm and an ACGME Task Force; cognitive interviews were conducted with a diverse set of 27 residents and fellows. Results Expert reviews resulted in 21 new survey items to address the ACGME’s updated accreditation requirements; these reviews also led to improvements that align the survey items with evidence-informed standards. After these changes were made, cognitive interviews resulted in additional revisions to sensitive items, item order, and response option order, labels, and conceptual spacing. In all, cognitive interviews led to 11 item deletions and other improvements designed to reduce response error. Conclusions Expert reviews and cognitive interviews were used to redesign the Resident/Fellow Survey. The content of the redesigned survey aligns with the updated accreditation requirements and items are written in accordance with evidence-informed standards. Additionally, cognitive interviews resulted in revisions to the survey that seem to improve respondents’ comprehension and willingness to respond to individual survey items.