TY - JOUR
T1 - Rating the Rater
T2 - A Technique for Minimizing Leniency Bias in Residency Applications
AU - Seaward, James R.
AU - Carter, Lillian R.
AU - Nagarkar, Purushottam
AU - Zhang, Andrew Y.
N1 - Publisher Copyright:
© 2023 Lippincott Williams and Wilkins. All rights reserved.
PY - 2023/4/24
Y1 - 2023/4/24
N2 - Background: Each program in the highly competitive match for a surgical residency needs a way to review applicants effectively. Often this task is undertaken by individual faculty members, reviewing an applicant's file and assigning a score. Despite being asked to rate on a standardized scale, our program found that ratings of the same applicants varied dramatically, with certain faculty consistently scoring higher or lower than others. This is termed leniency bias, or the Hawk-Dove effect, and can affect who is invited to interview depending on which faculty are assigned to review an applicant's file. Methods: A technique to minimize leniency bias was developed and applied to this year's 222 applicants for our plastic surgery residency. The effect of the technique was evaluated by comparing variance between ratings of the same applicants by different faculty before and after our technique. Results: The median variance of ratings of the same applicants reduced from 0.68 before correction to 0.18 after correction, demonstrating better agreement between raters of the applicants' scores after our technique had been applied. This year, applying our technique affected whether or not 16 applicants (36% of interviewees) were invited for interview, including one applicant who matched to our program but who otherwise would not have been offered an interview. Conclusions: We present a simple but effective technique to minimize the leniency bias between raters of residency applicants. Our experience with this technique is presented together with instructions and Excel formulae for other programs to use.
AB - Background: Each program in the highly competitive match for a surgical residency needs a way to review applicants effectively. Often this task is undertaken by individual faculty members, reviewing an applicant's file and assigning a score. Despite being asked to rate on a standardized scale, our program found that ratings of the same applicants varied dramatically, with certain faculty consistently scoring higher or lower than others. This is termed leniency bias, or the Hawk-Dove effect, and can affect who is invited to interview depending on which faculty are assigned to review an applicant's file. Methods: A technique to minimize leniency bias was developed and applied to this year's 222 applicants for our plastic surgery residency. The effect of the technique was evaluated by comparing variance between ratings of the same applicants by different faculty before and after our technique. Results: The median variance of ratings of the same applicants reduced from 0.68 before correction to 0.18 after correction, demonstrating better agreement between raters of the applicants' scores after our technique had been applied. This year, applying our technique affected whether or not 16 applicants (36% of interviewees) were invited for interview, including one applicant who matched to our program but who otherwise would not have been offered an interview. Conclusions: We present a simple but effective technique to minimize the leniency bias between raters of residency applicants. Our experience with this technique is presented together with instructions and Excel formulae for other programs to use.
UR - http://www.scopus.com/inward/record.url?scp=85157984183&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85157984183&partnerID=8YFLogxK
U2 - 10.1097/GOX.0000000000004892
DO - 10.1097/GOX.0000000000004892
M3 - Article
C2 - 37101610
AN - SCOPUS:85157984183
SN - 2169-7574
VL - 11
SP - E4892
JO - Plastic and Reconstructive Surgery - Global Open
JF - Plastic and Reconstructive Surgery - Global Open
IS - 4
ER -