‘Going beyond hawks and doves’: new paper suggests ways to enhance high stakes exam and improve fairness for candidates

Dr Matt Homer has a new paper in Medical Teacher called ‘Going beyond hawks and doves – Measuring degrees of examiner misalignment in OSCEs’.

Dr Homer’s paper contributes to the theoretical understanding of how examiners vary in their scoring of candidate performance in OSCEs (Objective Structured Clinical Examinations).

The paper also shows how a new empirical measure of examiner misalignment in scoring can be used over time to improve OSCE validity and fairness to candidates. This work comes out of Dr Homer’s consultancy with the General Medical Council on their exam (PLAB2) for international medical graduates who want to come and work in the NHS. It is one of several in this important area that Matt has led on or contributed to.

Minimising examiner differences in scoring in OSCEs is key in supporting the validity of the assessment outcomes. This is particularly true for common OSCE designs where the same station is administered across parallel circuits, with examiners nested within these.

However, the common classification of extreme examiners as ‘hawks’ or ‘doves’ can be overly simplistic. Rather, it is the difference in patterns of scoring across circuits that better indicates poor levels of agreement between examiners that can unfairly advantage particular groups of candidates in comparison with others in different circuits.

Dr Matt Homer is Associate Professor at the School of Education. His research interests include medical education assessment and post-16 mathematics education.