Dr Matt Homer publishes paper on variation in examiner stringency
The paper is entitled "Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test".
In the paper, published in the highly rated journal Advances in Health Sciences Education, Dr Homer investigates sources of variation in marking, a well-known issue in the summative assessment of clinical and wider medical skills. The paper investigates differences in cut-scores produced by examiners in a test required for doctors wanting to come to the UK to practice medicine in the NHS. Statistical modelling is used to estimate different effects on cut-scores derived under a commonly used standard setting technique called the borderline regression method (BRM).
Dr Homer’s research aims to answer two key questions: “what is the extent of variation in station-level examiner cut-score stringency, in comparison to that of stations and examinations?” and “what impact on exam-level cut-scores does station-level examiner cut-score stringency have?”.
The paper finds that examiner variation in cut-scores is quite large, but that there are ways that this can be ameliorated. It concludes with an argument for a re-thinking of the concept of examiner stringency — moving away from only considering stringency in scores, towards also including stringency in the cut-scores that they set via their pattern of scoring under BRM.
