Letters to the Editor

To The Editor:

Barr and Burke present an intriguing study on the use of confidence-based marking in a laboratory setting, and the correlation between multiple choice questions (MCQ) scores and confidence-based marking scores.¹ They conclude that learners are able to self-assess their knowledge accurately, but it is worth exploring to what degree their conclusion is based on the evidence presented.

First of all, the MCQ assessments that were completed each week had not been validated. If the MCQ assessments had been shown to be valid and reliable, and to have a positive effect on learners' behaviors, and the confidence-based marking scores correlated with such validated MCQ assessments, then we would place much more weight upon such a correlation.² However, the authors do not report any prior validation of the MCQ assessments.

Secondly the sample MCQs that the authors list at the end of the study are questionable. The first two sample questions are almost purely tests of academic anatomic knowledge. The last two questions appear to turn best practice in MCQ construction on its head. Good MCQs generally should start with a clinical scenario and then a question, and then a number of possible answers. However, the last two sample questions start with an anatomic lesion and ask the candidate to link this lesion with a clinical scenario. This is counterintuitive – it is the opposite of the typical clinical encounter – and immediately calls into question the validity of these questions.³

There is one final strategic point to make. Even if we put these doubts aside, and assume that the assessment was valid and reliable, what should we do with the positive results discovered? Are we so convinced that we can conclude that MCQs are not necessary, and that we can dispense with them and rely solely on learner's confidence scores? I suspect that the answer for most of us is probably not – and if the majority of readers were to agree with this, then it calls into question the core purpose of the research – insofar as it is unlikely to change educational or assessment practice.

To the Editor:

The reliability and validity of the multiple choice questions would impact the interpretation of results on confidence-based marking. Calculations of point-biserial correlation coefficients for every multiple choice item provide an index of the reliability and validity of the multiple choice questions within a sample. More than 90% of point-biserial correlation coefficients for the multiple choice questions from each quiz administered were greater than .30 (Table 1). A matrix of the distributions of point-biserial correlation coefficients by the percentage of students answering correctly from each quiz administered revealed that 80% of multiple choice questions were acceptable, 17% may require review and modifications, and 3% should be eliminated (Table 2). The distributions of the Kuder-Richardson formula 20 (KR₂₀) coefficients from the quizzes administered over the four trimesters met the criterion of adequate test reliability and validity (Table 3). In summary, the reliability and validity of the multiple choice questions used to measure student performance in the neuroscience laboratory course were adequate.

The face validity of the multiple choice questions to assess factual recall, and application of core neuroanatomic topics regarding structure and function was adequate for this first trimester introductory laboratory course. This course directs its efforts primarily at the knowledge and comprehension level of Bloom's and Krathwohl's taxonomies. Therefore, if a student can take either a patient presentation and identify where the lesion might be in the neuroaxis, or be given the level of the lesion and explain what the patient presentation could be, they begin to develop confidence in their knowledge and reasoning skills that reinforces structure/function comprehension. As students progress through the 10-trimester curriculum, they receive training on clinical scenarios that increase in complexity with appropriate correlation to their level of educational training. At these time points in the curriculum, tests of learning objectives for clinical scenarios are presented within the context of clinical encounters.

The authors did not generalize their data findings beyond the current descriptive research to imply any change in educational or assessment practices. The authors adequately presented their results on confidence-based marking to allow readers to assess internal validity of the current descriptive research. With the information presented in the Letter to the Editor and the authors' response, readers have insights on the impact of reliability and validity of the multiple choice questions on confidence-based marking.