The American Board of Radiology (ABR) has been using the oral examination as part of its evaluation process since it began in 1934. The oral exam is seen to provide a number of advantages:

• Provide direct personal contact with candidates
• Provide a way to evaluate how candidates communicate with peers
• Provide opportunity to the candidate to explain how they would mitigate
• Provide flexibility in moving from candidate's strong areas to weak areas and vice-versa
• Require the candidate to formulate his/her own replies without cues
• Provide opportunity to question the candidate about how he/she arrived at an answer

There are some obvious weaknesses in the oral exam format:

• Lack of standardization (everyone gets a slightly different test).
• Reproducibility of results are difficult to establish and assess.
• Perceived to enable favoritism toward certain candidates.
• Potentially influenced by irrelevant factors.

The ABR believes in the strength of the oral exam especially since there are substantial differences in the training of oral exam candidates.

This newsletter article will discuss the many ways the ABR insures the fairness, reliability, and validity of the oral exam. This process begins with the organization of the exam. Each medical physics specialty (DMP, NMP and TMP) has an oral exam committee that writes new oral exam questions and reviews existing questions in the oral exam question bank each year to ensure that the available oral exam questions are both relevant and current. They also review the supplementary material given to each examiner. They then assemble the actual oral exam question sets. Each of the three specialty exams has five categories with five questions in each category. The questions for each category are selected to fill in an established grid. The questions in each category are distributed among the examiners. This means a candidate gets one question in each category from each examiner. The five examiners give five differing perspectives on the candidate's performance in each category.

Another way to ensure fairness is to recruit diverse examiners with respect to gender, degree (MS and PhD), and practice environment (academic and private practice).

All examiners are ABR certified and participate in ABR MOC. Each examiner goes through training to ensure that they understand the exam procedures. More importantly, they are taught to treat each candidate in the same fashion and to avoid comments on how the candidate is doing. New examiners must observe experienced examiners before they are allowed to examine. Each examiner is encouraged to provide comments on each question.

At the end of each examination period the group of examiners, called a panel, gets together to check the scores and verify the final assessment (pass, condition, or fail).

Each examiner is observed by an ABR Trustee, ABR Governor, or Panel Chair several times to ensure they are adhering to ABR policies about how exams should be conducted. If any discrepancies are noted, examiners are counseled as soon as practicable. There are also additional ABR staff observers that observe the exam processes to ensure that administration of exams is uniform. The ABR is always accepting of examinee comments (oral or written) on the conduct of the exam. Each comment is thoroughly reviewed by the ABR.

From a statistical point of view, we review each examiner's scores to determine their average score and compare that to the other examiners. After many years of doing this we know that it is almost unheard of that an examiner is significantly different than their colleagues. If that were to happen the trustees would consider the matter and either counsel the individual or remove them from the examiner pool. As part of this review we also check that an examiner does not give an unusually high percentage of high or low scores.

We also review the performance of the panels to ensure there are no panels that behave significantly differently. Experience has shown that the average scores for each panel are very close and statistically identical.

We review the performance of each question and compare the average score of each category to be sure there are no anomalies.

The ABR believes the oral exam is of the utmost importance and does everything it can to be sure it is effective, properly administered, and fair.