Predictors of performance on the National Board of Chiropractic Examiners Parts I and II*

INTRODUCTION

The National Board of Chiropractic Examiners (NBCE) administers licensing exams for chiropractic students.¹ The NBCE process includes 4 main areas of testing (Parts I, II, III, and IV) and physiotherapy (PT) that are offered internationally. Chiropractic schools are required by the Council of Chiropractic Education (CCE), the accrediting body, to publish Part I, II, III, IV, and PT passing rates as a measure of accountability.² Canadian Memorial Chiropractic College found that the best predictor for success with the Canadian Chiropractic Examining Board (CCEB) was prior academic performance and 2nd-year grade point average (GPA).³ Wolfenberger⁴ found physiology I and II grades were highly correlated to all 6 NBCE sections, and gross anatomy I grades were highly correlated to all but microbiology and public health grades. Cunningham et al⁵ found a correlation between students' prior education and GPA during the program. Thus, course grades (ie, GPA) proved to be a reliable indicator of NBCE scores. There is a lack of research pertaining to Part II performance.

Related health fields have also invested some significant research in looking at the indicators for their certification exams. In nursing education, there is research on the predictors of success on the National Council Licensure Examination for Registered Nurses (NCLEX-RN). Nursing students' success can be predicted by factors such as “admission, cumulative and nursing grade point average and grades in specific courses such as certain science and nursing courses.”⁶ Health Education Systems Incorporated (HESI) develops diagnostics tests for nursing and allied health markets.⁶ These tests are used in many nursing schools to predict licensing outcomes. The mid-curricular HESI exam predicts academic success and passing rates at 81% and 95%, respectively.⁷ The HESI Admission Assessment taken by new nursing students was found to be almost 90% predictive of course grades in the associate degree program and 50% for bachelor degree–seeking students.⁸ Harding⁹ found those students who were not successful had a score below 800 on the mid-curricular HESI exam. This, in turn, is a good predictor of at-risk students early in their curricular progression.⁹ Studies have also shown entering GPA, academic success in the program,^10,11 and the HESI exit exam provide predictive evidence in nursing student success.¹² The American Board of Anesthesiology/American Society of Anesthesiologists uses the In-Training Examination (ITE) predictor test for medical residents, and it accounts for approximately half of the variance of the Part I of the licensure exam.¹³ Thus, in addition to course grades, other internal assessments such as admissions tests, preliminary exams, and practice exams have been indicators for board examination scores.

With the NBCE being such a large measure of accountability and perceived success, Logan College of Chiropractic implemented 4 strategies to help students succeed on NBCE exams. First, beyond the requirements NBCE has for sitting for Part I of the test, the institution has implemented a basic science GPA requirement of 2.0. The courses used in this calculation are the core set of courses identified with subject areas of NBCE Part I including general anatomy, spinal anatomy, physiology, chemistry, pathology, and microbiology. Second, in 2001, the institution contracted a board preparation company and required all students taking Part I, II, and III to attend at the expense of the university. Third, in 2006, the institution implemented a National Board Part I practice test given to any student preparing to sit for Part I. Last, the institution implemented a Spinal Camp to help students specifically on the spinal anatomy section of the boards.

The institution sought to identify predictors of success on the NBCE examinations. In looking for a successful predictor for Part I and Part II of the NBCE, the institution examined entering GPA, pre-chiropractic GPA, Part I GPA, Part II GPA, performance on the NBCE Part I practice test (administered and developed by the institution), and individual chiropractic course grades. Two questions arose that targeted the purpose of this research: (1) What are the predictors of success for the National Boards Part I and Part II? (2) Is the NBCE Part I practice test developed by the institution's faculty a valid measure of NBCE Part I performance?

METHODS

Two validity studies were conducted to examine the criterion validity of the college's assessments for NBCE Part I and NBCE Part II success in order to examine the strength of various predictors of Part I and II NBCE performance. The study was deemed exempt by the Logan College of Chiropractic institutional review board. In both studies, archival data were collected on students' incoming GPA, courses related to Part I of the NBCE (Part I GPA), Part I NBCE scores, Part I practice test scores, and other demographics. A variety of regression-based statistical analyses were used to analyze the data. The primary method of analysis was hierarchical linear regression (HLR). HLR allows for an incremental examination of predictors. In comparison to standard multiple regression, HLR allows the analyst to examine whether adding additional predictors beyond a starting variable yields any additional significant and useful information to the regression model in a prespecified stepwise manner. This procedure yields a regression equation that is efficient in that it removes statistical redundancies. Another statistical procedure that is used in these studies is the bootstrapping method of testing mediation effects, as designed by Preacher and Hayes.¹⁴ This method has become the gold standard for conducting mediational analyses because it improves upon previous tests of mediation by providing a statistical test of the indirect effect as well as a 95% confidence interval (CI) for such effects. Mediation allows researchers to examine the process by which a 3rd variable can intervene between a proposed cause variable (ie, independent) and effect variable (ie, dependent). One final statistical method employed in these 2 studies was the usage of effect sizes. Statistical significance is based in large part on sample size (ie, significance thresholds decrease as the degrees of freedom increase). Thus, with a large sample (ie, over 200) virtually any effect is likely to be statistically significant. Thus, it is advisable to report effect sizes as better indicators of the variance within the dependent measure that is accounted for by the predictors. Since our sample (over 500 participants) is very large, virtually all of our statistical analyses were significant. Hence, we chose to report effect sizes (as percentages) in addition to significance levels to provide a more accurate and fair assessment of the effect of the tested variables.

Study 1 - Criterion Validity Part I Study Methods

The 1st study sought to examine the criterion validity of assessments within the chiropractic program and the NBCE Part I exam. Archival analyses of stored demographics, assessments, and NBCE scores were analyzed. The analyses were focused around 3 main questions: (1) Do entrance GPAs predict Part I NBCE scores? (2) Do Part I GPAs uniquely predict Part I NBCE scores when controlling for entrance GPA? (3) Do practice exam scores uniquely predict Part I NBCE scores when controlling for GPA?

Archived data for 528 chiropractic students who took Part I of the NBCE from spring 2008 to fall 2010 were collected. Archived data on students demographics, entrance GPA, school status, course grades, Spinal Camp attendance, practice exam scores, and Part I NBCE scores were extracted from different databases and matched on at least 2 criteria (name and student identification number).

Entrance GPA was determined based upon the cumulative GPA of courses that students took prior to entrance to the chiropractic program. Part I GPA was based upon the course grades of those courses identified as prerequisites and sufficient/necessary determinants of success on the Part I NBCE as seen in Figure 1. Cumulative GPA was also used. Part I of the NBCE covers 6 basic science subjects—general anatomy, spinal anatomy, physiology, chemistry, pathology, and microbiology. Both the overall score and subscores for each of the subtests, as reported by the NBCE were used. Finally, several demographic variables were obtained from student records including age, sex, whether the student attended a Spinal Camp event, and whether the student repeated any courses at the Part I level.

View Figure

• Download as Powerpoint
• Download Figure

Study 2 - Criterion Validity Part II Study Methods

The procedure and methods for the 2nd study were identical to the procedure and methods in the 1st study. A statistically based criterion validation process was utilized to examine the validity of pre-chiropractic GPA, in-program chiropractic GPA (Part I and Part II), the institutional practice exam, Part I NBCE scores, and a variety of other variables (demographics, Spinal Camp attendance, repeated courses, and performance) on Part II of the NBCE. Part II GPA was based upon an a priori selection by the faculty of 45 courses that were designed to address the core areas of the Part GPA NBCE as seen in Figure 2.

View Figure

• Download as Powerpoint
• Download Figure

RESULTS

Study 1 - Part I Criterion Validity

Overall Part I NBCE Scores

In order to examine the relative impact and contributions of entrance GPA, Part I GPA, and practice exam scores to the prediction of Part I NBCE scores, HLR was performed. The correlation matrix for all variables is shown in Table 1.

Table 1. Correlation Matrix for the National Board of Chiropractic Examiners Part I Scores

View Fullscreen

Entrance GPA

Entrance GPA scores were entered into the 1st step of the regression equation. Entrance GPA scores significantly predicted overall Part I NBCE exam scores (β = .36, t[521] = 8.90, p < .001). The strength of this prediction is reflected in the fact that entrance GPAs were able to explain approximately 13% of the differences in the distribution of overall NBCE exam scores (Fig. 3).

View Figure

• Download as Powerpoint
• Download Figure

Part I GPA

Part I GPA scores were entered into the regression equation in the 2nd step. When added to the equation, Part I GPA scores proved to be significant predictors of NBCE scores (β = 120.56, t[521] = 24.60, p < .001). The strength of this prediction is evidenced in the fact that Part I GPA could be used to explain approximately 60% of the differences in NBCE exam scores. However, the relationship between entrance GPAs and NBCE scores appeared to have been mediated by the addition of Part I GPA. That is, when Part I GPA (basic science course grades) scores were added to the equation, entrance GPA was no longer a significant predictor of NBCE exam scores (β = .01, t[521] = .28, p = .78). The mediational role of Part I GPA with regard to the link between entrance GPA and NBCE scores was confirmed in an independent bootstrapping analysis that has become standard practice for testing mediational effects.¹⁴ We found evidence of complete mediation (indirect effect = 82.44, z = 13.22, p < .001, BCa 95% CI = [70.22, 94.67]). Thus, when Part I GPA was taken into consideration, entrance GPAs lost all predictive power for NBCE scores.

Practice Exams

Practice exam scores were added to the equation in the 3rd step. They proved to be significant predictors of NBCE exam scores, (β = .36, t[521] = 8.90, p < .001). The strength of this prediction is reflected in the fact that practice exam scores were able to explain 12% of the differences between NBCE exam scores.

Overall, the regression model demonstrated that including both Part I GPA and practice exam scores, the total model was able to account for 72% of the differences in Part I NBCE exam scores (Fig. 3). While entrance GPAs no longer appeared necessary as indicators of NBCE exam scores, they nonetheless remained useful as significant predictors of Part I GPA (β = .69, t[758] = 14.61, p < .001). Entrance GPAs explained approximately 22% of the differences in Part I GPA. Thus, entrance GPAs do serve as predictors of how well students will perform in the classroom.

Study 2 - Part II Criterion Validity

Entrance GPA, Part I GPA, Part I NBCE scores, and Part II GPA were regressed on Part II NBCE scores in a hierarchical multiple regression analysis. As in the 1st study, entrance GPA was not a significant predictor of Part II NBCE scores. However, Part I GPA and Part I NBCE combined accounted for a significant amount of variance (68%) in overall Part II NBCE performance, (ΔR² = .68, p < .001). Part II GPA accounted for an additional 7% variance in overall Part II NBCE performance, (ΔR² = .07, p < .001). The overall model with Part I GPA, Part I NBCE score, and Part II GPA as predictors explained a combined 75% of the variance in overall Part II NBCE performance (Fig. 4). However, the primary determinant of performance on Part II of the NBCE was Part I GPA. The final model suggested that Part I GPA explained 52%, Part I NBCE explained 18%, and Part II GPA explained 6% of the variance in Part II NBCE scores.

View Figure

• Download as Powerpoint
• Download Figure

DISCUSSION

When no other predictors were taken into account, the study found that entrance GPA was a significant predictor as Cunningham et al found⁵; however, when additional variables such as program GPA were included, entrance GPA was no longer a significant predictor. The study did find that entrance GPAs were predictors for how students performed in the classroom, given that they account for approximately 22% of the differences in Part I GPA. Results indicated that course GPA and practice exam scores alone determined 72% of the differences within students' Part I NBCE scores. These results coincide with those at nursing schools and at Canadian Memorial Chiropractic College where in-program success was a predictor of success on the licensure examination.^3,10,11 This is also in line with the results of the American Board of Anesthesiology (ABA)/American Society of Anesthesiologists, In-Training Examination predictor test, which accounts for approximately half of the variance of Part I of the ABA licensure exam.¹³

Whereas entrance GPA was initially a significant predictor, it became a nonsignificant predictor of NBCE Part I scores when Part I course GPAs and practice exams were added to the prediction model. Thus, entrance GPA ultimately provided very little value in deciding success on the NBCE test. This is contrary to what was found by Cunningham et al.⁵ However, it should be mentioned that these findings do not suggest that entrance GPAs should be abandoned altogether. The results demonstrate that although Part I GPA supplanted entrance GPA as a predictor of the NBCE, entrance GPA was nonetheless a direct predictor of Part I GPA. Thus, entrance GPAs might serve a role in determining how well students actually do in the classroom. Nevertheless, these results suggest that regardless of the different levels of preparedness of the students entering our college, it was ultimately their education within the basic science courses that determined their success on the NBCE test.

The results of study 2 indicate that success on Part II of the NBCE hinges on Part I GPA and Part I NBCE performance, with both of these variables accounting for 68% of the variance in scores. This could explain why there was a lack of research into Part II, given the high correlation between Part I and Part II.

CONCLUSIONS

This collection of studies illustrates that it is possible to validate course performance and other criterion (eg, practice exams, programs) against NBCE scores. This illustrates an advantage within chiropractic education relative to other forms of graduate and professional training that do not have external certification exams or simply do not provide them as frequently. Furthermore, this set of studies demonstrated that it was ultimately the students' grades in their chiropractic education rather than their entrance GPA that affected NBCE scores. However, this work does not suggest that there is no value in having entrance requirements. For example, we did find that entrance scores were predictors of course grades, suggesting that initial levels of preparation did affect classroom performance.