Restructuring of an evidence-based practice curriculum and assessment with structural mapping by course outcome verb

INTRODUCTION

This paper describes how we restructured an evidence-based clinical practice (EBCP) subcurriculum within our overall doctor of chiropractic (DC) training program. Our program is an integrated curriculum. In this curriculum, some courses are a combination of related disciplines instead of covering a discrete discipline. For example, anatomy, physiology, diagnosis, patient management, and philosophy are integrated within most courses for the entire 13 quarter-based curriculum. Our program also has subcurricula, which are main themes (eg, communication or diagnosis) that are weaved throughout the integrated curriculum. Sometimes these themes combine into 1 outcome such as, “The student is able to communicate the diagnosis and treatment to a mock patient using EBCP terms such as absolute risk ratios in a report of findings.”

At the time of the revision of the EBCP subcurriculum it was not fully meshed within the integrated curriculum. The EBCP subcurriculum was limited to the outcomes in a first quarter course. Some professors who were early adopters of EBCP covered related topics in other courses; although, there was not an accounting of those EBCP outcomes. Clinic faculty members had started to integrate EBCP in their teaching as well, yet there was not a known curricular structure where the EBCP topics were intentionally integrated in the curriculum. Given the integrated curriculum, it was a challenge to match discrete subjects with EBCP topics. For example, all classes about diagnostic procedures (eg, physical examination) should have included diagnostic validity (likelihood ratios) and reliability (κ, interclass correlation coefficients) in related course outcomes.

We had to develop a strategy that would provide a basic structural goal, reveal where professors were already teaching EBCP topics, determine where we needed volunteers to teach the topics, and solicit volunteers to teach the material in courses that related to the topics and were not already laden with content. In addition, we had to provide resources for those who were unfamiliar with EBCP topics. The challenges of the integrated curriculum resulted in what we believe is an innovative model for the incorporation of EBCP outcome measures into a health care provider curriculum.

Different outcome verbs carry different “depth of processing” levels. For example, “After this course, the student will be able to define evidence-based clinical practice.” The verb “define” means that the student will memorize the definition, although the outcome does not require critical analysis of the definition. The various depth of processing levels were developed by Benjamin Bloom around 1956. Mary Allen listed the following ordinal Bloom's verb levels: knowledge, comprehension, application, analysis, synthesis, and evaluation and included a table of “relevant verbs” that are related to these levels.¹ The verbs can be used to take a student from memorization of a concept, to application, to critical analysis throughout an entire curriculum. The verbs can also be used to encourage a teacher to assess deeper levels of processing. Thus, students must master the content in order to apply it to their patient's benefit and further the development of the profession through critical analysis. The ability for professionals to question established protocols and taboos enables them to think through nonscripted situations in practice and improve patient management skills.

The background of EBCP has been described by many authors in the past and should be a familiar topic to chiropractic educators. Therefore, we defer to other sources for details on EBCP.^2,3 Basic implementation of EBCP can be achieved by using the 5 As (Ask, Acquire/Access, Appraise, Apply, Assess) model adapted from Sackett.⁴ Basically, a clinician asks a clinically relevant question. The main terms in that question are used in a database search to acquire related articles. The clinician appraises the article for quality and applicability to the clinical situation. If applicable, the article's recommendations are applied to the patient and then an assessment is made to see if the intervention helped. The applicability of mentioning this model in this paper is 3-fold: (1) because the 5 As parallel critical thinking constructs,⁵ then using the process demonstrates critical thinking in student learning with respect to patient care; (2) helps to delineate topics for outcome measures for an evidence-informed curriculum; and (3) provides a descriptive overview for those who are new to developing an EBCP curriculum.

There are many stated benefits to EBCP and the importance to stakeholders (patients,^6,7 students,^8–10 clinician¹¹, other health professions,^12,13 and accreditors^6,14) is evident. These benefits include improved patient outcomes, common ground, and meeting accrediting requirements. The benefits of EBCP make it a worthy inclusion into the content of educational programs for physicians. Generally, content should be assessed for its usefulness to stakeholders, especially patients. Otherwise, the content becomes a waste of time to teach, manage, and assess for the teacher and an unnecessary barrier for the student to learn more applicable concepts.

Significant barriers exist to the integration of EBCP into curricula, into the practice of EBCP in clinical training settings, and into practice. The following barriers are prominent themes in the literature: time constraints,^{2,4,6,7,15,16} confidence,^9,17 a lack of adequately trained faculty,^7,15,16,18 teaching methods,^6,19–24 clinical challenges,^{7,8,12,18,25,26} negative attitudes toward EBCP,^4,7,8,18 and the belief that EBCP might degenerate into “cook-book” clinical practice.⁴

Given the above, the overreaching research question during our process was, “How does an educational institution prepare students for their intended profession?” More specifically, we questioned “How do we reevaluate curricula and determine if curriculum design promotes student learning to the benefit of their patients?” We also wanted to know the specific criteria that should be present in the curricular structure. To determine the degree of student learning, we wanted to know the level of depth that content was being taught and the assessment scores compared with the threshold. Finally, we wanted to identify if the content was necessary for students to perform their intended profession. For this project, we concentrated on the connection between the structure and level of student learning.

Because the subcurriculum was complex, visual representation of the way the EBCP topics progressed was desired to avoid gaps and a lack of Bloom's verb progression. We designed a graphical analysis of the curriculum to provide the visual representation. Therefore, our aims with this paper were to (1) describe the process we used to recreate our EBCP subcurriculum; (2) identify if the Bloom's verb level seemed to progress upward from the beginning of the curriculum to the end; (3) using the same plot, determine if there were gaps where a topic was not addressed periodically during the curriculum for learning reinforcement; and (4) determine if our approach to restructuring the EBCP subcurriculum provided adequate integration.

METHODS

Implementation

Based upon the levels of training of our faculty in EBCP through the R-25 grant, an EBCP faculty committee was formed comprising clinical and academic faculty. An EBCP topics list was created with proposed sequencing throughout the curriculum by 2 to 3 topic experts who formed an initial curriculum map. EBCP topics were determined by categorizing content into similar topics. Sequencing was based on where topics related to outcomes in the courses. EBCP committee faculty were asked to review the map and provide feedback. Changes were made based upon this feedback.

Committee members were then asked what EBCP topics they already taught and were willing to teach to complete the proposed sequencing. Learning outcomes were assigned based upon the sequencing and committee faculty feedback. Competencies and learning outcomes were created for each of the evidence-based topics by content experts. Most topics were intended to be introduced in quarter 1. Inclusion of 1 to 3 new EBCP learning outcomes in 1 class per quarter was implemented with reinforcement of each learning outcome in a later quarter. Evidence-based competencies were placed sequentially in courses so that the competency was introduced in first quarter, covered in greater depth in a subsequent quarter and reinforced in a later quarter as exemplified in Table 1. These sequential competencies were added to existing course syllabi as learning outcomes from first to 13th quarter. Learning outcomes were debated, refined, and approved by a learning outcomes committee and then approved by a faculty and administration level curriculum management committee and administrative committee. This project was authorized by the academic dean and granted exemption by the Palmer College of Chiropractic institutional review board (# N2018-10-1-M).

Table 1- Sample of Actual Evidence-Based Clinical Practice Curricular Map Including 2 Topics, Faculty Assigned to Instructional Topics, and Corresponding Learning Outcomes

View Fullscreen

Assessment (Structure)

A frequency analysis of the use of Bloom's verbs served as a starting point that we synthesized with statistics software to make a graphical representation. First, the outcomes from the syllabi were paired with relevant EBCP topics. Table 2 lists the EBCP topics and corresponding content. Most outcomes covered multiple EBCP topics. Some of the outcomes were at 2 Bloom's levels in 1 course, although most were at 1 level. Some courses had projects that often incorporated multiple topics and levels of understanding. Microsoft Excel (Microsoft Corp; Redmond, WA) was used to prepare data for analysis (Table 3).

Table 2- List of EBCP Topics

View Fullscreen

Table 3- Example of Spreadsheet Preparation for Data Analysis in R

View Fullscreen

Our structural analysis involved creating graphs of the topics that plotted Bloom's verbs on the y-axis and the quarter on the x-axis. Assessments were structural or learning performance focused. Structural assessments included determining aspects of sequencing, repetition, and level of rigor (Bloom's verb level). Learning performance assessments involved determining the grades achieved and whether they demonstrated clinically meaningful learning.

We used the ggplot2 (Wickham, Hadley; Switzerland) graphics program of R (version 3.4.3, R Foundation for Statistical Computing, Vienna, Austria) to plot our subcurriculum.²⁹ Structural analysis methodologies used by other researchers include Academic Analytics, Walktrap algorithm³⁰ and Bloom's verb based Frequency Analysis.²² In our analysis, the quarter was assigned to the x-axis, Bloom's level to the y-axis, and the topic number to functions called color and faceting. Figure 1 shows the ggplot2 script.

View Figure

• Download as Powerpoint
• Download Figure

RESULTS

The mapping of the curriculum quarters versus Bloom's verb level is presented in Figure 2. The value for Bloom's verb should increase as the student goes through the curriculum, showing a progression from memorization to critical analysis. Another feature was the ability to find gaps. A gap in the quarter on the x-axis shows that the topic is not being reinforced during that quarter. This could be an issue or not depending upon how much reinforcement is needed for that topic. A gap in the Bloom's verb level means that the topic is not being assessed at that level. These gaps showed that the outcome depth of processing did not have foundation, mid-level building blocks, or a cap stone, higher level exposure.

View Figure

• Download as Powerpoint
• Download Figure

Table 3 and Figure 2 were used to determine the amount of EBCP topics per quarter and where they are more concentrated. Table 4 showed that 14 of the 19 topics had increasing linear model trends, 5 had short linear trends, and 1 had no trend. Decreasing, short trends were found in 3 topics. Quarters 5–8 showed decreased attention to EBCP topics, especially quarter 5.

Table 4- Structural Analysis Guide and Results

View Fullscreen

DISCUSSION

The 14 of 19 increasing linear model trends were preferred patterns because they represent topics assessed at higher levels as the quarters progressed. The 5 short linear trends represent topics that are not assessed later in the curriculum. The 1 topic with no trend showed that the topic is taught only once. The 3 decreasing short trends represent topics that were developed at a higher level in the beginning but at a lower level later in the program. This is not a preferred trend. This structural analysis of the EBCP subcurriculum was helpful for us to detect trends and gaps for further refining of the curriculum. In our assessment plan, we have addressed building a curricular structure that includes early, frequent, repetitive, progressive development of EBCP topics. Based on our analysis, further development of the structure is planned.

Our future plans involve tracking the EBCP competencies tied with the learning outcomes within the courses. We will need to overcome some barriers to assessment, such as the compilation and analysis of data from all the courses into an easily understandable product. As well, it will be important to limit the changing of course outcomes contrary to an adequate process of revision (ie, curricular drift).³¹ It will also be important to identify outcomes that are not sufficiently demonstrated by students if they are intentionally doing well enough on other content to receive a passing grade. For example, a student could do well on the easier topics and not the harder topics.

Specific structural changes may be needed to address shortcomings identified with the structural analysis. We also plan to gather information about the students' mastery of EBCP topics through programmatic assessments. To assess our students' ability to apply EBCP principles to patient care, we can review data on our interns in quarters 9 through 12.

We believe that 1 of the unique aspects of our project was the goal of EBCP outcomes being introduced early, frequent, repetitively, and progressively throughout the entire curriculum. As stated by Bussieres et al⁷ in 2016, “academic programs should incorporate EBCP content early in a program of study and ‘weave' it throughout the 4–5 years of professional education.” We look forward to performing our learning assessment following this “weaving” concept. Our desired end goal is to assure a progression from knowledge to comprehension to application in all topics and lead to successful clinical integration of EBCP by our students. As noted by Nkangienieme³³:

“In the traditional pre-clinical years cognition by students is mostly at the knowledge and comprehension levels. In the first clinical year it moves to the application and analysis levels with expansion of the knowledge base and reduction of comprehension time. By the final year, students should be functioning mostly at the analysis and synthesis levels with occasional evaluation level processing to effectively solve the clinical problems that they are likely to encounter.”

It is of particular interest to note that students have been found to value higher order learning opportunities for EBCP; as noted by Thomas et al³⁴ in 2017, “Students believe that application of EBCP in the clinical context is of utmost importance and that the academic program should design opportunities for this to happen as often as possible.”

Comparison with another method by Whillier et al³¹ is presented in Table 5. We are interested in their more open format, research stream to increase research capacity, and inclusion of recent graduates and chiropractic clinicians in noncollege-related practice.

Table 5- Comparison Between Methods

View Fullscreen

CONCLUSION

The graphical verb-based, structural analysis seemed useful to identify progression of intended learning complexity and gaps in presentations of topics in our EBCP subcurriculum. Fourteen of the topics had increasing linear model trends indicating verb progression. Insufficient attention to EBCP topics was identified in some quarters. We hope that our experience incorporating EBCP into an integrated curriculum may be helpful to others.