Abstract
The advent of electronic health records has led to significant changes in the role of health information management (HIM) professionals. These changes have required significant changes in the educational training required for HIM professionals, leading to new curriculum requirements. To meet the need for well-prepared professionals in the changing HIM environment, it is essential for educational programs to ensure that they are effectively preparing students for the new roles they will be expected to fill. This in–depth case study of one HIM academic program contributes to a program review and evaluation to determine the impact of the recent required curriculum changes on the program. This study used a utilization-focused evaluation framework that included evaluation capacity building. The study specifically assessed the ability of the recent curriculum changes to adequately prepare students for the current HIM field as well as the effects of the changes on the program as a whole. This evaluative study goes beyond the use of metrics such as enrollment, retention, and RHIA exam pass rates and delves deeper through surveys, interviews, and focus groups to collect data on the students’, graduates’, and employers’ perceptions. Through the study, major themes were identified, including perceptions of student satisfaction and student preparedness, perceptions of the new curriculum, concern about loss of parts of the old curriculum, and future needs for the HIM program and for the HIM field in general. While many findings were expected, some surprising or unexpected findings included concern about loss of parts of the old curriculum, and close alignment between respondents’ area of interest or career path and perceptions of the new versus old curriculum. The study allowed the program to assess preparation of students for the current HIM field as well as other effects of the curriculum changes on the program. This assessment allowed for a true evaluation of the modified curriculum as well as identification of areas for improvement.
Keywords
Academic program evaluation, health information management, AHIMA CEE curriculum competencies, curriculum changes, utilization-focused evaluation, evaluation capacity building
Introduction
In 2009, the U.S. government enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act to encourage the use of technology in healthcare. This act focused on the use of “technology to improve the quality of care and to decrease healthcare costs” 1 and led to the widespread adoption and use of electronic health records (EHRs) throughout the healthcare field, resulting in significant revisions in the role of health information professionals. These revisions have also significantly changed the type of educational training necessary for health information professionals, leading to new curricular requirements.
Since the advent of the HITECH act, health information professionals have been called upon to serve in roles not dreamt of even 20 years ago. No longer are health information professionals buried in paper, working to get physicians to complete their charts. These professionals now need to have a much higher level of information technology knowledge as well as more data and information governance skills and analytic skills. “‘Big data,’ electronic record-keeping, and a shifting regulatory environment have reshaped [the health information field], and now these positions often involve sophisticated, judgment-based work.”2 Health information professionals now need skills to work with the technology as well as to analyze the vast quantities of data provided by the technology.
To meet these changing needs in the profession, health information management (HIM) academic programs have been called upon to focus their curriculum more closely on these areas. In 2014, the AHIMA Council for Excellence in Education (CEE) published new curriculum competencies requiring this additional focus in all health information education curricula starting in 2017. These curriculum changes were significant in that they required new topic areas and more extensive coverage of many curricular competencies. Implementation of the curricular changes challenged health information educators in the extent of the changes as well as the addition of new topic areas. Now that the changes have been implemented, the question arises as to whether they have resulted in improvements in the educational programs and in better preparation of graduates to meet the current professional needs.
This evaluative study was designed to focus on the evaluation of the changes in one HIM academic program, specifically an evaluation of the ability of the revised program to adequately prepare students for the current HIM field. The goal of this evaluation was utilization-focused so that the results of the evaluation could be used by stakeholders for continuous program improvement. The hope was that the findings would not only benefit the program under review but also provide insight for other HIM programs.
Theoretical Framework
This study used the utilization-focused evaluation framework and thus focused on use of the findings to make positive changes in the program. In conjunction with this theoretical framework, the study used the concept of evaluation capacity building to ensure that the HIM faculty, department chair, and other stakeholders understood the evaluative process and were ready to use the results for positive change.
This study was modeled after Patton’s utilization-focused evaluation framework. 3 Patton’s theoretical framework focuses on the use of evaluation information. Evaluation and assessment completed under this theoretical frame uses a “targeted group of stakeholders whom it empowers to determine the evaluation questions and information needs.” 4 In such evaluations, “the primary intended users of the evaluation”5 determine the evaluative criteria. The theory behind this high level of stakeholder involvement is that, by including the stakeholders in the design of the evaluation, the results will be more useful and more likely to be used. The purpose is to “give them [the stakeholders] the information they need to fulfill their objectives.” 6 This theoretical framework can be valuable in educational program improvement since “this approach is geared toward maximizing evaluation impacts [and] fits well with the key principle of change.” 7 The weaknesses of this theoretical frame include the possibility of stakeholder turnover and the potential for stakeholders to “look for evidence to confirm [their] preconceptions and biases.” 8
The personalized nature of utilization-focused evaluation can result in faster and more readily accepted changes and improvements. The “crucial point is that evaluators must determine and focus their studies on intended evaluation uses and produce and report findings that an identified group of intended users can and probably will value and apply to program improvement.” 9 The goal of this theoretical framework is to use the evaluation findings to improve educational programs, thus resulting in improved programmatic outcomes and student learning. This framework was felt to be appropriate for this evaluative study as the program review and assessment was meant to not only evaluate student, graduate, and employer perceptions of the new curriculum but also aid in the identification of potential improvements that the HIM program could take to best prepare students for the HIM field.
This evaluative study was designed to go beyond the mere collection of metrics to truly evaluate the effectiveness of the revised curriculum in meeting the program’s goals and desired student learning outcomes. While HIM programs undergo accreditation and other types of internal program review, those program evaluations frequently focus on structure and process and do not focus on outcomes such as student learning and preparedness for the field. This study provided an in-depth evaluation that included students’, recent graduates’, and employers’ perceptions of the effectiveness of the revised curriculum in adequately preparing students for the HIM field. While quantitative measures such as enrollment, retention, new curriculum transfer rates, and RHIA exam pass rates were also used, this study focused on evaluating the effectiveness of the curricular changes based on the perceptions of students, graduates, and employers. Effectiveness in adequately preparing students cannot be measured by grades alone. While particular students may receive good grades on assignments in classes, they may not be adequately prepared to function in the workforce. Therefore, this study delved into the review of the HIM program through a utilization-focused evaluative methodology designed to determine the students’ readiness for the professional world based on their education in the program as a whole.
The program was well prepared for an evaluative study. The faculty and department chair had been involved in the planning process since the revised Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM) curriculum requirements were released. The faculty undertook a gap analysis of the existing curriculum to determine the need for changes in the curriculum. The faculty worked collaboratively to implement the changes in the curriculum, including changes in courses, addition of new courses, and removal of other courses. Finally, the faculty, department chair, and students had been involved in ongoing informal discussions of the curriculum change and the status of the HIM program.
Context
This evaluation was completed using an in-depth case study of the HIM academic program at a large midwestern public university. The required curricular changes may have been made more quickly at this institution than at other institutions because the program was scheduled to undergo the CAHIIM reaccreditation assessment and visit during the 2017–2018 academic year. Because the reaccreditation was to be based on the new curriculum, the HIM faculty wanted to ensure that the curriculum was being taught before the 2017-2018 academic year. Therefore, this site was ideal for an early evaluative study of the effects of the newly implemented AHIMA CEE competencies on HIM academic programs.
At the institution under study, the first step in implementing the new 2014 competencies was completion of a gap analysis. This gap analysis compared the existing curriculum to the newly required curriculum and identified areas in which changes were needed to comply with the new curriculum. Data from the gap analysis identified a number of areas for change. Between 2014 and 2016, the HIM faculty developed the newly required curriculum and determined the necessary programmatic changes. The program started implementing the required revisions to the curriculum during fall 2016 and fully transitioned to the new curriculum with students entering their junior year of the program in fall 2018. These curricular changes included the following:
- The addition of six information technology classes to cover the information technology (IT) competencies;
- The removal of three economics and management classes that were felt to overlap with other HIM courses;
- Revisions to existing courses to cover additional data analysis and data governance topics; and
- Integration of the content of two HIM courses into other courses and deletion of these two courses.
Based on university policies regarding curriculum changes, these changes were implemented with the 2016 university course catalog, meaning that all incoming freshmen in fall 2016 and some transfer students who started at the university in fall 2016 were required to complete the new curriculum. This resulted in a transition process in which two cohorts, the graduating classes of 2018 and 2019, had dual sequences. These cohorts had a mix of students completing the old curriculum and students completing the new curriculum, depending on which sequence they were assigned at the time of enrollment. The graduating class of 2018 had 4 students in the new “IT” curriculum and 17 students in the old curriculum; the class of 2019 had 6 students in the new “IT” curriculum and 11 students in the old curriculum. The graduating class of 2020 will be the first cohort in which all students will be required to complete the new curriculum.
Purpose and Questions
The purpose of this evaluative study was to complete an in-depth case study that contributed to a program review and evaluation to determine the impact of the recent curriculum changes. This included not only the impact on student learning outcomes but also the impact on the program as a whole, including enrollment and retention, as well as time to graduation and RHIA examination pass rates. This evaluation process was designed to be focused on quality improvement as well as utilization. The goal was to determine the effectiveness of the curricular changes as well as the effect of the changes on the program and to use those findings to improve the curriculum and program. The questions to be answered by this study were as follows:
1. What is the nature of current students’, graduates’, and employers’ experiences related to HIM curriculum shifts from the 2012 curriculum standards to the 2014 revision? (This question focused on the student, graduate, and employer perceptions regarding the curriculum changes and preparedness for the HIM field.)
2. In what ways have the 2014 curricular requirements shifted the entry-level skills and knowledge of HIM graduates in comparison to the 2012 curriculum standards?
3. Considering the ongoing changes in the curriculum requirements, what are the implications for the HIM field and for individual programs moving forward?
Methodology
To assess students’, recent graduates’, and employers’ perceptions of the success of the revised HIM program in preparing students for the HIM workforce, surveys (Appendix A) were administered to each of these groups. The researcher developed these surveys and administered them electronically to each group through Qualtrics. These surveys collected basic data regarding the student, graduate, and employer perceptions of the HIM program.
To gather more in-depth data, interviews were held with targeted employers who hosted professional practice students or hired new graduates from the HIM program. The employer interviews allowed for the collection of more in-depth data regarding the employers’ experiences with the school’s HIM students and graduates and their perceptions of the HIM program.
Focus groups were also held to obtain more in-depth data from the HIM students regarding their perceptions of the program. These two focus groups included a group of five students following the new IT curriculum and a group of five students following the traditional curriculum. The program academic advisor facilitated these focus groups, which were guided by basic questions designed to start discussion.
Documents displaying program metrics, such as enrollment trends, retention rates, time to graduation, and new curriculum transfers, were also used for a review of student and potential student actions related to the specific changes. RHIA exam pass rates were analyzed as a measure of student preparedness. The university’s internal program review report was also reviewed for findings and recommendations related to the curricular changes and student preparedness. Multiple types of data, including these documents as well as data from the surveys, interviews, and focus groups, were reviewed and triangulated to determine answers to the evaluative study questions and to assess the effect of the curricular and programmatic changes in adequately preparing HIM program graduates. Triangulation, the use of multiple data sources to enable confirmation of findings, adds to the “trustworthiness of our analysis.”10
Participants
All participants in this evaluative study were affiliated with the program being evaluated. All 17 final–year HIM students were surveyed during their fall semester. Students who electronically consented to participate in the study were given access to the survey questions. The first question on the survey asked which sequence the student completed (new versus old curriculum). Other questions were aimed at gathering information regarding the students’ perceptions of the HIM curriculum, the program, and student preparedness.
All 17 final-year HIM students were invited to participate in the appropriate focus group for their curriculum sequence to gather more in-depth data about the students’ perceptions of the HIM program. All graduates (21 from the 2018 cohort and 25 from the 2017 cohort) were sent an electronic survey, six months (for the 2018 cohort) or one year (for the 2017 cohort) after graduation. Graduates who consented to participate in the study were given access to the survey questions, which asked about the graduates’ perceptions of their preparedness for the HIM workforce at the time of graduation and which curriculum sequence they completed. The graduates’ perceptions were expected to differ from the students’ perceptions since the graduates may have had six months to one year of work experience and were able to reflect on the questions with that experience in mind.
All employers who hosted a junior or senior professional practice student (37) were surveyed as well. The employers included hospitals and other healthcare-related employers throughout the state. The employer survey was sent electronically to the HIM director or individual who oversaw the professional practice. Employers who consented to participate in the study were given access to the survey questions, which included inquiries about the employers’ perceptions of the students’ and new graduates’ preparedness for the workforce based on the newly revised curriculum. Five employers were selected for interviews on the basis of the number of professional practice students they had hosted and/or the number of new graduates they had hired from the HIM program. The interviews included more in-depth questions regarding the employers’ perceptions. These employers provided valuable information in this regard since most of these employers not only provided professional practice opportunities for students but also hired many graduates.
Data Analysis
The constant comparative method outlined by Glaser was used for analysis of the data collected in this study.11 The interview transcripts, focus group summaries, and open-ended survey question responses were coded to identify themes. General categories were identified through this coding. Initially broad categories were identified, followed by identification of specific details within the broad categories. As further data were analyzed and coded, they were “compare(d) . . . with the previous incidents coded in the same category.”12 Strauss and Corbin note that “categories have conceptual power because they are able to pull together around them other groups of concepts or subcategories.”13 Categorizing provides the ability to organize large amounts of data into manageable pieces that can be analyzed.14 Eventually, themes can be identified and analyzed. Finally, the coded data and themes were used to summarize the data and the overarching themes found in the data. The constant comparative model allowed for a thorough, organized approach to understanding the data collected through the various methodologies used in this evaluative study.
Program metrics, exam pass rates, and closed-ended survey questions were analyzed using basic descriptive statistics, such as means, modes, standard deviations, and frequencies, to determine trends and variations related to the implementation of the new curriculum. These data were tied into the constant comparative analysis through triangulation to gain a deeper picture of the effectiveness of the new curriculum in meeting the program’s goals and desired outcomes. These metrics were used to evaluate basic program outcomes and trends over time and were compared and combined with qualitative data to further evaluate relationships between the groups’ perceptions and quantitative outcomes.
Data Analysis and Findings
First, the student, graduate, and employer groups were analyzed individually, and then the constant comparative method15 was used to analyze the data and identify prevailing themes among the groups. The HIM program metrics were also included to provide further depth of analysis. Once the data were analyzed, several major themes emerged. These included perceptions of student satisfaction and student preparedness, perceptions of the new curriculum, concerns about loss of parts of the old curriculum, and future needs of the HIM program and the HIM field in general. These major themes aligned closely with the research questions of this study.
The broad themes identified through the analysis and triangulation of data were therefore able to be used to answer the research questions. While the themes that were identified closely aligned with the research questions, some findings stood out as interesting or unexpected. The concerns about the loss of parts of the old curriculum were interesting and somewhat unexpected in light of the changes in the HIM field. In addition, the perceptions regarding the new and old curriculum sequences and the alignment of these perceptions with the sequences based on the individuals’ areas of interest were also somewhat surprising.
Student, Graduate, and Employer Experiences and Perceptions
Overall, the students, graduates, and employers expressed positive experiences with and perceptions of the HIM program, the new curriculum, and student preparedness. All students surveyed stated that they were satisfied or highly satisfied with the HIM program. Students cited specific classes that they felt would benefit their career and noted the attainment of skills, such as time management, that contributed to their satisfaction. Students also commented that the HIM program’s small size and significant individualized attention was a positive aspect of the program that set it apart from other programs on the university’s campus. Graduates also expressed overall satisfaction with the HIM program, with statements such as “great program.” Graduates commented that the majority of classes were useful to them in their career and that their education had served them well in their first jobs and early career advancement. This view is reflected in one graduate’s statement, “I highly recommend this program.” Students and graduates also commented on the rigor of the program and expressed appreciation for the fact that the hard work required in the program helped prepare them for the field. Employers praised the HIM program, stating that the graduates were professional and that they were consistently pleased with the graduates they hired. One employer stated that this program was their “go-to” school when looking to hire new graduates to fill open positions.
When focusing on the new curriculum specifically, the respondents had positive comments overall, although some areas of concern were noted. The groups were all aware that the HIM profession was moving toward a much greater focus on IT and data analytics and that this was the impetus behind the changes in the HIM program curriculum. All groups understood that these changes were a result of the movement to electronic health records (EHRs) and the additional data available through EHR systems. The students indicated a division in their perceptions of the new curriculum based on the curriculum they were completing. The students completing the new curriculum had more favorable perceptions of this curriculum than those completing the old curriculum. Students completing the new curriculum were pleased with the IT component because they felt that IT was the “future” of HIM and that it would broaden their career opportunities. Students in the new curriculum did express some concerns about a perceived disconnect between the IT and HIM classes as well as a lack of focus on healthcare IT. They recognized that the program was in a period of transition and that the volume of changes during this time could lead to these issues.
Students completing the old curriculum felt that while the new curriculum may have advantages, pieces of the old curriculum were still needed. Many students in the old curriculum chose it because they were more interested in management and traditional HIM roles than the IT or technical side of the profession. These students expressed concern over the loss of some of the management portions of the curriculum due to the change. Overall, however, students in both curriculum sequences expressed satisfaction with the sequence they were completing as well as the feeling that the sequence was the best fit for them.
Graduates also commented that IT was a growing area of emphasis in the HIM arena and that these skills would open new doors for employment in areas such as EHRs, healthcare IT, and data analytics. Graduates provided specific IT skills they had learned and described how they had applied these skills in their workplaces. Employers noted that they used varying levels of IT skills on the job; however, they stated that they felt future graduates would need a strong background in IT and data analytics. To support this assertion, they referenced the changes in the field and the increased emphasis on IT as well as the fact that IT courses and knowledge increase skills such as critical thinking, which are also needed by new graduates. However, employers in more traditional HIM roles had concerns about the loss of skills from the old curriculum, including management and coding skills. These employers tended to be somewhat wary of the curriculum changes.
Program metrics point to the fact that the curriculum changes were not initially perceived as positive but that these perceptions may have changed over time. When the new curriculum was first implemented, a number of students transferred out of the new curriculum or out of the HIM major. As can be seen in Table 1, in the first cohort, 14 students started in the new curriculum, and only 4 (28 percent) of them finished in the new curriculum. Changes were made during the first year of the new curriculum implementation, including hiring an IT tutor for HIM students in their first two IT courses, An Evaluative Study of a Health Information Management Program Following a Significant Curriculum Change—Part 1 rearranging HIM class schedules to accommodate IT courses, and adding recommended courses to be taken before the first prerequisite IT course. Table 2 shows that the second cohort of students in the new curriculum had 70 percent of the students on track to complete the IT sequence by May 2019.
New Graduate Preparedness
When asked about perceptions of preparedness, 100 percent of the students surveyed felt that they were prepared or highly prepared for their career in HIM. Students emphasized that they felt that the applied focus of the HIM program was preparing them for jobs in the field. One student stated, “It’s preparing us for the real world.” Students stated that in their professional practices they were praised for their knowledge. One student stated, “When I was on my Junior Professional Practice, my supervisor told me that she was very surprised about all the knowledge that I had about everything, like some of the stuff that I knew, she honestly didn’t even know.” Overall, HIM program graduates felt prepared, with 82 percent of survey respondents stating that they felt they were prepared well for the HIM field. The graduates who felt they were not as well prepared cited specific areas in which they felt their skills were lacking. One such area was billing and revenue cycle management. A number of graduates are working in billing and revenue cycle jobs, a newer career track for HIM graduates. Other graduates emphasized that while they felt the HIM program provided them with basic HIM skills and knowledge, they had to learn many things on the job. These graduates also noted that the HIM field is very diverse and therefore it is hard for an educational program to prepare students completely in all areas. Graduates in both curriculum sequences stated that they felt prepared for the field, with 3 of 4 graduates (75 percent) in the new curriculum (75 percent) and 17 of 18 graduates in the old curriculum (95 percent) stating that they felt they were prepared or highly prepared.
When employers were asked about the preparedness of the HIM students and graduates, 100 percent responded that the students and graduates were prepared or highly prepared. Many respondents commented that they felt that students from this HIM program were better prepared than students from other schools. Areas in which employers felt that these students or graduates were particularly well prepared included medical coding, business operations and processes, traditional HIM functions, and professionalism. Some employers mentioned that they felt preparedness was also a function of student maturity or new graduate discipline. They mentioned as well that some students or new graduates were lacking skills in communication, both oral and written, as well as other basic workplace skills. While very few employers that were surveyed or interviewed had hosted professional practice students or hired new graduates who had completed the new curriculum, most were familiar with the curriculum changes. Many employers pointed to the need for new graduates to have basic IT, critical thinking, and database skills. The employers stated that they felt the recent curriculum changes would improve the preparation of students in the needed areas of IT, data analytics, critical thinking, and data governance.
RHIA exam pass rates were also analyzed to evaluate student preparedness. No graduates who had completed the new curriculum had taken the exam. However, the exam pass rates did reflect other changes made to the curriculum in 2016, such as the greater emphasis on data analytics. As can be seen in Table 3, there were no substantial differences in the exam pass rates between the 2016, 2017, and 2018 cohorts (93 percent, 82 percent, and 86 percent, respectively). These pass rates are all above the national average, which was 64 to 67 percent during this time frame.16 This indicates that the program’s HIM students were well prepared for the national certification exam.