Abstract
Introduction: Automated self-scheduling may benefit healthcare organizations, yet uptake has been slow. The aim of this study was to develop a consensus statement regarding the organizational-level determinants of implementation success based on the collective knowledge of experts. A three-stage modified Delphi method was used to reach consensus on the top determinants of implementation of self-scheduling solutions by healthcare organizations. A panel of 53 experts representing 41 academic health systems identified barriers and facilitators involving the organization’s inner and outing settings, as well as the characteristics of the intervention and the individuals engaged in the solution. Offering convenience for patients is the leading enabler for organizations to implement the technology. The consensus may aid healthcare organizations and suppliers engaged in adopting and developing self-scheduling technology to improve implementation success. Further research is recommended to diagnose and examine each barrier and facilitator and how these factors interact.
Objective: The aim of this study was to develop a consensus statement regarding the determinants of implementation success based on the collective knowledge of experts working in the field.
Methods: A Delphi panel was constructed based on selected participants employed by academic health systems and experienced with self-scheduling implementation. Panelists were recruited based on participation in an educational event that featured the topic. Purposive and snowball sampling were used. Panelists participated in surveys collected over three rounds. An 80 percent agreement among panelists and interquartile range (IQR) <1 determined the barriers and facilitators. The top-10 determinants were presented in rank order.
Results: Between January 6, 2021, and May 26, 2021, 53 panelists representing 41 academic health systems participated in three rounds of surveys to reach consensus on the barriers and facilitators to implementation of self-scheduling by healthcare organizations in the United States. In round one, panelists documented 530 determinants. In round two, the determinants were grouped into 72 barriers and 85 facilitators, each of which participants rated on a five-point Likert scale. Fifteen determinants met the 80 percent threshold and 1.0 IQR. The final round concluded with a top-10, rank-ordered listing of determinants (seven facilitators and three barriers) that also incorporated a median rating score using five-point Likert scale.
Conclusion: A three-stage modified Delphi method was used to reach consensus on the top determinants of implementation of self-scheduling solutions by academic health systems. The consensus may aid healthcare organizations and suppliers engaged in adopting and developing self-scheduling technology to improve implementation success. Further research is recommended to diagnose and examine each barrier and facilitator and how these factors interact.
Keywords: Delphi panel, automated self-scheduling, barriers and facilitators, implementation
Introduction
Appointment management in the ambulatory setting is important for healthcare organizations as waits and delays lead to poor management outcomes: dissatisfied patients, scheduling disruptions, and wasted appointment slots. For example, according to the Patient Access Collaborative,1 the median new patient lag time for a scheduled ambulatory appointment in the United States is 16.35 days, while the median utilization rate of appointment slots is only 73.6 percent.
Automated self-scheduling may benefit healthcare organizations in their efforts to manage staffing costs,2-7 patient satisfaction,8-10 appointment attendance,11-15 patient accountability,16 and information transparency.17,18 Self-scheduling may offer the convenience patients seek.19-20 Automated, self-service reservation systems have helped other industries striving for improvements in customer loyalty,21 operations,22,23 profitability,24 and customer wait times.25
Despite evidence to support the value of the technology to healthcare organizations, the uptake of self-scheduling in healthcare has been minimal to date.
Providers have expressed reluctance about self-scheduling based on cost, flexibility, safety, and integrity.26 As the technology emerged in the early 2000s, physician and software developer Dr. Jonathan Teich elucidated a critical challenge related to complexity: “Before you can successfully implement self-scheduling, you have to implement ‘Mabel.’ Mabel is the generic scheduling administrator who has been working for Dr. Smith for 35 years, and knows a thousand nuances and idiosyncrasies and preferences that have been silently established over the years … Unfortunately for the computer world, it’s extremely difficult to find out what Mabel really knows, let alone try and put it into an algorithm.”27 In addition to the Mabel factor, physicians have conveyed a fear of losing control.28-30
Researchers have also documented patients’ hesitancy about self-scheduling based on concerns about accuracy, security, and a lack of empathy as compared to a human interaction.31 Further, patients’ prior experience with technology, as well as communication preferences, have been recognized as barriers.32
What remains unknown is the potential influence of the current diffusion of self-service technology, the changes in patients’ access to virtual delivery platforms, and the heightened expectations for convenience that have resulted from the COVID-19 pandemic as it relates to healthcare organizations’ uptake of self-scheduling. By identifying barriers and facilitators to automated self-scheduling, the research will assist healthcare organizations seeking solutions to the management of the ambulatory enterprise, ultimately benefiting patients through improved service, reduced disruptions, and enhanced utilization of providers’ time.
Background
Despite the acknowledged benefits of administrative technology in healthcare, adoption has been slow, with implementation barriers cited as evidence of the limited diffusion.33
The goal of the study was to derive a consensus statement regarding the organizational-level barriers and facilitators to implementation of automated patient self-scheduling by healthcare organizations in the United States. The primary research question posed was: “What is the consensus regarding the barriers and facilitators as identified by professionals employed by academic health systems engaged in the implementation of patient self-scheduling?” The research aimed to inform healthcare organizations considering the implementation of self-scheduling. More broadly, the study may enlighten suppliers in the creation and maintenance of the technology for healthcare organizations.
Methods
Design
The consensus process was conducted using a three-stage modified Delphi technique to solicit, identify, and synthesize determinants of the implementation of self-scheduling technology by healthcare organizations. The modified Delphi technique is a structured, participatory qualitative research method.34 Named for the Oracle at Delphi in Ancient Greece, the Delphi technique, which was originally developed by the RAND Corporation in 1948,35,36 involves an iterative process until consensus is obtained. Due to the anonymity of the process, the risk of domination by one individual or coalition is avoided.37 The Delphi method has become a popular technique in health sciences research38 and technology foresight.39 The research method was selected, as the literature lacked evidence of the determinants of implementation of the technology under study.40 As self-scheduling represents an emerging technology for healthcare organizations, the opinions of stakeholders engaged as practitioners of the intervention are important.41,42
The Delphi method can account for key informants who are geographically and professionally diverse.43,44 Given the workload of the panelists during the COVID-19 pandemic, the Delphi technique was selected, as it does not require a specific meeting time, thereby allowing a thoughtful response at a convenient time for participants.45 This research was conducted electronically and was considered to yield the same results as a traditional paper-based survey.46,47
Participants
Participants were identified based on attendance at an educational event held in September 2020 that featured the implementation of automated self-scheduling solutions by academic health systems. Snowball sampling was subsequently applied to identify additional key informants with knowledge of the research subject. Panelists with direct expertise in the implementation of the technology were sought to ensure validity of the consensus statement.48 The author sent communication to 74 potential participants between December 16, 2020, and January 6, 2021, inviting them to participate in the study. The goal was participation from 40 to 60 participants based on other research studies that developed consensus about a complex subject involving different stakeholders.49 Fifty-three agreed to participate; 41 academic health systems were represented. Panelists were from all US Census Bureau-designated regions. (Table 1 presents the count of Delphi panelists by region.) The outpatient enterprises of the academic health systems represented by the panelists ranged from 500,000 to more than 4 million patient encounters per annum.
Delphi Technique
The first Delphi survey was distributed between January 6, 2021 and February 21, 2021, via an online survey tool (SurveyMonkey®) to participants’ email addresses. In the initial round, data on participant demographics were collected to include role, training, and geography. The first round featured an open-ended response to avoid introducing bias in the study.50,51 Participants were asked: “Describe six factors that negatively shape the implementation of self-scheduling at your organization” and “Describe six factors that positively shape the implementation of self-scheduling at your organization.” The responses were documented as barriers or facilitators and mapped in alignment with the Consolidated Framework for Implementation Research (CFIR).52 CFIR enabled the research to be presented in a standard, evidence-based framework, thereby facilitating the opportunity for industry adoption of the research findings.53,54 The CFIR Domains and Constructs are presented in Figure 1.
The second survey was distributed between March 2, 2021, and April 4, 2021. Participants scored agreement or disagreement with statements on a five-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree): “To what extent do you agree with this statement.” The survey was self-administered, thus allowing participants to respond without risk of influencing one another’s answers.55 To describe the relative importance of each item, the median and interquartile range were calculated.56 A consensus point of 80 percent was determined to prioritize the determinants.57-59 During the second round of the online survey, four barriers and 11 facilitators received equal or greater than 80 percent of participants’ votes. The 15 factors were compiled for the next round of the survey.
The third and final step of the Delphi was initiated with the panel on April 27, 2021. Responses were collected through May 26, 2021. Three survey rounds were employed to reach consensus.60 The third and final survey included the 15 factors that received greater than 80 percent of participants’ agreement during the second round. From these, participants were asked to rate each determinant using a Likert scale of 1 to 5 ranging from 1 (strongly disagree) to 5 (strongly agree): “To what extent do you agree with this statement.” The median and interquartile range were calculated.61 Participants were also asked to place in rank order the most important determinants of self-scheduling by healthcare organizations. The percent of the expert panel ranking the factor in the top 10 was also calculated to support the informants’ consensus.62 This iterative process permitted participants to reassess their views considering the aggregated results.63 (See Figure 2.)
The Johns Hopkins Bloomberg School of Public Health’s Institutional Review Board (IRB) determined that this research study does not qualify as human subjects research as defined by DHHS regulations 45 CFR 46.102, and therefore does not require IRB oversight. All participants provided informed consent to take part at the beginning of the process as part of the online survey.
Results
Of the 74 informants identified to participate, 53 agreed to contribute. Of the 53 who agreed to participate, 52 responded to the first round, 53 in the second round, and 52 in the third round. Up to four reminders were sent to encourage participation for each round beginning on the survey due date. The 53 participants represented 41 academic health systems and all regions of the United States. The participants were recruited from three categories: technology professionals (n=9), management professionals (n=41), and other stakeholders in self-scheduling (n=4) (e.g., clinicians who were engaged in the implementation). Management professionals included roles such as executive director of ambulatory operations, chief access officer, and vice president of ambulatory services. Participants could select more than one role. Eight participants were clinicians by training; 44 were not; one was unknown. All participants were individuals employed by healthcare organizations and experienced with a past, current, or future implementation of automated self-scheduling.
In the first round, a total of 530 factors that contributed to the implementation of self-scheduling by healthcare organizations were identified. Fifty-two participants cited a total of 277 factors that negatively shape the implementation of self-scheduling at their organization (barriers), and 253 positively shape the implementation of self-scheduling at their organization (facilitators). Participants submitted an average of 10.2 barriers and facilitators.
Fifty-three key informants responded to the second survey. The author categorized responses from Round 1 into 72 barriers and 85 facilitators based on CFIR. (See Table 2.) The key informants were asked to rate the 157 determinants using a five-point Likert scale. Fifteen factors were identified based on a consensus of 80 percent and higher than 1.0 interquartile range.
The 15 factors were presented to the expert panel in the third and final round. Fifty-two participants rank ordered the 15 factors between 1 (most important) and 15 (least important). The participants were also asked to rate each factor using a five-point Likert scale. The consensus of the panel is presented in Table 3. Three barriers and seven facilitators were identified.
Panel members gave the highest ranking to the enabler that reflected the patients’ needs: “Convenience for patients to schedule appointments via our self-scheduling solution.” The determinant also had the highest consensus, median, and percentage of experts who agreed or strongly agreed. Two other facilitators topped the list: the organizations’ culture to support access and the relative advantage of self-scheduling as compared to the call center. Other facilitators were identified as peer pressure from competitors, the engagement of the academic health systems’ executives, and the buy-in of leaders. Complexity was the primary barrier, as well as providers’ resistance based on specialization and the variability of scheduling protocols.
In total, seven facilitators and three barriers were identified by the expert panelists. Four of the five domains of the CFIR were incorporated in the 10 determinants considered most important to panelists, providing evidence of the broad array of components that influence the implementation of automated self-scheduling. The CFIR domain of “process” was the only one not included in the consensus of key factors. The results of the Delphi panel confirmed myriad determinants of an effective implementation of technology by healthcare organizations.64
Discussion
With this study, stakeholders rated the determinants of implementation for automated self-scheduling by academic health systems. The use of the modified Delphi technique successfully yielded a consensus of the top determinants of implementation to automated self-scheduling as offered by academic health systems. Development of an evidence-based consensus of implementation determinants can be used to further the diffusion of the technology. To the best of the author’s knowledge, this was the first study of its kind for this technology. The discussion of the determinants is presented in the framework of the eight CFIR constructs represented in the panelists’ top-10 list.
Patient Needs and Resources
The most-cited determinant by the Delphi panel was an enabler based on the users’ needs: “Convenience for patients to schedule appointments via our self-scheduling solution.” Recognition of the need for innovation is the initial stage of Rogers Diffusion of Innovation Theory.65 Awareness of the users’ interest was also evident in another top-10 determinant cited by the expert panel: ease of use for patients. Implementation may have been hindered historically by a lack of attention to patient needs. A perceived benefit for patients has led to implementation success by healthcare organizations.66,67 Proactive, clear communication about the benefits of the technology for patients facilitates implementation.68
Culture
Panelists documented and prioritized an organizational culture to promote access to care. There is evidence that culture impacts the success of technology implementation.69-71 The determinant tracks closely with the awareness of the need for the technology to facilitate access for patients, yet it establishes the panel’s perceived priority of the organization’s culture to achieve it.
Relative Advantage
Automated self-scheduling can effectively replace the same transaction over the telephone. The ranking of the relative advantage may reflect the panelists’ belief that automated self-scheduling offers a reduction of personnel costs, access outside of normal operating hours,72,73 improved staff utilization,74-80 and patient time savings.81-83 Regardless of the source of the advantage, the perception that such exists is an essential condition for successful technology implementation.84
Complexity
Acknowledging and analyzing complexity to avoid inadvertent consequences is crucial to the effectiveness of an implementation.85 The ranking of this determinant as the highest barrier may reflect the panel’s perception that the complexity may not be diagnosed or addressed by current solutions. There is an adverse association between the perception of complexity and the success of an intervention.86,87 Automated self-scheduling is a technology purchased, built, and deployed by a healthcare organization. Unlike other well-studied technology solutions like electronic health record systems, however, the primary user is the patient, not the organization, provider, or employee. The implementation of a patient-facing solution adds to the complexity and may increase the challenges of implementation.88
Knowledge and Beliefs about the Intervention
Providers’ resistance has been demonstrated in other studies about novel healthcare technology.89 Factors include fear and dissatisfaction with roles and responsibilities,90 a lack of trust,91 resistance to change,92 and uncertainty.93 Studies regarding physician receptivity, however, have centered on the implementation of electronic health record systems or their components. Similar barriers may exist for an administrative technology. The rationale regarding specialization may reveal the source of resistance, one that tracks closely with the previous barrier related to complexity.
Peer Pressure
The need to be competitive was revealed as the sixth facilitator to self-scheduling implementation. Panelists may consider self-scheduling to be a requirement rather than a luxury. This may reflect a mimetic response by healthcare organizations as it relates to competitors, considered to be highly influential for adoption of technology.94 The competitive environment for healthcare organizations is significant, with mergers and acquisitions predicted to increase in the future based on various policy changes and financial positions.95 Reacting to peers has been demonstrated to be particularly influential for organizations that are late adopters.96
Readiness for Implementation – Leadership Engagement
The expert panel concluded the involvement of leaders as important facilitators. Engagement of leaders has been determined to be of significance in all facets of technology implementation in healthcare.97-99 The inclusion of two determinants related to the involvement of leaders in the top-10 list promotes its import as a facilitator of implementation. The ranking of executive leadership engagement may reveal that direct management support is not sufficient for implementation success. As self-scheduling involves stakeholders both internal and external to the organization, executive leaders may be a crucial facilitator for automated self-scheduling.
Adaptability
The final top-10 determinant, “variability about scheduling protocols across providers or specialties within a department,” reflects an intervention characteristic. Adaptability is recognized as a critical factor as an intervention is disseminated more broadly within an organization.100 Ease of modification is positively correlated with an effective implementation.101-103
The Delphi panel’s key barriers and facilitators for self-scheduling offered insight into experts’ perceptions of determinants of implementation success. The factors that are absent from the list may be of equal import. “Process” was the only CFIR domain that was not represented in the consensus of determinants. According to CFIR,104 the domain, which incorporates engaging, executing, planning, and reflecting and evaluating, is the “single most difficult domain to define, measure, or evaluate in implementation research.” The lack of the domain being considered as a barrier or facilitator may confirm the placement of automated self-scheduling at the beginning of the technology’s life cycle.
The absence of cost (a construct within the “intervention characteristics” domain) and available resources (a construct within “inner setting”) may indicate that financial outlay for the technological solution is not a barrier. Time, effort, and resources, however, may be needed for healthcare organizations to address barriers to patients’ technology acceptance, a journey that has been determined to be present, complex, and nonlinear.105,106 The presence of a digital divide has been well documented for other technologies,107,108 and its absence as a barrier may also reflect the stage of the technology’s life cycle. As the technology is diffused, additional research regarding the digital divide is warranted.
Opportunities for Research
Further research is warranted to identify actions that may address the barriers and facilitators to implementation of self-scheduling technology. The research ascertained the determinants. Healthcare organizations may now proactively tackle the barriers and seek facilitators to increase diffusion of the technology. For example, organizations may survey patients regarding their expectations for a digital access experience, using reports that feature the voice of the customer to draw the organizations’ attention to the most important facilitator, the delivery of convenience. An inventory of competitors’ capabilities may be shared with leadership to address peer pressure. Known barriers such as providers’ resistance may be addressed proactively by open dialogue with providers about the technology, a step that may have otherwise been overlooked in the belief that the technological solution was solely administrative. Table 4 lists actions for healthcare organizations to consider to remove barriers and promote facilitators based on the determinants identified by the expert panelists. Further research is warranted to identify effective actions to address each determinant.
Limitations
The Delphi technique has been criticized for the potential for bias in participant selection and engagement.109 This study strived to overcome the bias through the variety of participants as it relates to geography, professional roles, and training.110 The value of the Delphi technique is determined by the quality and stability of the panel of participating experts and the time between rounds, which were proactively managed by the author.111 Participants represented various roles in academic health systems; however, they may not have represented persons from all areas of responsibility for implementation. The panel did not contain the opinions of suppliers (persons creating the technology) or patients (persons using the technology). As the research study aimed to develop a consensus for the implementation of the technology by healthcare organizations, these stakeholders were purposely excluded. This may have introduced bias in the results. The panels of experts represented healthcare organizations that were academic health systems; the ambulatory clinics associated with these healthcare organizations are large and complex. Gathering consensus from experts who represented academic health systems may limit generalizability of the results.
Conclusion
The purpose of this study was to provide consensus from a panel of experts engaged in automated self-scheduling about the barriers and facilitators to this novel technology. The Delphi method was effective in identifying 10, rank-ordered determinants of implementation success. The research may inform stakeholders about current priorities to consider the deployment and dissemination of this technology within healthcare organizations, thus contributing to the adoption of evidence-based practices to promote improvement efforts in managing service, access, and utilization of the ambulatory enterprise.
Acknowledgements
The author would like to thank the expert panelists for their participation in this research and Drs. Doug Hough, Kathryn McDonald, Michael Rosen, Aditi Sen, Jonathan Weiner, and Christina Yuan of Johns Hopkins Bloomberg School of Public Health for their guidance and support.
Notes
1. Collaborative, Patient Access. 2021 Benchmark Report. Atlanta: Patient Access Collaborative, 2021.
2. Friedman, J. “Internet Patient Scheduling in Real-life Practice.” The Journal of Medical Practice Management. 2004. 20(1), pp. 13-15.
3. Jones R, Menon-Johansson A, Waters AM, Sullivan AK. “eTriage - a novel, web-based triage and booking service: enabling timely access to sexual health clinics. Int J STD AIDS. 2010. 21(1), pp. 30-33.
4. Idowu, P., Adeosun, O., and Williams, K. “Dependable Online Appointment Booking System for NHIS Outpatient in Nigerian Teaching Hospitals.” International Journal of Computer Science and Information Technology. 2014. 6, pp. 59-73.
5. Kamo, N, Bender, J, Blackmore, C. “Meaningful use of the electronic patient portal – Virginia Mason's journey to create the perfect online patient experience” 2017. Healthcare.
6. Zhao, P., Yoo, I., Lavoie, J., Lavoie, B. J., & Simoes, E. “Web-Based Medical Appointment Systems: A Systematic Review.” Journal of Medical Internet Research. 2017. 19(4), e134. https://doi.org/10.2196/jmir.6747.
7. Lee, Y. P., Tsai, H. Y., and Ruangkanjanases, A. “The determinan