The Leapfrog CPOE Evaluation Tool
November/December 2010
The Leapfrog CPOE Evaluation Tool: One Academic Medical Center’s Experience
A large proportion of medication errors occur at the prescribing phase. Computerized provider order entry (CPOE) offers a possible means by which to avoid these potentially harmful mistakes through implementation of electronic ordering and clinical decision support software (Kuperman, et al., 2007). Prior studies have demonstrated that CPOE and decision support reduce medication errors (Kadmon, et al., 2009; Kaushal, et al., 2003; Kuperman, et al., 2003), with less evidence directly linking this technology to a reduction of adverse drug events (Peterson, et al., 2005; Wolfstadt, et al., 2008). Despite these advantages, only 17% of healthcare institutions have fully implemented CPOE, and even fewer are using CPOE with decision support (Jha, et al., 2009; Pedersen, et al., 2008). The software is not only expensive but also requires resources to integrate it into the current information technology (IT) infrastructure, train staff, and support ongoing maintenance. Some institutions favor less expensive alternatives—such as barcode medication administration—as a patient safety investment (Scanlon, et al., 2008).
Formed following publication of the highly publicized Institute of Medicine report, To Err Is Human, in 2000, the Leapfrog Group is a coalition of healthcare purchasers with a mission to help make great “leaps forward” in the safety and quality of healthcare (Kilbridge, et al., 2006). The group’s strategy centers on a set of purchasing principles voluntarily agreed to by member purchasers that target consumer awareness and provide incentives to patients and providers based on the comparative value of healthcare delivery. Ideally, these principles would create a business case for providers and hospitals to drive advancements in safety and quality (Milstein, et al., 2000). Recognizing the need to promote CPOE adoption, the group selected CPOE as one of its four safety standards. The CPOE standard targets medication prescribing error reduction, one of several capabilities of these order entry systems. To this end, the Leapfrog CPOE evaluation tool was designed to assess the error trapping performance of implemented CPOE systems. Metzger and colleagues recently reported on the wide variability in performance of CPOE decision support across 62 U.S. hospitals using the Leapfrog tool (Metzger, et al., 2010). In this article, we describe our institution’s experience with the adult and pediatric versions of the CPOE evaluation tool for inpatients.
Leapfrog CPOE Evaluation Tool
To meet the Leapfrog CPOE standard, providers must enter at least 75% of medication orders through a computerized application that includes prescriber error prevention safeguards. This system must alert providers to at least 50% of common, serious prescribing errors (e.g. ordering penicillin for a patient with a known allergy to this antibiotic) (The Leapfrog Group, 2010a, 2010b). Leapfrog tests the operational functionality of an implemented CPOE system using a series of mock medication orders and test patients derived from published literature of medication errors, industry experience, and expert opinion (Kilbridge, et al., 2006). A separate set of patients and orders was developed for the pediatric version of the evaluation tool. The orders evaluated are divided into 10 medication error categories (therapeutic duplication, single and cumulative dose limits, allergies, contraindicated route of administration, drug–drug interactions, drug–food interactions, drug–diagnosis interactions, and contraindication/dose limits based on age and weight, laboratory studies, and radiology studies), plus three additional categories to assess nuisance alerts, cost of care, and corollary orders. When completed, hospitals receive percentage scores for each medication checking category (Kilbridge, et al., 2006). Using an algorithm, these scores comprise the overall score for the evaluation tool (The Leapfrog Group, 2010a, 2010b). Solid performance is indicated by a score of at least 50% for both the drug–drug interaction and allergy checking categories. The overall score is reported as “completed the evaluation,” “good early stage effort,” “good progress in implementing,” or “fully implemented.” This rating is combined with implementation status to form a composite score reported publicly on Leapfrog’s website.
Duke University Hospital
Duke University Hospital (DUH) is a tertiary care facility with over 1,000 beds and >40,000 admissions annually. Adult care areas include 14 non-intensive care units (ICUs), an adult bone-marrow transplant unit, and five ICUs. Within DUH, pediatric patients are cared for in four ICUs (pediatric, cardiac, neonatal, and nursery), a bone marrow transplant unit, and two general care units, collectively known as Duke Children’s Hospital (DCH). DCH averages 7,000 admissions per year across 187 beds, nearly half of which are critical care beds. The pharmacy department practices an integrated model with decentralized clinical pharmacists servicing each care area.
The DUH/DCH system is supported by an IT staff consisting of six nurse analysts, three clinical pharmacists, three programmers, three technicians, and one senior manager. The pediatric version of the CPOE system was implemented in early 2007 as an enhancement of the adult version (implemented in 2004) in a joint development project with the vendor. To maximize safety and accommodate the needs of this unique and diverse population, an advanced dosing model (ADM) was designed incorporating age, weight, indication, renal function, and location to guide prescribing. Full details of the ADM will be published pending peer review.
Over 900 CPOE order sets have been built by the IT group, from simple admission order sets to complex interfaces with capabilities to reason over patient-specific parameters (e.g., labs, age, diagnoses, and weight). The system processes over 22,000 orders daily, of which 6,000 are for medications. Ninety-five percent of inpatient medication orders are entered via CPOE. In 2009, a bi-directional interface between the CPOE and pharmacy systems was implemented to reduce transcription errors and increase drug ordering efficiency.
While the CPOE features are robust, the additive benefit of a linked, homegrown electronic health record (EHR) provides patient-specific information at the time of order entry; that is, one-click, easy access to all laboratory, microbiology, radiology, and medication data plus clinical documentation. Ambulatory clinics also post clinic notes in the EHR, thereby facilitating transitions in care, including medication reconciliation.
Leapfrog Evaluation: Results Overview
According to the Leapfrog’s CPOE standard composite score, DUH has made “substantial progress” towards meeting the CPOE safety standard. This is the second highest ranking, owed mostly to the fact that over 75% of orders are entered into CPOE. The categories for which we received high and low scores are shown in Table 1. Despite the extensive resources committed to tailoring CPOE to diverse patient safety needs, some scores suggest our CPOE system does not perform well at mitigating ordering errors. In the following sections, we share our interpretation of these scores with the hope that this may aid other organizations participating in the Leapfrog evaluation to bring into focus the full value of their CPOE systems for patient safety and quality.
Table 1. Duke University Hospital Leapfrog CPOE Evaluation Scores: Adults and Pediatrics
Interpretation of Low Leapfrog Scores: Common Themes
Implementing a vended CPOE product straight “off the shelf” is unlikely, and there is a need to customize the system, which is highly vendor-dependent. Often organizations must develop or utilize alternative safety features. At times, the Leapfrog evaluation process may not capture these workarounds, which nonetheless accomplish the objective of medication prescribing safety.
Therapeutic Duplication
Duplicate therapy is often clinically appropriate (e.g. combined warfarin and heparin bridge therapy). Inability to tailor alerts for such cases can produce low-specificity alerts, shown to lead to alert fatigue and unjustified overrides, which reintroduce medication error risk (van der Sijs, et al., 2006). An article posted by Margalit Gur-Arie, former COO of GenesysMD, reiterates that the Leapfrog tool only evaluates whether CPOE systems provide alerts rather than the actions of users in response to those alerts (Gur-Arie, 2010). In other words, the presence of alerts does not guarantee that they will be acknowledged by providers. In fact, previous articles have commented on the high override rates of CPOE warnings (Kuperman, et al., 2007; van der Sijs, et al., 2006). As a result, a conscious decision was made at DUH to direct medication alerts to the most appropriate healthcare professionals within the best possible IT forum, thereby limiting disruptions to physician workflow. Therapeutic duplication checking, therefore, was deactivated in CPOE and left activated in the pharmacy IT system, where pharmacists review and approve medication orders before dispensing drugs. Upon review of the Leapfrog tool’s scenarios assessing duplicate therapy checking within CPOE, our pharmacy system would have alerted a pharmacist in each case in which CPOE did not. As a passive alerting method, when a provider begins entry of an order for a drug already active on a patient, the first search result displayed in CPOE is a prompt to “change current order.” Although we did not count this towards the CPOE test responses, it could still be acknowledged as a method of provider notification within the CPOE application.
Drug–Drug Interactions
Low scores were also received on both the pediatric and adult tests for drug‚Äìdrug interaction checking. Internal review following implementation of these alerts revealed high override rates of roughly 90%. Improved specificity of drug–drug interaction checks to clinicians and avoidance of alert fatigue was sought by activating only the highest level of drug interaction alerts (i.e., contraindicated interactions) within CPOE. The interaction levels are pre-defined and maintained by a third-party database and cannot be altered by our IT group. Should we have fully activated the therapeutic duplication and drug–drug interaction applications supplied by the vendor, our CPOE system would have scored near 100% on these categories but at the potential price of introducing new errors and safety risks from alert fatigue (van der Sijs, et al., 2006). The pharmacy IT system, on the other hand, maintains more comprehensive interaction checking in addition to therapeutic duplication screening during order verification. Targeting alerts to this system initiates review by a qualified drug expert.
Contraindication/Dose Limits Based on Labs
Our CPOE system is unable to reason over patient-specific laboratory values to alert providers of important results when entering a medication order via an order set. However, in a passive notification effort, we can display a patient lab result of interest or provide general lab warning messages (e.g., do not give if serum creatinine is >2.5 mg/dL) in the processing instructions. Drug information content provided in the processing instructions is limited by the same consideration as other alerts – minimization of alert fatigue. However, DUH has put workarounds in place within the pharmacy department workflow. For example, the pharmacy system populates recent lab result alerts in the pharmacist’s work queue (e.g., low platelets 1.5 mg/dL), prompting review of a patient’s entire medication profile for adjustments throughout the course of therapy. Pharmacists review daily monitoring sheets identifying medications requiring laboratory monitoring (e.g., digoxin, enoxaparin) and manage all aminoglycoside and vancomycin therapy (i.e., drugs requiring serum creatinine and drug level consideration). Though CPOE may not alert providers to certain lab values upon order entry, the organization clearly has incorporated alternative methods to maintain a level of safety.
| |
 |
| Figure 1. Processing Logic of the Advanced Dosing Model (ADM). A clinician initiates an order for a medication. ADM pre-processing logic retrieves available patient information and prompts the user for any additional information needed. Once all patient parameters are collected, the decision support algorithm will resolve the list of all potential dosing regions from the clinical knowledge base. The appropriate dosing region is presented to the clinician for completion of the medication order. |
Contraindication/Dose Limits Based on Age/Weight
Upon completion of the Leapfrog evaluation tool, a surprisingly low score for “contraindication/dose limits based on age/weight” was received for the pediatric test. As mentioned previously, pediatric advanced dosing functionality was co-developed with our vendor as an enhancement to the adult-based CPOE system. Once a drug is selected in CPOE, this advanced decision support is triggered and reasons over patient-specific parameters – including chronological age, gestational age, weight, location in the hospital, indication, and/or level of renal impairment – to present providers with safe dose/frequency combinations for a given patient and avert potential medication prescribing errors (Figure 1). Though generally not part of Duke’s safety culture, providers can override the dosing region logic by manual entry of a medication regimen. As a safeguard against erroneous orders, programming effort was made to provide maximum dose limit alerts for each medication. Furthermore, the pharmacy system is equipped with dose-checking warnings that take into account patient age and/or weight to alert for daily and individual dose limits as outlined by the third-party database.
The Leapfrog tool, however, provides scripted test orders requiring entry of a specific medication dose and frequency on a predefined test patient that bypasses the dosing guidance provided by the ADM; this is not the standard practice for ordering drugs at DUH. Thus, we received a low score for this medication checking category despite the advanced decision support provided by the ADM feature. Because the Leapfrog evaluation tool does not account for alerts provided by neighboring systems, we did not receive acknowledgment of age/weight-based dose limit alerts within the pharmacy system.
Summary Recommendations
Consideration for Alternative Methods
The Leapfrog CPOE evaluation tool is a novel method for assessing the capability of implemented CPOE systems to alert for common prescribing errors for adult and pediatric inpatients. Assessing CPOE functionality in isolation, however, may oversimplify complex healthcare environments, such as those at DUH. The tool may paint a clearer picture of hospital medication safety if test questions are shaped with consideration for the socio-technical factors that influence decisions to tailor off-the-shelf vendor products for optimal performance within an organization. Prevention of prescribing errors may be approached through various avenues based on the interplay of supportive IT systems, as well as the need to balance alerts against the risk of alert fatigue.
In a recent report from the Leapfrog Group, a review of 214 hospitals’ performance on the CPOE evaluation tool showed 52% of medication orders did not prompt warnings, including a third of potentially fatal orders (The Leapfrog Group, 2010c). A similar CPOE evaluation study by Metzger et al. across 62 U.S. hospitals found that 10‚Äì82% of medication test orders prompted system warnings. Vendor software accounted for over a quarter of the CPOE performance variation observed, but differences in performance were apparent even among hospitals using the same vendor product (Metzger, et al., 2010). Technical limitations may require hospitals to consider designing workarounds within CPOE or using alerts in alternative systems.
Given these alternate methods, consumers who consider Leapfrog’s online results may not have a full appreciation for the factors outside of the Leapfrog tool’s review that affect prescribing safety within a complex healthcare environment. Low scores on the evaluation tool may be due to alternative methods used to achieve the same safety benefits. This is particularly important when considering Leapfrog’s CPOE scoring algorithm. Without exceeding a score of 50% on both the drug‚Äìallergy and drug‚Äìdrug interaction checking, the overall CPOE evaluation score cannot surpass “completed the evaluation,” regardless of how well a system performs on all other medication checking categories. For DUH, a score of “fully implemented” potentially could have been received if drug‚Äìdrug interaction checking by our pharmacy system was considered. It seems prudent that these limitations be relayed to and understood by patients and purchasers for this CPOE safety metric to be accurately interpreted.
Expansion of the Leapfrog Tool: Other Medication Safety Areas in CPOE
Additional opportunity exists for Leapfrog to assess the error-trapping capabilities of CPOE systems. Inclusion of such performance may more fully depict the benefits of CPOE being used in an organization beyond prescribing errors alone.
Patient care transitions are known to be susceptible to handoff communication failures that can lead to medication errors and patient harm (Arora, et al., 2005; Forster, et al., 2003; Kripalani, et al., 2007). We use specific electronic order reconciliation forms in CPOE to support the continuity of care when patients transfer to and from ICU and non-ICU locations or between care services. More importantly, we have an ongoing metric in place to monitor policy compliance and hold physicians accountable to this medication reconciliation process in CPOE.
CPOE systems may be utilized to not only protect against order entry errors but also errors of omission and adherence to quality of care. These development efforts are usually made in conjunction with evidence-based performance measures defined by organizations such as The Joint Commission and Centers for Medicare and Medicaid Services. For example, we have developed an electronic advisor to facilitate prescribing of venous thromboembolism prophylaxis on all patients admitted to our hospital, thus reducing order omission for prophylactic therapy and improving the quality of patient care. Hospital measures around community-acquired pneumonia led to the development of a pneumonia order set to guide appropriate antibiotic selection and meet other measure components (The Joint Commission, www.jointcommission.org/PerformanceMeasurement/PerformanceMeasurement/).
In other cases, efforts for improved prescribing and disease management may be centered on high-risk medications (e.g., insulin, opiates) that are known to result in significant patient harm. Our organization designed a sophisticated insulin advisor with advanced decision support using patient-specific parameters to offer tailored insulin regimens, improve blood glucose control, and ultimately prevent adverse events in our inpatients. The Leapfrog tool does not currently detect the organizational use of CPOE for improved care quality, which could be an area for future evaluation and more complete representation of medication prescribing safety within the hospital.
Summary
The Leapfrog Group developed the first evidence-based methodology to evaluate the error-trapping performance of implemented CPOE applications. Their strategy to increase transparency of care delivery and influence healthcare safety and quality through market drivers is noteworthy and should be lauded. At DUH, we agree ongoing testing and enhancement of CPOE is necessary; however, isolated review of CPOE does not account for safety checks performed by other IT systems and workflow processes used by hospitals to minimize medication errors. Acknowledgment of these alternative methods may more wholly represent the care delivered in an institution.
Andrea Long is a clinical pharmacist with the Health Analytics and IT Patient Safety group for Duke Health Technology Solutions (DHTS). She obtained her doctor of pharmacy degree in 2006 and completed a general practice residency at Duke. In addition to patient care activities at Duke University Hospital, Long has assisted in the evaluation of adverse drug events as well as extraction and analysis of electronic data for quality improvement and patient safety metrics. She can be contacted at andrea.long@duke.edu.
Monica Horvath is a senior research analyst with the Health Analytics and IT Patient Safety group for DHTS. She obtained her PhD in molecular biophysics & computational biology in 2005. She serves as the team’s statistics and data analytics expert for evaluating the impact of health IT interventions. She can be contacted at monica.horvath@duke.edu.
Jeanette Jansen is a senior clinical analyst with the computerized provider order entry (CPOE) team. She holds a master of science in nursing, with experience as a pediatric nurse and more than 7 years of experience in CPOE. Jansen was poject lead for the implementation of pediatric CPOE and the ADM. She also contributed to Duke University Hospital’s participation in the CPOE Leapfrog evaluation tool. She can be contacted at janse002@mc.duke.edu.
Jeffrey Ferranti is associate chief information officer for the Duke University Health System and Associate Director of the Duke Center for Health Informatics. He is responsible for the strategic planning of IT initiatives related to patient safety and quality improvement. He is an assistant professor of pediatrics and informatics, and holds a master’s degree in medical informatics. He maintains an active practice in neonatal critical care at Duke University Hospital and Durham Regional Hospital. He can be contacted at ferra007@mc.duke.edu.
Heidi Cozart is the clinical director of CPOE and IT Patient Safety for Duke Medicine. A graduate of Iowa College of Pharmacy in 1994, she began her career in informatics in 1996 and is currently persuing a masters in biomedical informatics. Cozart has been involved in informatics initiatives focused on patient safety and outcomes research, software design, and major clinical system integration and deployment. Her research interests include measuring the impact of IT interventions on patient care processes and outcomes. She can be contacted at cozar011@mc.duke.edu.