High Reliability: Memorial Hermann Health System’s IT Network
By Mitch Work, MPA, FHIMSS, and Chuck Appleby
When your vision is to become a “high reliability” organization in healthcare like those in the nuclear power and commercial airline industries, you’ve set the bar at a height no health system has yet reached. Add a burgeoning uninsured population with diminishing reimbursement in one of the most competitive health marketplaces in the nation and, well, “Houston, we may have a problem.” Memorial Hermann Health System—a Houston-based integrated delivery system with 12 hospitals and more than 20,000 employees serving a 115-mile-diameter region centered on the metro area—has not only set such a bar, its exemplary record in patient safety and clinical quality makes it a good bet to reach it.
Critical to achieving that goal are the organization’s IT network and applications. Memorial Hermann supports first-rate patient care with a robust clinical IT platform: a fully implemented EHR with CPOE supported by clinical decision support and—flying under the radar—an innovative data center and IT network that combines the high speed, dependability, and resiliency demanded by high-reliability organizations. The IT network, which carries data, voice, and video communication, has become the silent IT partner that supports the highly visible entity known as Memorial Hermann Health System.
As part of the largest not-for-profit health system in southeast Texas, Memorial Hermann Texas Medical Center is the teaching hospital for the University of Texas Health Science Center at Houston Medical School. It is one of the nation’s busiest Level 1 trauma centers and is where Arizona Congresswoman Gabrielle Giffords went to recover from her wounds following the recent assassination attempt.
Memorial Hermann has won its share of national recognitions, last year Thomson Reuters, a leading provider of information and solutions to improve the cost and quality of healthcare, named Memorial Hermann one of the top 15 U.S. health systems. Also in 2012, four Memorial Hermann hospitals were listed among America’s 50 Best Hospitals by HealthGrades®, the leading independent healthcare ratings company.
First Do No Harm
Still, as healthcare reform continues to unfold, and the industry moves toward a model based on value rather than volume, Memorial Hermann leadership has set its course on a higher star.
“We’re striving to be the first healthcare system to become a high-reliability organization like commercial aviation and the nuclear industry,” said President and CEO Dan Wolterman. “That means zero harm for patients. In the past 18 months, we’ve awarded 78 ‘Certified Zero’ citations to our hospitals that have demonstrated a minimum of 12 months with zero hospital-acquired infections or other harmful events. We’ve gone months when the entire system has had zero adverse events” (CEO Viewpoint, 2012).
While safety and quality require a multifaceted approach that includes changing work performance behaviors, IT-enabled tools like clinical decision support (CDS) have allowed Memorial Hermann to incorporate “automated prevention bundles” into its EHR to eliminate hospital-acquired infections and other adverse events. That strategy has enabled Memorial Hermann hospitals to rack up more than six-dozen High Reliability Certified Zero Awards (Shabot, 2012).
From the Ground Up
Creating such a high-reliability environment, however, is impossible without an equally highly reliable IT infrastructure. High reliability is built from the ground up, and high-reliability environments by definition demand foundational high-speed IT networks that are up and available 24 hours a day, 7 days a week, 365 days a year. Clinical applications that support life-and-death decisions don’t accommodate downtime.
“Networks are at the foundation of health information technology solutions, much like the heart and circulatory system are for the human body. Without a reliable network, we simply cannot survive. It delivers the blood and oxygen—the information we need—at the point of care,” said Robert Murphy, MD, CMIO at Memorial Hermann.
Also—especially in healthcare—the IT network must have the capacity to grow or “scale” as bandwidth needs expand. As health systems consolidate into larger and larger entities, accountable care organizations (ACOs), value-based purchasing, and population health transform the healthcare landscape, high-performance health systems require powerful software analytics engines, petabyte—whatever happened to terabytes?— databases and fail-safe communication networks for health information exchange in the community. Add bandwidth-devouring medical images, the proliferation of mobile computing, Bring Your Own Device (BYOD), and the need to engage patients electronically and the demand for network capacity is practically unlimited.
The healthcare industry is at an inflection point when IT networks are once again assuming the strategic importance they had two decades ago when hospitals invested heavily in fiber-optic network “backbones” to link computer workstations within the four walls. Today the healthcare enterprise has moved beyond those walls to encompass the entire hospital campus as well as clinics and labs operating throughout the surrounding community. This is not your father’s legacy network.
A Tall Task
Fortunately, while building its clinical IT prowess, Memorial Hermann had the prescience to begin converting its network into a platform for high reliability. The health system uses scalable, high-performance Storage Area Network (SAN) and Local Area Network (LAN) solutions to support its next-generation growth within the data center and campus.
Memorial Hermann has implemented high-capacity core routers and switches, using a new Ethernet fabric technology developed by San Jose, Calif.-based Brocade, to automate, streamline and deliver reliable access to business and patient-critical applications and data.
This infrastructure now supports more than 300 software applications, including the health system’s EHR from Kansas City-based Cerner Corporation, which includes a comprehensive suite of clinical applications for the patient medical record, from physician notes and medication orders to lab results and surgical information. These applications run on Citrix servers that are linked across the enterprise via the network, on the front end by switches and then at the network core by powerful routers that act as high-speed gatekeepers of information between all the LANs and subnetworks in the enterprise.
Desktop computers, laptops, and mobile devices combined account for more than 30,000 points of entry or ports into the network. Each of those represents a user who is one way or another involved in the delivery of care to the several hundred thousand patients that Memorial Hermann sees each year in bothinpatient and outpatient settings.
“Our goal is to keep an environment up and running 24x7x365,” said Amanda Hammel, who as the system executive responsible for all technology, oversees the network infrastructure at Memorial Hermann. “It is such a tall task. No one wants their system down—ever. The network can bring down an entire environment, especially if there is not enough bandwidth,” she says, and that can stop patient care delivery in its tracks. “A fast, stable network is a must-have and is no longer an option. Fortunately, our senior management understands this and has provided the resources necessary for the continued growth of our network operations,” says Hammel.
Eliminating a Data Bottleneck
Bandwidth became an issue in early 2011 when the health system realized its 1 Gigabit Ethernet (GbE) Cisco network was “oversubscribed,” experiencing unacceptably long back-up times and needed to be replaced by a faster, more reliable 10 GbE network. Because Memorial Hermann had since 2005 used a highly reliable SAN solution—creates pools of data storage for servers on the network—and was satisfied with its speed and reliability, the organization opted to replace its aging and bottlenecked network with a new technology.
Following a due-diligence process, which included phone interviews and site visits to nearby organizations using similar solutions for their network infrastructure, Memorial Hermann selected Brocade in July 2011. “I was more comfortable with a company moving from a heavy SAN-experienced environment to IP, rather than a company who is moving from IP to creating a fabric. With Brocade it was a better heritage, built on reliability,” said Mike Romero, director of technical services.
Network engineers and other IT staff from Memorial Hermann were trained by the vendor and an outside consulting company over a 5-month period beginning in fall 2011.Today there are three Memorial Hermann engineers certified on the new system to both train new staff and help maintain the network infrastructure. Implementation began in December 2011 with a “go live” of the new network running the Cerner EHR occurring in February 2012.
Data, voice, and video networks are rarely single-vendor plays. Like the applications that run on them, they must be interoperable in order to help a health system achieve its goal of seamless health information exchange both within a health system and the community at large. This heterogeneity becomes clear in the data center that houses the routers, switches, SAN devices, and other network gear provided by many vendors. Memorial Hermann made sure its new network was built on an open-systems architecture, which not only made it easily interoperable with other vendor technology, but made the conversion to the new network quick and relatively simple.
The new network approach was also appealing because of its innovative Ethernet fabric architecture, a breakthrough in network technology developed over the past 15 years and commercially offered to healthcare in 2012.Ethernet fabric architecture enables a network to act as a single, holistic unit—like a fabric crisscrossed with threads—that connects any port with all possible outputs. This enables organizations to create flatter, virtualized, and converged data networks. Previous network technology was less resilient and generally slower because it allowed more latency or delay.
Virtually the Same
A fabric-based architecture has become a critical part of Memorial Hermann’s healthcare network because of its reliability and speed. However, server virtualization has also become a signature of the new healthcare network because of its efficient resource utilization. Virtualization allows servers to be shared by multiple applications economizing space and reducing costs associated with dedicating a single server to a single application.
Virtualized environments lay the foundation for cloud-based computing and modern IT environments. They replace and simplify the older network architecture that uses Spanning Tree Protocol (STP) which is no longer necessary because the Ethernet fabric appears as a single switch to connectservers, devices, and the rest of the network.Memorial Hermann has become a highly virtualized network environment, with more than 600 virtual servers.
Data Backup, Disaster Recovery
Memorial Hermann also adopted a disaster-recovery solution that enables the health system’s backup data center to be current within 4 hours with the primary data center for specified business-critical applications. This provides two key advantages: it helps assure the key goal of network availability 24/7/365 and serves as a reliable backup source in case of a natural or other disaster. Traditional disaster recovery—also known as business continuity—strategies would typically establish the primary data center as the “hot” site and the backup as the “cold” site. By enabling the backup datacenter to be current within four hours this strategy allows for a near-seamless conversion if one data center goes down without users even being aware of the change.
“The goal is 100-percent uptime,” said Chris Kar, the network engineering manager at Memorial Hermann, who prior to Memorial Hermann managed the data networks for the U.S. House of Representatives. “The Certified Zero systems are always running, and we’re the transport. Memorial Hermann is so deeply entrenched in technology, there is not one segment of the health system not touched by the network,” he said, adding that the only time the network would become visible to most people is if it were to go down. With the new network there’s only one way to describe its performance: “It works. It just works.”
Current Metrics; Future Directions
While Memorial Hermann continues to convert its legacy network, the health system is already reaping benefits. Data backup—remember we’re talking petabytes of critical healthcare patient information—now takes only four hours using the new 10 GbE network, compared to 30 hours previously with the old 1-GbE system.
Since implementing the new technology in its new data center, Memorial Hermann has experienced nearly zero downtime. That means the main EHR application is available 24/7 for Memorial Hermann’s clinician users. “The biggest benefit is speed and uptime. That’s the main reason we went with Brocade,” said Ty Hall, network architect.
Another key reason for upgrading the backup data center and network was the promise of future capability. Data storage now has the capability to grow exponentially to accommodate the increasing number of high-bandwidth medical images, addition of new physician practices, and the burgeoning requirements of physician documentation and quality reporting—and it’s growth that cannot be quantified. “We’re in a time we’ve never been,” says Chris Kar. “We have no idea what the future demand for data will be except that it’s continuing to accelerate. As demand for managing population health grows, the volume of the required data also grows.”
Future growth will include implementation of even faster networks. Mike Romeropredicted that in 3 years, Memorial Hermann would implement 10 GbE data transmission between hospitals, the data center will have the capacity for 100 GbE transmission, physician offices 100 Mbs, and imaging centers 10 GbE. He expects the health system will routinely conduct video conferencing, and provide voice-to-text dictation and desktop-to-desktop video conferencing at the bedside.
While future data growth seems unlimited, so too are the possibilities. Romero recalls that 20 years ago he had to turn down requests for network applications because the network was so limited. “We are at a most exciting time in healthcare. I can do nearly everything I am asked to do because finally the bandwidth is there.”
Mitch Work is president of The Work Group, Inc., a Chicago area-based healthcare technology marketing company specializing in emerging technologies. He may be reached at mitch@workgroupinc.net
Chuck Appleby is senior vice president of The Work Group Inc and a veteran journalist who has covered healthcare IT for more than 25 years. He may be reached at chuck@workgroupinc.net
References
CEO Viewpoint. (2012). The Scottsdale Institute.
Shabot, M. M. (2012, October 25). Memorial Hermann as a high reliability organization.Paper presentedat Scottsdale Institute Fall Forum 2012.