Cenegenics Builds Resilient Cisco Network to Support Its Growing Practice

It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about.”
– Dror Nisenbaum, IT Manager, Cenegenics

Pressures of Growth on IT

Dror Nisenbaum, IT Manager at Cenegenics, clearly remembers his first day of work. He arrived at eight o’clock in the morning, sat down and the phone rang before he could take the first sip of coffee. It was a disgruntled user who had been waiting a month to fix an IT problem. Nisenbaum walked him through a solution right there on the phone. The user was pleasantly surprised and relieved, while for Nisenbaum it was a not-so-subtle hint of the road ahead. In fact, after spending four intense months working through the IT support backlog, he turned his attention to the challenge of redesigning and rebuilding the network and other IT infrastructure to support the company’s fast growth.

Cenegenics, based in Las Vegas, Nevada, is the largest age management practice with 20 medical centers in the United States serving more than 20,000 patients. Age management emphasizes prevention, as opposed to treating disease, and employs nutrition, exercise, supplements and hormone treatment under the care of a physician to maximize the quality of life for its patients as they age. Cenegenics is well known for ads showing one of its physicians, Dr. Jeff Life, a 72-year old man with the toned body of a youthful muscle builder.

Cenegenics experienced many years of rapid growth. From 2006 to 2012, it expanded from 3 medical centers in Las Vegas, Boca Raton and the Carolinas to 20 medical centers throughout the United States. The number of corporate staff at its headquarters in Las Vegas quadrupled. By any measure, the practice was successful and growing.

However, the expansion and growth put a strain on its aging IT infrastructure. Ten years ago, Cenegenics had a flat network with HP equipment supporting its users and applications. They added a voice over IP (VOIP) system. The VOIP provider installed Cisco networking equipment with power over Ethernet (POE) support, so they had two networks connected via uplinks, one for data and the other for voice. Unfortunately, the features and configurations were inconsistent and many problems arose. Phones did not power up if connected to the HP switches, and computers would not acquire the right IP addresses if connected to the Cisco switches. There were many dropped calls because the HP switches did not support QoS. Data storms flared up on a couple of occasions because of misconfigurations between the networks. The IT contractor who managed Cenegenic’s network was an HP reseller trained to support HP networking equipment, but not Cisco. They had to manufacture artificial phone problems and call the VOIP provider to get Cisco support. “It was very difficult. It was the perfect storm,” said Nisenbaum.

After the support backlog was finally under control and response times were reasonable, Nisenbaum focused on diagramming and labeling the entire network, including switches, cables, distribution points and servers. “I had to redo all of the wiring closets because it looked like a spaghetti ball,” he said. The jumble of cables and switches was so confusing they resorted to pulling cables after business hours and walking to the adjacent suite to see which device turned off. The result of the effort was a diagram that provided a comprehensive view of the infrastructure and identified the single points of failure.

Building a Consolidated, Highly Resilient Network

Next they developed a plan for a consolidated, highly-available, virtualized infrastructure. It included two Cisco Catalyst 3560-X switches as the core of the network and a Cisco 3945 Integrated Services Router to replace a SonicWALL router and one other that were too small for the expanded user load. The two 3560-X switches had redundant power supplies and were paired up for failover. Six existing Cisco Catalyst 3560s would serve as edge switches. Connections around the headquarters building would have four strands trunked in pairs to make a redundant fiber ring. Pipes between suites in the building would carry not more than one fiber optic cable to eliminate the risk of a severed pipe disrupting communication. The design also included three powerful Dell servers with dual six-core processors running VMware and a Dell EqualLogic iSCSI SAN with dual controllers connected via the 3560-X core switches.

Nisenbaum felt strongly that the network should be consolidated on Cisco equipment, not a mixed-vendor environment. “It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about,” he said. Nisenbaum is Cisco-certified and very comfortable using their equipment. Though he had some experience with HP and SonicWALL equipment, he thought some of their configuration procedures were unnecessarily complicated. “Having multiple vendors brings so much complexity… I wanted to consolidate everything to one vendor, so I can train my employees and make sure the team we have in place can manage it all.”

The plan was presented to Cenegenics’ executive board. The project cost was in the six figures, a substantial investment for a company of that size. Nisenbaum’s approach was to show them all the points of failure in their existing infrastructure. “If any of these devices goes down, this is the catastrophic effect. How much is it going to cost you per day if this doesn’t work? That was my pitch. I came in six months after one of their servers crashed and they were down for almost a week. What I was saying was very fresh in their minds,” he said. The board decided to approve and fund the project.

I can sum it up with one statement:  I can sleep at night.”
– Dror Nisenbaum

Uptime Makes Everyone Happy

Installation of the new equipment took four months because some aspects of the transition had to be handled carefully to avoid disrupting users. Today the company’s applications and users run on the rebuilt and highly resilient network as well as server and storage infrastructure. It has been 16 months since a server went down. While individual components have failed and had to be replaced, such as a GBIC on a switch, the redundancy of the design ensured that users did not experience downtime. Executives and staff are pleased because they can focus on running the practice and serving patients with a stable IT infrastructure supporting them.

Asked about how he feels about the situation now, Nisenbaum replied, “I can sum it up with one statement: I can sleep at night. When I had all the aging hardware and there was such a support backlog, I didn’t have a personal life. Now that things have slowed down, it gives me the ability to see how I can improve Cenegenics as a whole.”

 Copyright © 2012 Apropos LLC. All rights reserved.

At Banfield Pet Hospital, PACS and Telemedicine Bridge the Distance across Its Hospital Network

We needed a way to consolidate this image data and make it universally available across our distributed network of hospitals.”
– Dan Baldock, Senior VP and CIO, Banfield Pet Hospital

Banfield Pet Hospital was challenged by all of its digital x-ray and ultrasound images stored locally at each of 500 disparate locations across the country. They had no way to share those images among all of the 780 Banfield pet hospitals. There was no backup or disaster recovery plan in place, so the medical data was at risk of loss if a hard drive failed. Telemedicine (getting a second opinion from a radiologist working remotely) was slow, expensive, and many veterinarians were not taking advantage of the service to improve patient care. “We needed a way to consolidate this image data and make it universally available across our distributed network of hospitals,” said Dan Baldock, Senior VP and CIO of Banfield Pet Hospital.

Banfield Pet Hospital is a network of 780 veterinary hospitals located throughout the United States. Banfield partners with PetSmart, the nation’s largest pet-related retailer, and operates pet hospitals inside PetSmart stores.

First Step: Centralize and Protect Digital Images

To solve these challenges, Banfield implemented a centralized PACS system (Picture Archiving and Communication System) at its headquarters in Portland, Oregon. They selected the open source software ClearCanvas as both the back-end archiving system and as the front-end viewer. Banfield had the support of the team at ClearCanvas and leveraged their existing in-house resources and expertise to develop and deploy the system. The only additional staff they needed to hire was a single PACS Administrator to aid in the setup and ongoing administration of the system.

Banfield set up the infrastructure for all 500 digital x-ray and ultrasound machines across the country to transmit images over a WAN to their centralized servers for long-term storage. They set up redundant, mirrored server and storage infrastructure with daily and monthly backup processes to protect the data. In addition, once that data was centralized, they implemented a disaster recovery plan for failover to a remote site in Arizona. Now the medical imaging data (x-ray and ultrasound images) is protected and readily accessible.

Data centralization also allowed Banfield to implement nationwide image sharing. Many clients are very mobile across the country. The hospital has countless stories of truckers that drive nationwide with their pets or snowbirds that go south with their pets for the winter. Having the ability for images taken at any Banfield hospital to be viewed with the click of a button at any other Banfield hospital has improved dramatically the continuity of care their veterinarians could provide.

Second Step: Telemedicine

With the foundational PACS system in place, Banfield then implemented a robust tele­medicine system. To do this, they custom-built a RIS system (Radiology Information System) to track the requests for telemedicine consults, allow the radiologist or other specialist to view the cases, and communicate the consultation results back to the requesting veterinarian and team. They partnered with various veterinarian telemedicine vendors, even designing a custom integration with one to send cases that could be read in their existing system.

By building this RIS system, they were able to improve the turnaround time of telemedicine consultation results getting back to the hospital to an average of 30 minutes for STAT cases (emergency cases) and 90 minutes for routine cases. Historically the turnaround time was hours to days. This made an enormous impact at the point of care for the veterinarians and their clients and patients to have a specialist give a second opinion as needed on a case to improve the treatment decision-making process.

In the end, Banfield was able to save millions of dollars in the first few years by selecting the ClearCanvas technology and leveraging their existing resources. Since the initial imple­mentation, they have also expanded the PACS system to include digital dental x-ray images after adding 60+ digital dental x-ray units to its hospitals. The system was built to be very scalable so they could add different types of imaging equipment and allow for the growth of new hospitals. Banfield has been opening about 50 new hospitals per year and will continue to do so. They are also in the process of implementing another feature, DICOM CD burn, using another ClearCanvas product. “The solution we chose has really allowed us to grow in many directions,” added Baldock.

Next Step: Integration with EMR

The future for PACS, RIS, telemedicine and all imaging systems at Banfield Pet Hospital holds even greater promise. The next step is to integrate the imaging systems with Banfield’s proprietary EMR (electronic medical record system) called PetWare. The goal will be to integrate images that reside in ClearCanvas PACS and radiology telemedicine consultation results directly into the EMR. The solution will also allow them to integrate images into their client/patient portal system, so clients can view their pets’ images.

 Copyright © 2011 Apropos LLC. All rights reserved.