Cenegenics Builds Resilient Cisco Network to Support Its Growing Practice

It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about.”
– Dror Nisenbaum, IT Manager, Cenegenics

Pressures of Growth on IT

Dror Nisenbaum, IT Manager at Cenegenics, clearly remembers his first day of work. He arrived at eight o’clock in the morning, sat down and the phone rang before he could take the first sip of coffee. It was a disgruntled user who had been waiting a month to fix an IT problem. Nisenbaum walked him through a solution right there on the phone. The user was pleasantly surprised and relieved, while for Nisenbaum it was a not-so-subtle hint of the road ahead. In fact, after spending four intense months working through the IT support backlog, he turned his attention to the challenge of redesigning and rebuilding the network and other IT infrastructure to support the company’s fast growth.

Cenegenics, based in Las Vegas, Nevada, is the largest age management practice with 20 medical centers in the United States serving more than 20,000 patients. Age management emphasizes prevention, as opposed to treating disease, and employs nutrition, exercise, supplements and hormone treatment under the care of a physician to maximize the quality of life for its patients as they age. Cenegenics is well known for ads showing one of its physicians, Dr. Jeff Life, a 72-year old man with the toned body of a youthful muscle builder.

Cenegenics experienced many years of rapid growth. From 2006 to 2012, it expanded from 3 medical centers in Las Vegas, Boca Raton and the Carolinas to 20 medical centers throughout the United States. The number of corporate staff at its headquarters in Las Vegas quadrupled. By any measure, the practice was successful and growing.

However, the expansion and growth put a strain on its aging IT infrastructure. Ten years ago, Cenegenics had a flat network with HP equipment supporting its users and applications. They added a voice over IP (VOIP) system. The VOIP provider installed Cisco networking equipment with power over Ethernet (POE) support, so they had two networks connected via uplinks, one for data and the other for voice. Unfortunately, the features and configurations were inconsistent and many problems arose. Phones did not power up if connected to the HP switches, and computers would not acquire the right IP addresses if connected to the Cisco switches. There were many dropped calls because the HP switches did not support QoS. Data storms flared up on a couple of occasions because of misconfigurations between the networks. The IT contractor who managed Cenegenic’s network was an HP reseller trained to support HP networking equipment, but not Cisco. They had to manufacture artificial phone problems and call the VOIP provider to get Cisco support. “It was very difficult. It was the perfect storm,” said Nisenbaum.

After the support backlog was finally under control and response times were reasonable, Nisenbaum focused on diagramming and labeling the entire network, including switches, cables, distribution points and servers. “I had to redo all of the wiring closets because it looked like a spaghetti ball,” he said. The jumble of cables and switches was so confusing they resorted to pulling cables after business hours and walking to the adjacent suite to see which device turned off. The result of the effort was a diagram that provided a comprehensive view of the infrastructure and identified the single points of failure.

Building a Consolidated, Highly Resilient Network

Next they developed a plan for a consolidated, highly-available, virtualized infrastructure. It included two Cisco Catalyst 3560-X switches as the core of the network and a Cisco 3945 Integrated Services Router to replace a SonicWALL router and one other that were too small for the expanded user load. The two 3560-X switches had redundant power supplies and were paired up for failover. Six existing Cisco Catalyst 3560s would serve as edge switches. Connections around the headquarters building would have four strands trunked in pairs to make a redundant fiber ring. Pipes between suites in the building would carry not more than one fiber optic cable to eliminate the risk of a severed pipe disrupting communication. The design also included three powerful Dell servers with dual six-core processors running VMware and a Dell EqualLogic iSCSI SAN with dual controllers connected via the 3560-X core switches.

Nisenbaum felt strongly that the network should be consolidated on Cisco equipment, not a mixed-vendor environment. “It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about,” he said. Nisenbaum is Cisco-certified and very comfortable using their equipment. Though he had some experience with HP and SonicWALL equipment, he thought some of their configuration procedures were unnecessarily complicated. “Having multiple vendors brings so much complexity… I wanted to consolidate everything to one vendor, so I can train my employees and make sure the team we have in place can manage it all.”

The plan was presented to Cenegenics’ executive board. The project cost was in the six figures, a substantial investment for a company of that size. Nisenbaum’s approach was to show them all the points of failure in their existing infrastructure. “If any of these devices goes down, this is the catastrophic effect. How much is it going to cost you per day if this doesn’t work? That was my pitch. I came in six months after one of their servers crashed and they were down for almost a week. What I was saying was very fresh in their minds,” he said. The board decided to approve and fund the project.

I can sum it up with one statement:  I can sleep at night.”
– Dror Nisenbaum

Uptime Makes Everyone Happy

Installation of the new equipment took four months because some aspects of the transition had to be handled carefully to avoid disrupting users. Today the company’s applications and users run on the rebuilt and highly resilient network as well as server and storage infrastructure. It has been 16 months since a server went down. While individual components have failed and had to be replaced, such as a GBIC on a switch, the redundancy of the design ensured that users did not experience downtime. Executives and staff are pleased because they can focus on running the practice and serving patients with a stable IT infrastructure supporting them.

Asked about how he feels about the situation now, Nisenbaum replied, “I can sum it up with one statement: I can sleep at night. When I had all the aging hardware and there was such a support backlog, I didn’t have a personal life. Now that things have slowed down, it gives me the ability to see how I can improve Cenegenics as a whole.”

AIT_Profiles_Blogocon_small

 Copyright © 2012 Apropos LLC. All rights reserved.

At Banfield Pet Hospital, PACS and Telemedicine Bridge the Distance across Its Hospital Network

We needed a way to consolidate this image data and make it universally available across our distributed network of hospitals.”
– Dan Baldock, Senior VP and CIO, Banfield Pet Hospital

Banfield Pet Hospital was challenged by all of its digital x-ray and ultrasound images stored locally at each of 500 disparate locations across the country. They had no way to share those images among all of the 780 Banfield pet hospitals. There was no backup or disaster recovery plan in place, so the medical data was at risk of loss if a hard drive failed. Telemedicine (getting a second opinion from a radiologist working remotely) was slow, expensive, and many veterinarians were not taking advantage of the service to improve patient care. “We needed a way to consolidate this image data and make it universally available across our distributed network of hospitals,” said Dan Baldock, Senior VP and CIO of Banfield Pet Hospital.

Banfield Pet Hospital is a network of 780 veterinary hospitals located throughout the United States. Banfield partners with PetSmart, the nation’s largest pet-related retailer, and operates pet hospitals inside PetSmart stores.

First Step: Centralize and Protect Digital Images

To solve these challenges, Banfield implemented a centralized PACS system (Picture Archiving and Communication System) at its headquarters in Portland, Oregon. They selected the open source software ClearCanvas as both the back-end archiving system and as the front-end viewer. Banfield had the support of the team at ClearCanvas and leveraged their existing in-house resources and expertise to develop and deploy the system. The only additional staff they needed to hire was a single PACS Administrator to aid in the setup and ongoing administration of the system.

Banfield set up the infrastructure for all 500 digital x-ray and ultrasound machines across the country to transmit images over a WAN to their centralized servers for long-term storage. They set up redundant, mirrored server and storage infrastructure with daily and monthly backup processes to protect the data. In addition, once that data was centralized, they implemented a disaster recovery plan for failover to a remote site in Arizona. Now the medical imaging data (x-ray and ultrasound images) is protected and readily accessible.

Data centralization also allowed Banfield to implement nationwide image sharing. Many clients are very mobile across the country. The hospital has countless stories of truckers that drive nationwide with their pets or snowbirds that go south with their pets for the winter. Having the ability for images taken at any Banfield hospital to be viewed with the click of a button at any other Banfield hospital has improved dramatically the continuity of care their veterinarians could provide.

Second Step: Telemedicine

With the foundational PACS system in place, Banfield then implemented a robust tele­medicine system. To do this, they custom-built a RIS system (Radiology Information System) to track the requests for telemedicine consults, allow the radiologist or other specialist to view the cases, and communicate the consultation results back to the requesting veterinarian and team. They partnered with various veterinarian telemedicine vendors, even designing a custom integration with one to send cases that could be read in their existing system.

By building this RIS system, they were able to improve the turnaround time of telemedicine consultation results getting back to the hospital to an average of 30 minutes for STAT cases (emergency cases) and 90 minutes for routine cases. Historically the turnaround time was hours to days. This made an enormous impact at the point of care for the veterinarians and their clients and patients to have a specialist give a second opinion as needed on a case to improve the treatment decision-making process.

In the end, Banfield was able to save millions of dollars in the first few years by selecting the ClearCanvas technology and leveraging their existing resources. Since the initial imple­mentation, they have also expanded the PACS system to include digital dental x-ray images after adding 60+ digital dental x-ray units to its hospitals. The system was built to be very scalable so they could add different types of imaging equipment and allow for the growth of new hospitals. Banfield has been opening about 50 new hospitals per year and will continue to do so. They are also in the process of implementing another feature, DICOM CD burn, using another ClearCanvas product. “The solution we chose has really allowed us to grow in many directions,” added Baldock.

Next Step: Integration with EMR

The future for PACS, RIS, telemedicine and all imaging systems at Banfield Pet Hospital holds even greater promise. The next step is to integrate the imaging systems with Banfield’s proprietary EMR (electronic medical record system) called PetWare. The goal will be to integrate images that reside in ClearCanvas PACS and radiology telemedicine consultation results directly into the EMR. The solution will also allow them to integrate images into their client/patient portal system, so clients can view their pets’ images.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

Credit Union Takes Unorthodox Approach to Developing IT System for Member Care

We stress the point that there is no such thing as an IT project. Every project starts with a vision for the member experience that we desire and then works backwards to the technology.”
– Butch Leonardson, Senior VP and CIO, BECU

The credit union BECU took an unorthodox approach to defining its next-generation IT system. BECU sent 10 people from its member solutions group whose primary job was engaging with credit union members to the Disney Institute in Florida for professional training and to develop an IT vision. These were non-technical and non-senior management personnel. “Their charter was to come back with a dream of a member care framework for how we can provide an exceptional experience to our members,” said Butch Leonardson, Senior VP and CIO of BECU.

BECU, headquartered in Tukwila, Washington, is a community charter credit union in the state of Washington. It has 50 branches, 1,000 employees and 730,000 members, the majority of which are clustered around the Interstate 5 corridor in Western Washington.

While it may be unusual to assign staff other than IT and senior management with the task of proposing a major new IT system, it was consistent with BECU’s “outside-in” IT philosophy. Leonardson elaborated, “We stress the point that there is no such thing as an IT project. Every project starts with a vision for the member experience that we desire and then works backwards to the technology.” From this perspective, it made sense to start the IT project with the personnel who were most directly engaged with credit union members.

More to the point, it worked. The group came back with a compelling vision for a member care framework consisting of three parts: Member View which presents a holistic view of each member’s accounts and transactions; Member Interaction Tracking which presents a current summary of all communications and interactions with the credit union; and Member Access which automatically ranks members based on financial metrics and determines their eligibility for products and rates.

Member Care Framework

To implement this framework, BECU needed a software platform that could bring together information from many disparate applications and present it in a unified manner. They selected Microsoft Dynamics CRM because it seemed lighter weight and more developer friendly and cost less than competitive alternatives. They also liked Microsoft’s roadmap for developing and evolving the product.

Member View was the first component they developed. Prior to Member View, when a credit union member walked into a branch or called on the phone, the member consultant or call center representative did not have an up-to-date, 360-degree view of that person’s transactional activity. They could get it, but it would take several minutes of cutting and pasting across multiple applications to assemble a complete picture because the transactions came from a variety of channels, such as online, phone, mobile and ATM. Like most medium-sized banks, BECU used commercially-available software and transactional data was spread across many applications supporting these channels. Member View brought all of that data together in a holistic view.

Member Interaction Tracking followed. It presented a similar holistic view of communications between members and the credit union. “If you have a conversation with us or you attempt to do something online or on the phone through our integrated voice response system, we know about the conversation and we also know where you dropped off attempting to do something. So when you reach a representative, that representative knows exactly where you abandoned the automated process,” said Leonardson. Therefore callers do not have to re-identify themselves or re-explain what they were attempting to do before switching to a live representative.

The third component, Member Access, made it easier to present each member with a customized set of products and rates. By leveraging business intelligence (BI) software to analyze financial metrics such as account balances, credit scores, and how long that person has been with the credit union, Member Access automatically determined a member’s eligibilities.

Proof Is in the Pudding

When the member care framework was complete, it presented a current summary of member transactions, interactions and relevant product eligibilities in a single view. Member consultants loved it because it made their jobs easier and streamlined interactions with members. BECU even won an IT award for the system. What about the credit union members themselves? While they do not interact directly with the system, they seem to like their experience with BECU. The credit union’s net promoter score, which measures how likely members are to recommend it to friends and family, is in the 75% range as compared to 25 to 35% for major banks, according to Leonardson. BECU also adds 7,000 to 8,000 new members per month. The proof is in the pudding, so to speak.

Toward One-Click Fulfillment

Yet BECU is not standing still. After using the system for 2 or 3 years, member consultants began to ask if they could also execute transactions directly within Member View. Now BECU is layering a portal on top its transactional applications and integrating them with Member View. “It solves the problem that most branches and call centers worldwide have, which is whatever you want me to do, Mr. Customer, I need to go into that application to get it done. Car loans, mortgages, Visa cards are all different systems. Now this portal will homogenize everything and make it all look like one environment,” said Leonardson.

Ultimately BECU’s wants to achieve what they call “one-click fulfillment.” This refers to an end-to-end digital enterprise that eliminates the need for written forms and signatures. The goal is to minimize process and maximize engage­ment with credit union members.Butch Leonardson

We think what we do as leaders is more than lead functions. We lead hearts and minds.”
– Butch Leonardson

Hearts and Minds

Leonardson considers the journey of developing this member care framework to be a tangible aspect of his greater concern as CIO for the IT organization’s shared values and morale: “Do I have seventy five people who wake up in the morning and say, ‘Wow, I have a great gig.’ I think a vast majority of people have a desire to be valued, to be in the middle of something important. And if you can provide that authentically, you are going to have a great organization. That is the center of my gravity as a leader.”

“Somebody called me a unicorn CIO because the stuff I think about is so different from most CIOs. If you looked at the org chart for our IT organization, it would look very normal. We have two VPs and seven managers. There is nothing unique about the way we are organized. We think what we do as leaders is more than lead functions. We lead hearts and minds,” he said.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

Virtual Machines, Virtual Desktops and Zero Clients Deliver Real Benefits for PAE Consulting Engineers

We had all this hardware running and were going to replace it every three to five years, but it wasn’t really getting used. We were never going to wear it out.
– Dustin Rowe, IT Manager and Technology Planner,
PAE Consulting Engineers

The Drive to Virtualize

For PAE Consulting Engineers, the drive to virtualize began with realizing that its distributed server architecture was only 10% utilized. The company was running 20 application workloads on eight physical servers. Since mission-critical applications were distributed among the servers, this architecture offered a reasonable degree of high availability. Restarting one application, such as Microsoft Exchange, did not require bringing down other critical applications, such as Microsoft SQL Server or Active Directory. The downside was that these servers were poorly utilized. As servers are normally replaced every few years, the company was locked into a cycle of overbuying hardware and recycling still-usable equipment. “Virtualization takes advantage of that downtime. That is what drove us forward,” said Dustin Rowe, IT Manager and Technology Planner for PAE Consulting Engineers. “We had all this hardware running and were going to replace it every three to five years, but it wasn’t really getting used. We were never going to wear it out.”

At the same time, the company had about 35 traditional desktop PCs with Intel Pentium 4 processors that had reached end-of-life and needed to be replaced. These PCs ran applications with moderate processing requirements, such as email, web browsing and project management. Here they saw an opportunity to also consider virtual desktop infrastructure (VDI) instead of a typical PC upgrade. “As we moved forward, we realized that by virtualizing our application servers, we would free up hardware that could be used to explore virtual desktop systems, which we thought might be able to meet the needs of these people with traditional desktops,” said Rowe.

PAE Consulting Engineers, Inc., is a mechanical and electrical engineering firm that specializes in sustainable, efficient, “green” building design. The company is based in Portland, Oregon, and has a satellite office in San Francisco, California.

Virtual Machines, Virtual Desktops and Zero Clients

Working with a local computer consulting company called Tech Heads, PAE Consulting Engineers looked closely at various options for server virtualization and VDI, including Microsoft Hyper-V, Citrix and VMware. At the time Hyper-V lacked some features they wanted. Citrix would have required eight servers to support the application workloads and desktop virtualization, while VMware could do it with half the number. They also found VMware to be very easy to use. Performing physical-to-virtual migrations and provisioning virtual machines were straightforward, simple tasks. For a small business with one IT manager handling the entire infrastructure, this was an important factor. Furthermore, pricing for virtualization software had dropped a point where it was realistic for the company’s IT budget.

The company chose VMware vSphere for server virtualization, VMware View for desktop virtualization and ClearCube I9422 Zero Clients at the desktop. VMware vSphere is the leading platform for server virtualization. VMware View is a desktop virtualization platform that runs Windows sessions centrally on a server and streams the display data over the network to client devices. In this case, the client devices were ClearCube Zero Clients that reside at each user’s desk and perform the role of a PC except for storage and processing. These devices communicate with the View server over the network using a streamlined PCoIP display protocol that facilitates a fluid screen presentation. They provide a single restart button and support dual monitors, keyboard, mouse and peripherals.

Easier to Manage and Less Costly to Operate

With the new virtual environment, PAE Consulting Engineers consolidated its original eight application servers down to two VMware servers running the same 20 application workloads in virtual machines. This was a major improvement in hardware utilization and delivered cost savings that would carry forward into the future as they only needed to replace two servers instead of eight when they reached end-of-life. Three additional servers ran VMware View to support nearly 30 virtual desktops. Two of the servers were required to support the workload and the third was for redundancy and failover purposes. Rather than purchase new hardware, they added processors, memory and network cards to the existing servers and redeployed them, thus minimizing costs for the transition to the virtual infrastructure.

They also added two IBM DS3500 Express storage systems, dedicating one for the application servers and the other for the virtual desktop system. Beyond storage consolidation and high availability, SAN storage allowed them to derive more benefits from server virtualization. For instance, the VMware vMotion feature in a SAN environment can migrate applications live between servers without disrupting user access. “We moved Microsoft Exchange in the middle of the work day with seventy people running email through it, and no one could even tell it moved to a different server,” he said.

At the relatively small scale of the company’s virtual desktop environment, hardware costs as compared to traditional PCs were break-even. While Zero Clients cost less than PCs and have a longer expected life of 7 to 10 years since they lack moving parts, the VMware View servers and SAN storage also factored in to the total cost.

The greatest benefit PAE Consulting Engineers experienced from VDI was simplified management and reduced IT supports costs. Rowe continued, “Now IT has one place to go take care of hardware. Aside from training and educating users on the end point, we nearly never have to go to the desk. That foot traffic is an incredible amount of time for the IT staff. You’re up and constantly going to a different machine to do something physically that is unnecessary with the technology we are using these days. We can do everything from our remote session. And as you scale the business, it becomes more and more valuable – by an order of magnitude.”

Windows functionality for virtual desktop users was identical to traditional PCs, though it took time for users to adjust to some differences in the interface. For instance, when restarting a computer, users only saw a welcome screen instead of one proclaiming that Windows is restarting, so they wondered if the computer was working correctly. While issues like this did not affect productivity, they required some changes in how users interacted with the system. Most workers used ClearCube Zero Clients as client devices. A handful of people used an Apple laptop or mobile device as their main computer and ran a View software client for occasional Windows access.

A surprising benefit of virtualization was the ability to extend the useful life of server hardware. Normally servers are replaced when the warranty expires after 3 to 5 years because warranty costs becomes prohibitive as the hardware ages. But with redundant servers and components and the ease of moving workloads around in a virtual environment, it becomes possible to continue using servers outside of warranty until they fail, then replace them. “We can be more flexible and let the hardware push us to replace, not the warranty,” said Rowe. However, a thoughtful recovery plan should be in place to do this. The company knows which of its applications are mission-critical and which can afford a little downtime in case of a hardware failure and recovery scenario.

Virtualization is a big deal. It is literally everything that it proposed itself to be.”
– Dustin Rowe

Another benefit they experienced was energy efficiency and lower electricity costs. The ClearCube Zero Clients consume only 15 W of power compared to 100 W for the PCs they replaced, which reduced power and heat dissipation at the desktop by 85% and eliminated noise from fans and hard drives. The tasks of computer processing and storage were transferred to three 300 W servers and a SAN storage array. Even including this hardware in the calculation, the power consumption per desktop was still reduced by approximately 50%.

Potentially More to Come

to the future, Rowe noted that the industry is developing server offload cards for graphics processing that could open the door to run CAD and sophisticated graphic design applications in the virtual desktop environment. Currently the workers using these applications have workstations with 3D graphics cards that perform the heavy-duty processing because it would be too slow and choppy to run these applications on a central server. But if high-performance graphics processing were available for the View servers, the company could consolidate the remainder of its desktops and further streamline the IT infrastructure.

“Personally it is very exciting to see legitimate advancements in technology,” he added. “For a long time, there hasn’t been anything to really change the way we think about how hardware should operate and when we’re going to replace it. But virtualization is a big deal. It is literally everything that it proposed itself to be. It just takes using it a while to really see.”

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

Transtector Reins in Printing with a Shared, Consolidated Solution from Ricoh and IKON

Brian BradburyThey were spending a significant amount of money on toner cartridges, repairs and buying the actual printers.
– Brian Bradbury, Business Partner in IT organization at Smiths Group

*

A Proliferation of Printers

At Transtector headquarters in Hayden Lake, Idaho, print devices were numerous and abundant. “We had 76 different print devices at this location,” said Brian Bradbury, Business Partner in the IT organization at Smiths Group, Transtector’s parent company. “It was a wide variety of items, from laser printers to scanners, fax machines to desktop printers. There must have been six or eight different brand names and as many as forty or fifty different models of devices. So you can imagine what it was like to try to keep track of the toner cartridges and repairs of those devices as they needed assistance.”

Transtector Systems is a manufacturer of power protection products, including signal protection, EMP/EMI filters, power distribution units (PDU) and uninterruptible power supplies (UPS). Transtector is a subsidiary of Smiths Group, a global technology conglomerate based in London, England, that operates in more than 50 countries.  Smiths’ annual revenue is $7 billion, and Transtector represents $80 million of that amount.

A proliferation of printers is common in enterprises, especially when purchasing decisions are decentralized. You can imagine how it might happen: The sales team decides to buy inkjet printers for their desks. The accounting department has its own laser printer, and accounts receivable and accounts payable each have dedicated fax machines. Marketing invests in a scanner and color laser printer. And so it goes until print devices are ubiquitous.

From the perspective of Transtector’s users, the situation seemed to work well. The company had an agreement with a VAR to receive discounted pricing for top-model, brand-name printers. When printer ink ran low, employees used an expense report to buy new cartridges, and when a print device wore out, they bought a new one. This ad hoc approach met their basic printing needs.

“It is only when you reviewed the costs from the top-down that you see the impact to the overall organization,” noted Bradbury. “They were spending a significant amount of money on toner cartridges, repairs and buying the actual printers.”

Shared, Consolidated Printing Solution on Lease

As a member of Smiths centralized IT organization, Bradbury’s job was to present corporate best practices and special pricing agreements to Transtector and help manage implementations. Smiths had negotiated a contract with the printer manufacturer Ricoh to visit Smiths’ portfolio companies, review their printer needs and spending and supply consolidated printing “kits” on lease that would deliver hard savings. The kits included Ricoh Aficio multi-function printers as well as ongoing equipment maintenance and toner.

All I do is buy the paper and pay the use fee on the lease, and it saves five thousand dollars a month as we move forward.
– Brian Bradbury

A team from IKON, Ricoh’s document management subsidiary, performed an assessment of Transtector and proposed a kit that would consolidate print devices dramatically. Transtector approved it and the new solution was rolled out. “We ended up with 12 devices instead of 76. They are multi-function devices placed strategically throughout our facility. I don’t maintain it. All I do is buy the paper and pay the use fee on the lease, and it saves five thousand dollars a month as we move forward,” said Bradbury.

As part of the maintenance process, IKON monitored printer usage and performance. Their technicians noticed that 3 of the 12 devices were outputting higher than anticipated print counts. So IKON swapped them out for larger models that could maintain higher levels of printing without overtaxing the equipment. “They do a fantastic job of maintaining the process, reviewing it and making sure each location has the equipment necessary for the particular demand,” he said.

Everyone Pleased, Eventually

Transtector executives and financial staff were pleased with the $5,000 monthly savings and positive contribution to profit margins, though users initially had difficulty seeing the value of removing the printers from their desk or immediate work area and putting one 20 feet down the hall. It just seemed inconvenient. However, they eventually warmed up to the value of higher availability and faster repair times. If a print device had a problem, users no longer had to roll up their sleeves and try to troubleshoot it. Instead they submitted a helpdesk ticket and Ricoh/IKON technicians took care of it. They could send print jobs to next closest printer in the meantime. After a few months of experiencing the consistency and low hassle of a shared and externally-maintained printing solution, users also felt positive about it.

Successful IT projects like this also build credibility and make Bradbury’s job easier as he proposes new initiatives from Smiths’ IT organization. Winning acceptance for projects often requires overcoming people’s natural resistance to change. “As you have successes, as you show value, they start to believe your presentations back to the business, if you are consistent in delivery as projects move forward,” he said.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

St. Luke’s Radiology Group Accelerates Reporting with Speech Recognition Technology

As the IT group, you can do a good job with the technology, but it is the cultural change, the enthusiastic adoption by doctors and staff, that really makes it a success.
– Adrienne Edens, Chief Information Officer, St. Luke’s Health System

Toward Timely Transcription

As part of a new consolidated PACS installation for archiving digital images, St. Luke’s Health System wanted to implement a speech recognition technology for its radiology group to accelerate reporting and facilitate more responsive patient care. At the time, the turnaround time for manually transcribing a radiologist’s dictation averaged seven hours. Patients had to wait that much longer for the results of an x-ray or other medical image. “The goal of speech recognition was to eliminate transcription and shorten this process,” said Adrienne Edens, Chief Information Officer for St. Luke’s.

St. Luke’s Health System is an Idaho-based, not-for-profit health care provider for southern and central Idaho, northern Nevada and eastern Oregon. It operates medical centers in Boise, Meridian, Magic Valley, McCall and Wood River as well as more than 70 clinics. St. Luke’s Boise Medical Center is Idaho’s largest hospital, and St. Luke’s Children’s Hospital is the only children’s hospital in the state.

St. Luke’s medical imaging process involved several steps. When a physician ordered a medical image, a technician performed the scan and sent the image to a radiologist for diagnosis. The radiologist dictated an audio report that was forwarded to the transcriptionists. After the text version came back, the radiologist approved it and the official report and diagnosis went to the referring physician and became part of the patient’s medical record.

Manual transcription was the most time-consuming step. If automated speech recognition could mitigate or even eliminate the need for it, the overall time to deliver a diagnosis could be reduced dramatically.

Speech Recognition Development

Speech recognition technology has been in development for decades. Programming a computer to recognize and decipher human speech is particularly complex and challenging. While the accuracy rate has improved over time, progress has been slow and incremental. Today, the technology has advanced to a point where it is a viable for certain applications. “Even in the last couple of years the technology has come a long way,” said Dan Talley, IT Service Team Leader for St. Luke’s.

Collaborative Approach

St. Luke’s took a collaborative approach for reviewing options and approving a speech recognition system. All stakeholders had a seat at the table: the IT group who would manage it, the hospital administrators who oversaw budgeting and compliance, and the radiologists who would ultimately use the system as part of their jobs. They also enlisted Santa Rosa Consulting, an IT and management consulting firm for the healthcare industry, to provide guidance and expertise in this area.

From the beginning, the radiologists were active proponents of a speech recognition system. “It was not administration saying we needed the system, or even IT. The radiologists were the most enthusiastic advocates,” said Edens.

RadWhere from Nuance Healthcare

After considering several speech recognition solutions, the project team chose RadWhere from Nuance Healthcare. RadWhere is a speech recognition solution with reporting and workflow management designed specifically for radiology. The software transcribes a radiologist’s dictation in real-time, thereby speeding up report writing. It includes templates and macros based on procedure codes for structuring the reports. RadWhere’s workflow engine also automatically routes documents through the creation, review and delivery process.

To roll out the system, the IT group installed software on the radiologists’ workstations and the primary RadWhere application on a host server that is replicated to a remote site for disaster recovery. If a failure occurs at the local site, the system will fail over and continue to provide service to the radiology group. Furthermore, each user had to “train” the system to understand his or her voice by speaking scripts into the computer.

The radiologists were quick to incorporate the new system into their routine. “Normally these types of projects are phased in slowly, but in this case, radiology no longer needed transcriptionists after only two weeks,” said Talley.

It felt great to experience such an improvement, especially after being a part of this project.
– Dan Talley, IT Service Team Leader, St. Luke’s Health System

More Responsive Patient Care For Less

As a result of the fast adoption, the benefits to St. Luke’s and its patients accrued quickly. The average turnaround time for radiology reports dropped from 7 hours to about 30 minutes. St. Luke’s is saving $450,000 per year in radiology transcription costs for the Treasure Valley alone (Boise, Meridian and surrounding areas). Most significantly, patients receive faster and more responsive care.

Talley gave a personal example. His elderly mother had fallen down occasionally in the past, and Talley would take her to the hospital for an x-ray. Typically they had to wait hours for the results, but the last time this happened, after the speech recognition system was in place, the x-ray and diagnosis came back in only 20 minutes. “It felt great to experience such an improvement, especially after being a part of this project,” he said.

Edens added, “We are thrilled and wish all of our IT projects went this well. We give the radiologists credit for the success. As the IT group, you can do a good job with the technology, but it is the cultural change, the enthusiastic adoption by doctors and staff, that really makes it a success.”

Based on the success of the RadWhere system, St. Luke’s is looking at standardizing on Nuance technology for all of its medical speech recognition applications.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

Arlie and Company Revamps IT with Server Virtualization and Unified Storage

It took more than all night to do tape backups of the data. They would roll over into the daytime and really affect mail flow and access time to files on the file server.
– Adam Falk, Director of Technical Operations, Arlie & Company

Toward More Robust and Efficient IT

After a period of fast growth, Arlie & Company founds its server and storage infrastructure no longer provided the robustness and efficiency that the business needed. Nightly tape backups did not complete within the backup window. “It took more than all night to do tape backups of the data. They would roll over into the daytime and really affect mail flow and access time to files on the file server,” said Adam Falk, Director of Technical Operations for Arlie & Company. The servers were not configured for redundancy and failover, so the company was at risk of losing access to applications like email and printing potentially for days if the server hardware failed and had to be replaced. Furthermore, CPU and memory utilization on the servers was very low, and Falk felt the assets were not being well-utilized.

Arlie & Company is a privately-owned real estate development and management company based in Eugene, Oregon. By mid-2008, the company had doubled in size over a period of six years, reaching 28 employees, and was planning to double again. It was at this time they decided to upgrade the IT infrastructure.

The existing infrastructure consisted of two Windows servers running with about a half terabyte each of internal hard drive storage. One was a Dell PowerEdge 2800, a high-availability tower server with dual Intel Xeon processors, running Microsoft Small Business Server which included Exchange, SharePoint, the domain controller and a public-facing FTP site. The company had purchased it three years prior with the idea of growing into it. The other server was running Windows file and print services.

VMware and NetApp as Foundation

To address the problems of performance, resiliency and efficiency, Falk proposed two key technologies: server virtualization and consolidated, shared storage. After reviewing multiple products, they decided to purchase and install VMware vSphere 4 and a NetApp FAS2020 storage platform.

VMware is the market-leading server virtualization platform. It partitions a physical server into multiple virtual machines, each of which acts as a discrete server environment. Virtual machines are far easier to create, delete and move than physical machines and provide greater flexibility and higher server utilization.

The NetApp FAS2020 is a unified storage platform that includes both SAN and NAS, block and file storage, in one system. The FAS2020 supports iSCSI and Fibre Channel block protocols and CIFS, NFS and FTP file protocols. It expands to 12 disk drives internally and 68 drives overall through external expansion units. It runs NetApp’s Data ONTAP operating system that offers space-efficient differential snapshots and numerous other data management and protection features.

Arlie & Company based its upgraded IT infrastructure around the FAS2020 as a centralized, shared storage platform that connects to servers and clients over a Gigabit Ethernet network. Six of the twelve internal drive slots in the FAS2020 contain 1 TB SATA drives, and the others are available for future expansion. The drives are configured for redundancy using a combination of double-parity RAID and internal mirroring. Some storage capacity is allocated as a Windows file share accessed via CIFS for storing documents, photos, etc. The remainder is dedicated to iSCSI LUNs for VMware virtual machines. NetApp is integrated with Windows and VMware at the functional and management layers, enabling these technologies to leverage each other’s functionality and minimizing the learning curve for IT administrators. Of the storage platforms that Falk evaluated, the FAS2020 was the most tightly integrated with VMware and Windows.

The two existing host servers were repurposed to run Windows Server 2008 and VMware. Each has dual Gigabit Ethernet ports for high-bandwidth connections to clients and storage. The servers boot off USB drives and then access their primary storage over the network on the FAS2020. (Network boot for VMware is also an option available today.) Exchange, SharePoint, the FTP server, web presence, and primary and secondary domain controllers now run in virtual machines distributed between the two servers. In addition, VMware has a large community that provides free virtualized applications, where Falk found ProjectPier, an open-source, web-based application for project collaboration and task management. He installed ProjectPier in a virtual machine and saved $50 per month the company used to pay for a commercial hosted project collaboration service.

In this virtual environment, upgrading or moving an application is non-disruptive to the other applications running on the same physical server. Testing and development is streamlined as well. Instead of deploying a physical server, Falk tests new software by installing it in a virtual machine. If he decides not to keep the software, it is a simple matter to roll the virtual machine back to the previous state.

I am very pleased and very comfortable with the level of systems security and uptime.”
– Adam Falk

Robust, Non-disruptive Data Protection

The company’s critical data is now protected with disk-based snapshots, local replication and nightly backups to tape that are stored offsite for disaster recovery – all without slowing or disrupting user applications. The FAS2020 is scheduled to automatically take snapshots of the Windows file share five times per day and keeps 30 days of snapshots online. If a file is accidentally deleted or dragged and dropped into an unknown folder, users can retrieve any previous version right from the Windows interface, thanks to the NetApp integration. Each night Symantec Backup Exec 12.5 backs up the last daily snapshot to tape.

A different backup application called Veeam Backup and Replication protects the virtual machine images. Veeam is the market-leading backup and recovery solution designed specifically for VMware virtual environments. It offers instant recovery of virtual machines and files and recovery of application objects like email. It also verifies the recoverability of every backup image and replicates them locally or to a remote site for disaster recovery.

In this case, Veeam Backup and Replication takes nightly snapshots (changed blocks only) of the virtual machines and stores them on the backup server’s local hard drive. (Backup Exec uses the snapshots as the source for tape backups.) Furthermore, every seven hours Veeam takes snapshots of the images for Exchange, the domain controller and company’s web presence and replicates them to the local drive on a separate blade server. “If something happens to the OS in the virtual machine or if the hardware fails, I can immediately turn on that replica and bring up the mail server, which is our most critical application,” said Falk.

“I am very pleased and very comfortable with the level of systems security and uptime. IT is a service department. I don’t bring in money. I’m an expense area. So my job is to make sure everyone who does bring in money has the tools they need to do their jobs. That means the highest availability and least disruption possible to services,” he added. The upgraded infrastructure has been run smoothly since it was installed two and a half years ago. In fact, Arlie & Company has experienced only a single email outage, which lasted ten hours, in the last seven years – a solid record for a small business with a one-man IT shop.

Room to Grow, Space to Breathe

The virtualized servers and shared, unified storage are utilized at closer to capacity, so the assets are more efficient and “green.” At the same time, there is plenty of headroom to expand when business starts to grow again. The infrastructure is also easier to manage. “The amount of time I spend on IT has decreased to the point that I have actually picked up other duties in the company, because I don’t have to babysit so much,” said Falk.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.
Follow

Get every new post delivered to your Inbox.