Cenegenics Builds Resilient Cisco Network to Support Its Growing Practice

It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about.”
– Dror Nisenbaum, IT Manager, Cenegenics

Pressures of Growth on IT

Dror Nisenbaum, IT Manager at Cenegenics, clearly remembers his first day of work. He arrived at eight o’clock in the morning, sat down and the phone rang before he could take the first sip of coffee. It was a disgruntled user who had been waiting a month to fix an IT problem. Nisenbaum walked him through a solution right there on the phone. The user was pleasantly surprised and relieved, while for Nisenbaum it was a not-so-subtle hint of the road ahead. In fact, after spending four intense months working through the IT support backlog, he turned his attention to the challenge of redesigning and rebuilding the network and other IT infrastructure to support the company’s fast growth.

Cenegenics, based in Las Vegas, Nevada, is the largest age management practice with 20 medical centers in the United States serving more than 20,000 patients. Age management emphasizes prevention, as opposed to treating disease, and employs nutrition, exercise, supplements and hormone treatment under the care of a physician to maximize the quality of life for its patients as they age. Cenegenics is well known for ads showing one of its physicians, Dr. Jeff Life, a 72-year old man with the toned body of a youthful muscle builder.

Cenegenics experienced many years of rapid growth. From 2006 to 2012, it expanded from 3 medical centers in Las Vegas, Boca Raton and the Carolinas to 20 medical centers throughout the United States. The number of corporate staff at its headquarters in Las Vegas quadrupled. By any measure, the practice was successful and growing.

However, the expansion and growth put a strain on its aging IT infrastructure. Ten years ago, Cenegenics had a flat network with HP equipment supporting its users and applications. They added a voice over IP (VOIP) system. The VOIP provider installed Cisco networking equipment with power over Ethernet (POE) support, so they had two networks connected via uplinks, one for data and the other for voice. Unfortunately, the features and configurations were inconsistent and many problems arose. Phones did not power up if connected to the HP switches, and computers would not acquire the right IP addresses if connected to the Cisco switches. There were many dropped calls because the HP switches did not support QoS. Data storms flared up on a couple of occasions because of misconfigurations between the networks. The IT contractor who managed Cenegenic’s network was an HP reseller trained to support HP networking equipment, but not Cisco. They had to manufacture artificial phone problems and call the VOIP provider to get Cisco support. “It was very difficult. It was the perfect storm,” said Nisenbaum.

After the support backlog was finally under control and response times were reasonable, Nisenbaum focused on diagramming and labeling the entire network, including switches, cables, distribution points and servers. “I had to redo all of the wiring closets because it looked like a spaghetti ball,” he said. The jumble of cables and switches was so confusing they resorted to pulling cables after business hours and walking to the adjacent suite to see which device turned off. The result of the effort was a diagram that provided a comprehensive view of the infrastructure and identified the single points of failure.

Building a Consolidated, Highly Resilient Network

Next they developed a plan for a consolidated, highly-available, virtualized infrastructure. It included two Cisco Catalyst 3560-X switches as the core of the network and a Cisco 3945 Integrated Services Router to replace a SonicWALL router and one other that were too small for the expanded user load. The two 3560-X switches had redundant power supplies and were paired up for failover. Six existing Cisco Catalyst 3560s would serve as edge switches. Connections around the headquarters building would have four strands trunked in pairs to make a redundant fiber ring. Pipes between suites in the building would carry not more than one fiber optic cable to eliminate the risk of a severed pipe disrupting communication. The design also included three powerful Dell servers with dual six-core processors running VMware and a Dell EqualLogic iSCSI SAN with dual controllers connected via the 3560-X core switches.

Nisenbaum felt strongly that the network should be consolidated on Cisco equipment, not a mixed-vendor environment. “It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about,” he said. Nisenbaum is Cisco-certified and very comfortable using their equipment. Though he had some experience with HP and SonicWALL equipment, he thought some of their configuration procedures were unnecessarily complicated. “Having multiple vendors brings so much complexity… I wanted to consolidate everything to one vendor, so I can train my employees and make sure the team we have in place can manage it all.”

The plan was presented to Cenegenics’ executive board. The project cost was in the six figures, a substantial investment for a company of that size. Nisenbaum’s approach was to show them all the points of failure in their existing infrastructure. “If any of these devices goes down, this is the catastrophic effect. How much is it going to cost you per day if this doesn’t work? That was my pitch. I came in six months after one of their servers crashed and they were down for almost a week. What I was saying was very fresh in their minds,” he said. The board decided to approve and fund the project.

I can sum it up with one statement:  I can sleep at night.”
– Dror Nisenbaum

Uptime Makes Everyone Happy

Installation of the new equipment took four months because some aspects of the transition had to be handled carefully to avoid disrupting users. Today the company’s applications and users run on the rebuilt and highly resilient network as well as server and storage infrastructure. It has been 16 months since a server went down. While individual components have failed and had to be replaced, such as a GBIC on a switch, the redundancy of the design ensured that users did not experience downtime. Executives and staff are pleased because they can focus on running the practice and serving patients with a stable IT infrastructure supporting them.

Asked about how he feels about the situation now, Nisenbaum replied, “I can sum it up with one statement: I can sleep at night. When I had all the aging hardware and there was such a support backlog, I didn’t have a personal life. Now that things have slowed down, it gives me the ability to see how I can improve Cenegenics as a whole.”

AIT_Profiles_Blogocon_small

 Copyright © 2012 Apropos LLC. All rights reserved.

Virtual Machines, Virtual Desktops and Zero Clients Deliver Real Benefits for PAE Consulting Engineers

We had all this hardware running and were going to replace it every three to five years, but it wasn’t really getting used. We were never going to wear it out.
– Dustin Rowe, IT Manager and Technology Planner,
PAE Consulting Engineers

The Drive to Virtualize

For PAE Consulting Engineers, the drive to virtualize began with realizing that its distributed server architecture was only 10% utilized. The company was running 20 application workloads on eight physical servers. Since mission-critical applications were distributed among the servers, this architecture offered a reasonable degree of high availability. Restarting one application, such as Microsoft Exchange, did not require bringing down other critical applications, such as Microsoft SQL Server or Active Directory. The downside was that these servers were poorly utilized. As servers are normally replaced every few years, the company was locked into a cycle of overbuying hardware and recycling still-usable equipment. “Virtualization takes advantage of that downtime. That is what drove us forward,” said Dustin Rowe, IT Manager and Technology Planner for PAE Consulting Engineers. “We had all this hardware running and were going to replace it every three to five years, but it wasn’t really getting used. We were never going to wear it out.”

At the same time, the company had about 35 traditional desktop PCs with Intel Pentium 4 processors that had reached end-of-life and needed to be replaced. These PCs ran applications with moderate processing requirements, such as email, web browsing and project management. Here they saw an opportunity to also consider virtual desktop infrastructure (VDI) instead of a typical PC upgrade. “As we moved forward, we realized that by virtualizing our application servers, we would free up hardware that could be used to explore virtual desktop systems, which we thought might be able to meet the needs of these people with traditional desktops,” said Rowe.

PAE Consulting Engineers, Inc., is a mechanical and electrical engineering firm that specializes in sustainable, efficient, “green” building design. The company is based in Portland, Oregon, and has a satellite office in San Francisco, California.

Virtual Machines, Virtual Desktops and Zero Clients

Working with a local computer consulting company called Tech Heads, PAE Consulting Engineers looked closely at various options for server virtualization and VDI, including Microsoft Hyper-V, Citrix and VMware. At the time Hyper-V lacked some features they wanted. Citrix would have required eight servers to support the application workloads and desktop virtualization, while VMware could do it with half the number. They also found VMware to be very easy to use. Performing physical-to-virtual migrations and provisioning virtual machines were straightforward, simple tasks. For a small business with one IT manager handling the entire infrastructure, this was an important factor. Furthermore, pricing for virtualization software had dropped a point where it was realistic for the company’s IT budget.

The company chose VMware vSphere for server virtualization, VMware View for desktop virtualization and ClearCube I9422 Zero Clients at the desktop. VMware vSphere is the leading platform for server virtualization. VMware View is a desktop virtualization platform that runs Windows sessions centrally on a server and streams the display data over the network to client devices. In this case, the client devices were ClearCube Zero Clients that reside at each user’s desk and perform the role of a PC except for storage and processing. These devices communicate with the View server over the network using a streamlined PCoIP display protocol that facilitates a fluid screen presentation. They provide a single restart button and support dual monitors, keyboard, mouse and peripherals.

Easier to Manage and Less Costly to Operate

With the new virtual environment, PAE Consulting Engineers consolidated its original eight application servers down to two VMware servers running the same 20 application workloads in virtual machines. This was a major improvement in hardware utilization and delivered cost savings that would carry forward into the future as they only needed to replace two servers instead of eight when they reached end-of-life. Three additional servers ran VMware View to support nearly 30 virtual desktops. Two of the servers were required to support the workload and the third was for redundancy and failover purposes. Rather than purchase new hardware, they added processors, memory and network cards to the existing servers and redeployed them, thus minimizing costs for the transition to the virtual infrastructure.

They also added two IBM DS3500 Express storage systems, dedicating one for the application servers and the other for the virtual desktop system. Beyond storage consolidation and high availability, SAN storage allowed them to derive more benefits from server virtualization. For instance, the VMware vMotion feature in a SAN environment can migrate applications live between servers without disrupting user access. “We moved Microsoft Exchange in the middle of the work day with seventy people running email through it, and no one could even tell it moved to a different server,” he said.

At the relatively small scale of the company’s virtual desktop environment, hardware costs as compared to traditional PCs were break-even. While Zero Clients cost less than PCs and have a longer expected life of 7 to 10 years since they lack moving parts, the VMware View servers and SAN storage also factored in to the total cost.

The greatest benefit PAE Consulting Engineers experienced from VDI was simplified management and reduced IT supports costs. Rowe continued, “Now IT has one place to go take care of hardware. Aside from training and educating users on the end point, we nearly never have to go to the desk. That foot traffic is an incredible amount of time for the IT staff. You’re up and constantly going to a different machine to do something physically that is unnecessary with the technology we are using these days. We can do everything from our remote session. And as you scale the business, it becomes more and more valuable – by an order of magnitude.”

Windows functionality for virtual desktop users was identical to traditional PCs, though it took time for users to adjust to some differences in the interface. For instance, when restarting a computer, users only saw a welcome screen instead of one proclaiming that Windows is restarting, so they wondered if the computer was working correctly. While issues like this did not affect productivity, they required some changes in how users interacted with the system. Most workers used ClearCube Zero Clients as client devices. A handful of people used an Apple laptop or mobile device as their main computer and ran a View software client for occasional Windows access.

A surprising benefit of virtualization was the ability to extend the useful life of server hardware. Normally servers are replaced when the warranty expires after 3 to 5 years because warranty costs becomes prohibitive as the hardware ages. But with redundant servers and components and the ease of moving workloads around in a virtual environment, it becomes possible to continue using servers outside of warranty until they fail, then replace them. “We can be more flexible and let the hardware push us to replace, not the warranty,” said Rowe. However, a thoughtful recovery plan should be in place to do this. The company knows which of its applications are mission-critical and which can afford a little downtime in case of a hardware failure and recovery scenario.

Virtualization is a big deal. It is literally everything that it proposed itself to be.”
– Dustin Rowe

Another benefit they experienced was energy efficiency and lower electricity costs. The ClearCube Zero Clients consume only 15 W of power compared to 100 W for the PCs they replaced, which reduced power and heat dissipation at the desktop by 85% and eliminated noise from fans and hard drives. The tasks of computer processing and storage were transferred to three 300 W servers and a SAN storage array. Even including this hardware in the calculation, the power consumption per desktop was still reduced by approximately 50%.

Potentially More to Come

to the future, Rowe noted that the industry is developing server offload cards for graphics processing that could open the door to run CAD and sophisticated graphic design applications in the virtual desktop environment. Currently the workers using these applications have workstations with 3D graphics cards that perform the heavy-duty processing because it would be too slow and choppy to run these applications on a central server. But if high-performance graphics processing were available for the View servers, the company could consolidate the remainder of its desktops and further streamline the IT infrastructure.

“Personally it is very exciting to see legitimate advancements in technology,” he added. “For a long time, there hasn’t been anything to really change the way we think about how hardware should operate and when we’re going to replace it. But virtualization is a big deal. It is literally everything that it proposed itself to be. It just takes using it a while to really see.”

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

Arlie and Company Revamps IT with Server Virtualization and Unified Storage

It took more than all night to do tape backups of the data. They would roll over into the daytime and really affect mail flow and access time to files on the file server.
– Adam Falk, Director of Technical Operations, Arlie & Company

Toward More Robust and Efficient IT

After a period of fast growth, Arlie & Company founds its server and storage infrastructure no longer provided the robustness and efficiency that the business needed. Nightly tape backups did not complete within the backup window. “It took more than all night to do tape backups of the data. They would roll over into the daytime and really affect mail flow and access time to files on the file server,” said Adam Falk, Director of Technical Operations for Arlie & Company. The servers were not configured for redundancy and failover, so the company was at risk of losing access to applications like email and printing potentially for days if the server hardware failed and had to be replaced. Furthermore, CPU and memory utilization on the servers was very low, and Falk felt the assets were not being well-utilized.

Arlie & Company is a privately-owned real estate development and management company based in Eugene, Oregon. By mid-2008, the company had doubled in size over a period of six years, reaching 28 employees, and was planning to double again. It was at this time they decided to upgrade the IT infrastructure.

The existing infrastructure consisted of two Windows servers running with about a half terabyte each of internal hard drive storage. One was a Dell PowerEdge 2800, a high-availability tower server with dual Intel Xeon processors, running Microsoft Small Business Server which included Exchange, SharePoint, the domain controller and a public-facing FTP site. The company had purchased it three years prior with the idea of growing into it. The other server was running Windows file and print services.

VMware and NetApp as Foundation

To address the problems of performance, resiliency and efficiency, Falk proposed two key technologies: server virtualization and consolidated, shared storage. After reviewing multiple products, they decided to purchase and install VMware vSphere 4 and a NetApp FAS2020 storage platform.

VMware is the market-leading server virtualization platform. It partitions a physical server into multiple virtual machines, each of which acts as a discrete server environment. Virtual machines are far easier to create, delete and move than physical machines and provide greater flexibility and higher server utilization.

The NetApp FAS2020 is a unified storage platform that includes both SAN and NAS, block and file storage, in one system. The FAS2020 supports iSCSI and Fibre Channel block protocols and CIFS, NFS and FTP file protocols. It expands to 12 disk drives internally and 68 drives overall through external expansion units. It runs NetApp’s Data ONTAP operating system that offers space-efficient differential snapshots and numerous other data management and protection features.

Arlie & Company based its upgraded IT infrastructure around the FAS2020 as a centralized, shared storage platform that connects to servers and clients over a Gigabit Ethernet network. Six of the twelve internal drive slots in the FAS2020 contain 1 TB SATA drives, and the others are available for future expansion. The drives are configured for redundancy using a combination of double-parity RAID and internal mirroring. Some storage capacity is allocated as a Windows file share accessed via CIFS for storing documents, photos, etc. The remainder is dedicated to iSCSI LUNs for VMware virtual machines. NetApp is integrated with Windows and VMware at the functional and management layers, enabling these technologies to leverage each other’s functionality and minimizing the learning curve for IT administrators. Of the storage platforms that Falk evaluated, the FAS2020 was the most tightly integrated with VMware and Windows.

The two existing host servers were repurposed to run Windows Server 2008 and VMware. Each has dual Gigabit Ethernet ports for high-bandwidth connections to clients and storage. The servers boot off USB drives and then access their primary storage over the network on the FAS2020. (Network boot for VMware is also an option available today.) Exchange, SharePoint, the FTP server, web presence, and primary and secondary domain controllers now run in virtual machines distributed between the two servers. In addition, VMware has a large community that provides free virtualized applications, where Falk found ProjectPier, an open-source, web-based application for project collaboration and task management. He installed ProjectPier in a virtual machine and saved $50 per month the company used to pay for a commercial hosted project collaboration service.

In this virtual environment, upgrading or moving an application is non-disruptive to the other applications running on the same physical server. Testing and development is streamlined as well. Instead of deploying a physical server, Falk tests new software by installing it in a virtual machine. If he decides not to keep the software, it is a simple matter to roll the virtual machine back to the previous state.

I am very pleased and very comfortable with the level of systems security and uptime.”
– Adam Falk

Robust, Non-disruptive Data Protection

The company’s critical data is now protected with disk-based snapshots, local replication and nightly backups to tape that are stored offsite for disaster recovery – all without slowing or disrupting user applications. The FAS2020 is scheduled to automatically take snapshots of the Windows file share five times per day and keeps 30 days of snapshots online. If a file is accidentally deleted or dragged and dropped into an unknown folder, users can retrieve any previous version right from the Windows interface, thanks to the NetApp integration. Each night Symantec Backup Exec 12.5 backs up the last daily snapshot to tape.

A different backup application called Veeam Backup and Replication protects the virtual machine images. Veeam is the market-leading backup and recovery solution designed specifically for VMware virtual environments. It offers instant recovery of virtual machines and files and recovery of application objects like email. It also verifies the recoverability of every backup image and replicates them locally or to a remote site for disaster recovery.

In this case, Veeam Backup and Replication takes nightly snapshots (changed blocks only) of the virtual machines and stores them on the backup server’s local hard drive. (Backup Exec uses the snapshots as the source for tape backups.) Furthermore, every seven hours Veeam takes snapshots of the images for Exchange, the domain controller and company’s web presence and replicates them to the local drive on a separate blade server. “If something happens to the OS in the virtual machine or if the hardware fails, I can immediately turn on that replica and bring up the mail server, which is our most critical application,” said Falk.

“I am very pleased and very comfortable with the level of systems security and uptime. IT is a service department. I don’t bring in money. I’m an expense area. So my job is to make sure everyone who does bring in money has the tools they need to do their jobs. That means the highest availability and least disruption possible to services,” he added. The upgraded infrastructure has been run smoothly since it was installed two and a half years ago. In fact, Arlie & Company has experienced only a single email outage, which lasted ten hours, in the last seven years – a solid record for a small business with a one-man IT shop.

Room to Grow, Space to Breathe

The virtualized servers and shared, unified storage are utilized at closer to capacity, so the assets are more efficient and “green.” At the same time, there is plenty of headroom to expand when business starts to grow again. The infrastructure is also easier to manage. “The amount of time I spend on IT has decreased to the point that I have actually picked up other duties in the company, because I don’t have to babysit so much,” said Falk.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

WASSER Studios Reduces Long-Term Costs With a Consolidated, Virtualized Infrastructure

We went with leading edge hardware so that we could grow into it over time and then focus on minimizing our management. Over time, that pays for itself.”
– Trevor Koop, IT Manager, WASSER Studios

A Change in IT Strategy

WASSER Studios decided to change its IT strategy in an effort to better support the business. It transitioned from a near-term focus on acquisition cost to a broader, long-term emphasis on ease of management, high asset utilization and low total cost of ownership. As a result, it was able to deliver a more reliable and functional IT infrastructure that cost the company less over time.

Headquartered in Seattle, Washington, WASSER Studios creates technical documentation and content for clients such as HP, Sony, Microsoft and Weyerhaeuser. It employs a variable number of people based on project demand – some are contractors hired on temporary basis. At the main office, the number of workers at can range from about 50 to 125 people, plus project managers and writers working remotely and at client sites. Its computer systems must be able to support a fluctuating number of local and remote users.

“WASSER Studios previously had taken an approach where we would buy previous-generation technology and then support that over time with parts from various places,” said Trevor Koop, IT Manager of WASSER Studios. “It was a very low cost method for us to keep everything up and running.” While they were able to support the business reasonably well with this approach, it required a significant amount of troubleshooting and reactive problem solving to maintain the infrastructure. Savings on hardware were eventually consumed by the cost of managing it. Occasional downtime of secondary applications would also slow down office productivity. “It’s never that we had work stoppages, but we had definite bottlenecks or pinch points,” he said.

To a Consolidated, Virtualized Infrastructure

Rather than continue with the status quo, WASSER opted to consolidate and virtualize the server infrastructure and employ more sophisticated management tools. “We decided to do that in an effort to consolidate and get off of the older, faultier server hardware that we had. In doing so, the hope was to minimize management and mitigate hardware failures, which we did,” said Koop.

The company deployed Microsoft Windows Essential Business Server (EBS) initially as a beta customer in late 2006.  This solution, which has since become generally available, bundles and integrates several Microsoft technologies for IT management, messaging and security at a discounted price. These include Microsoft Windows Server 2008, System Center Essentials, Exchange Server and Forefront Threat Management Gateway. Windows EBS targets midsize enterprises.

The software ran on an HP BladeSystem c3000 with five HP ProLiant server blades running AMD multi-core Opteron processors, two disk storage blades and an LTO tape blade. The BladeSystem c3000 is a flexible, all-in-one infrastructure in a single enclosure. Three of the server blades ran the Windows EBS components for management, messaging and security, respectively. The fourth blade supported multiple applications in a Microsoft Hyper-V virtual environment, and the fifth was a Terminal Services Gateway Server for remote users to securely access to internal corporate resources.

“We have typically used AMD chips,” said Koop. “We’re very comfortable with the performance and the reliability of them, so we tend to stick with them. And it’s worked out really well for us.”

Easier Management and Lower TCO

“So with this new deployment,” he continued, “we went from having all these disparate hardware platforms to having most everything in the blade server. We went with leading edge hardware so that we could grow into it over time and then focus on minimizing our management. Over time, that pays for itself.”

In particular, System Center Essentials helped ease the management burden for clients and servers with its monitoring, alerts and automation. “It really allowed us to be more proactive in maintenance, patching clients, monitoring applications. We know something is going to happen before an end user reports something going haywire on their desktop,” he said.

Hyper-V boosted server utilization by running multiple applications in a virtual environment. “This enables us to maximize the capacity of the hardware and also minimize the cost of the hardware that we have to purchase, whether replacement parts or whole servers,” he said. “From an energy efficiency standpoint, I know that Hyper-V runs a lot cooler than if we had all these older physical servers still in the room. Although I don’t have metrics, I can make a safe assumption that we’re being a lot more energy efficient.”

When asked what he liked most about this new solution, Koop responded: “I like the fact that in my current situation I can really be proactive in management. I can see something may be a problem and take care of it before it becomes a larger problem. But it also gives time to focus on the future. Because I’m not running around putting out fires, I have more time to think about what I can do with our infrastructure in the future to be more productive and to allow WASSER to provide more services to potential clients. So I think in my small ways, I’m able to do that, helping contribute to the growth of the company.”

AIT_Profiles_Blogocon_small 

 Copyright © 2009 Apropos LLC. All rights reserved.