Cenegenics Builds Resilient Cisco Network to Support Its Growing Practice

It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about.”
– Dror Nisenbaum, IT Manager, Cenegenics

Pressures of Growth on IT

Dror Nisenbaum, IT Manager at Cenegenics, clearly remembers his first day of work. He arrived at eight o’clock in the morning, sat down and the phone rang before he could take the first sip of coffee. It was a disgruntled user who had been waiting a month to fix an IT problem. Nisenbaum walked him through a solution right there on the phone. The user was pleasantly surprised and relieved, while for Nisenbaum it was a not-so-subtle hint of the road ahead. In fact, after spending four intense months working through the IT support backlog, he turned his attention to the challenge of redesigning and rebuilding the network and other IT infrastructure to support the company’s fast growth.

Cenegenics, based in Las Vegas, Nevada, is the largest age management practice with 20 medical centers in the United States serving more than 20,000 patients. Age management emphasizes prevention, as opposed to treating disease, and employs nutrition, exercise, supplements and hormone treatment under the care of a physician to maximize the quality of life for its patients as they age. Cenegenics is well known for ads showing one of its physicians, Dr. Jeff Life, a 72-year old man with the toned body of a youthful muscle builder.

Cenegenics experienced many years of rapid growth. From 2006 to 2012, it expanded from 3 medical centers in Las Vegas, Boca Raton and the Carolinas to 20 medical centers throughout the United States. The number of corporate staff at its headquarters in Las Vegas quadrupled. By any measure, the practice was successful and growing.

However, the expansion and growth put a strain on its aging IT infrastructure. Ten years ago, Cenegenics had a flat network with HP equipment supporting its users and applications. They added a voice over IP (VOIP) system. The VOIP provider installed Cisco networking equipment with power over Ethernet (POE) support, so they had two networks connected via uplinks, one for data and the other for voice. Unfortunately, the features and configurations were inconsistent and many problems arose. Phones did not power up if connected to the HP switches, and computers would not acquire the right IP addresses if connected to the Cisco switches. There were many dropped calls because the HP switches did not support QoS. Data storms flared up on a couple of occasions because of misconfigurations between the networks. The IT contractor who managed Cenegenic’s network was an HP reseller trained to support HP networking equipment, but not Cisco. They had to manufacture artificial phone problems and call the VOIP provider to get Cisco support. “It was very difficult. It was the perfect storm,” said Nisenbaum.

After the support backlog was finally under control and response times were reasonable, Nisenbaum focused on diagramming and labeling the entire network, including switches, cables, distribution points and servers. “I had to redo all of the wiring closets because it looked like a spaghetti ball,” he said. The jumble of cables and switches was so confusing they resorted to pulling cables after business hours and walking to the adjacent suite to see which device turned off. The result of the effort was a diagram that provided a comprehensive view of the infrastructure and identified the single points of failure.

Building a Consolidated, Highly Resilient Network

Next they developed a plan for a consolidated, highly-available, virtualized infrastructure. It included two Cisco Catalyst 3560-X switches as the core of the network and a Cisco 3945 Integrated Services Router to replace a SonicWALL router and one other that were too small for the expanded user load. The two 3560-X switches had redundant power supplies and were paired up for failover. Six existing Cisco Catalyst 3560s would serve as edge switches. Connections around the headquarters building would have four strands trunked in pairs to make a redundant fiber ring. Pipes between suites in the building would carry not more than one fiber optic cable to eliminate the risk of a severed pipe disrupting communication. The design also included three powerful Dell servers with dual six-core processors running VMware and a Dell EqualLogic iSCSI SAN with dual controllers connected via the 3560-X core switches.

Nisenbaum felt strongly that the network should be consolidated on Cisco equipment, not a mixed-vendor environment. “It is easier to have just one specific vendor, especially for the switching gear, because there is a lot you have to worry about,” he said. Nisenbaum is Cisco-certified and very comfortable using their equipment. Though he had some experience with HP and SonicWALL equipment, he thought some of their configuration procedures were unnecessarily complicated. “Having multiple vendors brings so much complexity… I wanted to consolidate everything to one vendor, so I can train my employees and make sure the team we have in place can manage it all.”

The plan was presented to Cenegenics’ executive board. The project cost was in the six figures, a substantial investment for a company of that size. Nisenbaum’s approach was to show them all the points of failure in their existing infrastructure. “If any of these devices goes down, this is the catastrophic effect. How much is it going to cost you per day if this doesn’t work? That was my pitch. I came in six months after one of their servers crashed and they were down for almost a week. What I was saying was very fresh in their minds,” he said. The board decided to approve and fund the project.

I can sum it up with one statement:  I can sleep at night.”
– Dror Nisenbaum

Uptime Makes Everyone Happy

Installation of the new equipment took four months because some aspects of the transition had to be handled carefully to avoid disrupting users. Today the company’s applications and users run on the rebuilt and highly resilient network as well as server and storage infrastructure. It has been 16 months since a server went down. While individual components have failed and had to be replaced, such as a GBIC on a switch, the redundancy of the design ensured that users did not experience downtime. Executives and staff are pleased because they can focus on running the practice and serving patients with a stable IT infrastructure supporting them.

Asked about how he feels about the situation now, Nisenbaum replied, “I can sum it up with one statement: I can sleep at night. When I had all the aging hardware and there was such a support backlog, I didn’t have a personal life. Now that things have slowed down, it gives me the ability to see how I can improve Cenegenics as a whole.”

AIT_Profiles_Blogocon_small

 Copyright © 2012 Apropos LLC. All rights reserved.

Grigg Brothers Turns to VOIP Service from ATC to Reduce Phone Costs and Improve Communication

I wanted to simplify the sheer number of points of contact and make it easier to communicate.
– Jared Grigg, Director of Communication and Information, Grigg Brothers

Toward a Streamlined Phone System

Grigg Brothers wanted to streamline and consolidate its corporate phone system. Its existing system was an assortment of phone lines and services for its headquarters and operational office in southern Idaho and seven branch offices around the U.S. Each office in Idaho supported several employees. Most of the branch offices where field technical reps were based had two phone lines – one for voice and another for fax – plus an Internet connection and mobile phone. “Communications and technology can be a big line item on the budget,” said Jared Grigg, Director of Communication and Information. “The reps were turning in expense reports with fax numbers and landlines and mobile lines, and I said, this is too much. This can be considerably more condensed. I wanted to simplify the sheer number of points of contact and make it easier to communicate.”

Grigg Brothers is a manufacturer and wholesaler of specialty fertilizers for the turf grass industry, including golf courses and sports fields. The company’s flagship products are liquid foliar fertilizers that are absorbed through the leaf tissue of a plant instead of through the roots. Nutrients applied in this way are almost fully absorbed, even during stressful times like summer heat, and create healthier turf. Grigg Brothers’ foliar fertilizers are also more environmentally friendly because they avoid toxic soil buildup, leaching and runoff.

In 2005, when Jared Grigg first started considering options for a new phone system, the owners of the company expressed skepticism about voice over IP (VOIP). They wondered whether it was mature and robust enough to support a business. Because of this and his prior experience managing a Nortel Meridian phone system, Grigg decided to focus more intently on traditional key telephone systems. He dialogued extensively with a local reseller, received several proposals and even proposed a system to the board of directors. The system offered greater flexibility and better features, but ultimately the directors turned it down because the cost was higher than what they were already paying and exceeded the technology budget.

Meanwhile VOIP technology was improving and gaining acceptance in the market. “I decided to look at the cloud again because more and more solutions were becoming available,” he said.

His first step was to deploy an Internet fax service. This service automatically converted incoming faxes to PDF format and sent them by email to the recipients. Everyone in the company received a new fax number. Technical reps who spent much of their time traveling could receive faxes immediately on their PC instead of waiting until they came back to the office. Outgoing faxes could be scanned in to a PC and emailed to the fax service for delivery. The electronic fax service cost considerably less than the dedicated fax lines it replaced.

A Serendipitous Opportunity

In 2007, ATC Communications, the Internet and phone service provider for Grigg Brothers headquarters, acquired a VOIP solution provider called Nextphone. Here Grigg saw an opportunity. One of his concerns with a VOIP solution was that if a problem arose, the VOIP provider might blame the Internet service provider and vice-versa, leaving the user caught in the middle. “If I went down this road, if I backed ATC with their new acquisition, then they would be responsible regardless of whether it was a phone problem or an Internet connectivity problem,” he said.

The ATC Nextphone service had the features they were looking for:

  • Hosted IP phone service that all offices around the U.S. could access over an Internet connection
  • Programmable “find me, follow me” feature for incoming calls that can, for instance, simultaneously ring a desk phone and cell phone, or one then the other, before going to voice mail
  • Four-digit dial connecting everyone in their geographically distributed organization as if they were in the same building
  • Internet fax service with solid features and technical support
  • Unified messaging that sends voice mails and faxes to a user’s email inbox
  • Conferencing capabilities so Grigg Brothers can set up and control its own conference calls
  • Web portal for centralized programming and administration
  • Automatic call routing to cell phones in case of local power outage
  • Significantly lower cost than their existing phone system

This set of capabilities combined with Grigg Brothers’ established relationship with ATC made it attractive. “It was the perfect solution,” said Grigg. The company deployed Nextphone across all its locations. Branch offices received Linksys PAP2T Internet Phone Adapters. To access the new phone service, they only needed to connect the adapter to the Internet and plug in any standard telephone handset. If a technical rep preferred to rely exclusively on a cell phone, the system could be programmed to route all incoming calls there. The company also standardized on Google Apps Premier, so users can access email, voice mail and faxes anywhere using a Web browser.

Headquarters initially experienced intermittent call quality problems, much to Grigg’s chagrin, though his strategy of using the same VOIP and Internet service provider proved advantageous. Grigg was concerned that VOIP would get a black eye because the owners were already apprehensive. But he believed the hosted service was solid and worked with ATC to address the underlying network quality issue. While on the verge of deploying a second DSL at headquarters and a dedicated switch for the phone system, ATC offered to try boosting the bandwidth of the existing Internet connection. “It automatically improved. I have not had a single problem since the day they did that,” he said.

In my twenty plus years of doing this, I have never been more synchronized across all platforms.”
– Jared Grigg

Less Costly and Easier to Support

As a result of switching to VOIP, overall communication costs fell by more than 40% in the first 12 months and have remained level. At the same time, the company streamlined the number of phone lines and points of contact and improved its ability to communicate and collaborate.

Grigg is also pleased that he spends much less time supporting this new phone system. He uses the Nextphone web portal to handle routine administration and support requests. In fact, the company’s entire IT infrastructure is set up for centralized, remote administration. “In my office I have several computers and monitors in a semicircle around me, and I am remotely connected to every single asset in the company – all desktop computers, all laptops and even some cell phones as we move to smarter phones. In my twenty plus years of doing this, I have never been more synchronized across all platforms,” he said.

Soft Phones for International Roaming

More recently the company started using soft phones for placing calls when outside the U.S. to expensive international roaming charges. Soft phone software runs on PCs and smart phones such as iPhone and Android. It lets users place phone calls through that device over the Internet as if they were at their office desk. It is another way Grigg Brothers is using VOIP to keep down communication costs.

AIT_Profiles_Blogocon_small

 Copyright © 2011 Apropos LLC. All rights reserved.

From DEC to Allied Telesis – Butler County Evolves Its Data, Voice and Video Network

We wanted to make a decision on a product line that
we were going to stick with
.”
– Mike Felerski, Network Manager, Butler County

End of Life for DEC Networking Equipment

The networking equipment that Butler County relied on was about to be discontinued. At that time in 2003, Butler County, Ohio, used IT systems originally supplied by Digital Equipment Corporate (DEC). It ran VAX and Alpha systems and employed DEC networking equipment to connect its 28 government sites spread around the county. Unfortunately DEC is one of many once-great computer companies that no longer exist, having succumbed to the rapid pace of innovation and creative destruction that characterize the computer industry. At its peak it was the second-largest computer company in the world, but today it is only a memorable icon of the minicomputer era. As for Butler County, it needed to find a new networking equipment vendor to supply its aging network infrastructure.

Butler County is located just north of Cincinnati, sandwiched between the metro areas of Cincinnati and Dayton. In recent years it has been one of the fastest-growing counties in the state. With growth comes the need to build out infrastructure to serve its citizens.

At about the same time in 2003, a commissioner had the foresight to encourage the county to build a high-speed fiber optic ring that tied together its distributed government sites, and whose bandwidth might also be offered to businesses to encourage investment and development. This furthered the need to invest in new networking equipment.

“We wanted to make a decision on a product line that we were going to stick with. We didn’t want to hop around and try to find something else later. So we started looking at all the big names,” said Mike Felerski, Network Manager for Butler County. The requirement was for layer 2 switches and layer 3 switches for routing, including a scalable core switch to be located at its main Government Services Center.

Switch to Allied Telesis

After testing and evaluating products from multiple manufacturers, it chose to standardize on networking equipment from Allied Telesis. One feature that tipped the scales in its favor was that Allied Telesis did not charge extra to run the OSPF routing protocol on its switches. Butler County had already standardized on OSPF, and the other vendors in consideration wanted to charge for an additional license for it.

Felerski continued, “We also liked the way Allied was built. There are some jokes in the industry about how Allied must have an investment in steel companies because there is a lot of metal in their boxes. They’re built really well. Some older gear from Allied ended up in harsh environments where you wouldn’t want to put normal network equipment, and it survived.”

“Putting all these pieces together, we were happy with the price and the fact that the equipment did what it needed to do – that took us to Allied.”

The county deployed the new switches incrementally over time. “The IS department doesn’t have a huge capital improvements budget, so we rely on other departments. They propose projects for a move, expansion or remodel, and have money set aside for PCs and networking. So every time there was a change, we would go through and replace the Digital equipment. We eventually replaced the core switch at the Government Services Center, which was a MultiSwitch 900 from Digital, with the Allied SwitchBlade x908,” he said.

Voice over IP System

In the last year and a half, Butler County deployed a voice over IP (VOIP) system from Avaya. VOIP is more cost-effective than a traditional phone service because it uses a low-cost IP networks to send voice communication. In this case, all voice communication within the county government ran over its fiber-optic network. VOIP also enables more advanced features such as integration with email, video and contact management software.

To support the VOIP system, the county put in 8000 series PoE (Power over-Ethernet) switches from Allied Telesis to connect IP phones at each desk. These switches supply power to low-wattage devices so they do not require power adapters. Without the tether of a power cord, the IP phones can be placed anywhere that an Ethernet cable will reach.

“We had replaced a lot of the equipment over the years with the Allied gear and felt confident about the network. So it was time to replace the layer 2 switches and get the PoE switches to support the voice network,” said Felerski.

Radio System for Sheriff’s Department

Another major project was the sheriff department’s 800 MHz radio system. The county enlisted Motorola to build a radio system that would enable its disparate law enforcement agencies and fire departments to communicate with each other.

Motorola required T1 lines to connect each 911 and dispatch center. These are typically leased from telecommunications companies. Felerski added, “We are a county that has three different phone companies – Cincinnati Bell, AT&T and Verizon. Any time you have T1 lines crossing LATAs, it costs a good piece of change even with a government discount, if you can get one. When this project was brought to the commissioners, they looked at it and said, ‘Wow, this is impressive, but we have this fiber optic ring that passes either through or near every one of these dispatch centers. Why can’t we use that?’”

Felerski and his colleague Tom Line were tasked with finding a way to provide T1 services over the fiber optic network, and thereby saving a significant amount in data communications costs for the new radio system. Felerski was aware Allied supplied equipment for this purpose, so he discussed it with them and began researching equipment. “I said, ‘Here is the design I came up with. Let me know if this is going to work the way I understand this equipment.’ And they took a few days and went through it and said, ‘That is basically how we would design it.’ So we went ahead, ordered the equipment and put it in place. Allied was there at every step to make suggestions on how to tweak the equipment to get it to do what Motorola wanted,” he said.

Security Video over Network

Video for security cameras also runs over the network. A conversion project is underway to replace analog cameras in government facilities with IP cameras that connect directly to the network and stream video over it. They are also powered by PoE switches, so the cameras can be placed virtually anywhere. All video will eventually feed to the sheriff’s department, where law enforcements agents will be able to view real-time and historical footage at government sites all around the county.

In addition, the courts sometimes use the network for video arraignments.

Two Managers for One County Network

Felerski and Line use the AlliedView software management tool to centrally manage the network. “We have been very careful not to cut any corners because we don’t have a lot of people on staff. When we build out the network, we try not to break any rules. And with AlliedView and the fact that the product works very well, two of us can manage it,” said Felerski. “We have 5,860 ports that the management system can see. We were really surprised when we saw that number. We are responsible for a lot more than we thought.”

Everything Working Well

The quality of the networking products has been high. “Out of over 200 switches we have received from Allied, I don’t think anything has come in DOA,” he said. “We had to drop in all those POE switches for the VOIP project. They shipped us pallet loads of switches. We brought them in, unpacked them and just rolled them out.”

Felerski is pleased with the overall state of the network today: “We feel really good. There are just a couple of locations where we might replace an older switch with the newer X900 or X600 models – just to clean things up. Now of course Allied would like to sell us more gear, but everything is working well. That doesn’t do much for their sales, but it really makes our folks happy – that’s for sure.”

Looking forward, Felerski is certain the network will continue to grow and evolve to support the requirements of the county. “Government is always changing,” he said.

AIT_Profiles_Blogocon_small

 Copyright © 2009 Apropos LLC. All rights reserved.

TSI Switches from Analog Phone to VOIP Service from 8×8

It’s just improved the productivity of everybody. And from the business side of it, it cut our phone costs substantially.”
– Chuck Reynolds, Founder and Principal, TSI

Old Analog Phone System Too Cumbersome

Technical Systems Integrators (TSI), a provider of electronic design automation solutions, decided to upgrade its aging analog phone system. The existing system required significant ongoing maintenance and lacked the latest telecommunications features. “Configuring it was a pain,” said Chuck Reynolds, founder and principal of TSI. When the company moved its home office about two and a half years ago, it decided to take advantage of the opportunity to also invest in a new phone system.

TSI sells software, training and consulting services to companies that design and build integrated circuits and printed circuit boards. TSI is based in Maitland, Florida and also has field offices in Texas, North Carolina, Georgia and other parts of Florida.

The company wanted a phone system that could tie together its geographically-distributed offices. It was also looking for advanced features and a simpler, low-maintenance system.

Switch to Virtual Office from 8×8

After researching several phone systems, TSI chose Virtual Office from 8×8, a hosted voice-over-IP (VOIP) phone system. Virtual Office is a digital service that runs over broadband IP networks instead of traditional analog phone networks. 8×8 hosts a shared PBX in its data center, and customers access it via high-speed Internet connections from their offices.

“The software of the 8×8 system is quite friendly and easy to use. It has more features than you would ever care to use,” said Reynolds. The cost of using 8×8’s hosted service was also less than buying and managing their own PBX. “We ran the numbers on it and it was just so much more economical – less impact on us.”

TSI deployed Virtual Office, bought new phones from 8×8 and installed a high-end network router. “We got rid of our 800 numbers. We got rid of our conferencing service. All those things went away, and we just went to the VOIP system,” he said.

Reynolds recommends a good managed switch with QoS features and reliable Internet service for deploying a VOIP phone system. “That makes all the difference in the quality of service.”

Better Productivity and Lower Costs

All of TSI’s employees are now connected through the same virtual phone system. “The interface to our customers and vendors is extremely professional and streamlined,” he said. “We don’t have to have a receptionist. We can all conference in at the drop of a hat. We use it actually as our conference call-in for all of our [external] meetings, so we save money on the fees for the webinars that we do. Having everybody local from a phone perspective is great. It’s just improved the productivity of everybody. And from the business side of it, it cut our phone costs substantially.”

Reynolds continued, “We have it integrated into Outlook, so when somebody calls in, it pops up their contact record. I do my dialing out of Outlook. I can take the phone home or on the road and pretend I’m at the office.”

“If in fact the Internet does go down, each person’s extension is then automatically forwarded to their cell phone, so we never lose any calls that way,” he said. “So we don’t have phone lines now. It’s all Internet based. I recommend it highly.”

AIT_Profiles_Blogocon_small

 

 


Copyright © 2009 Apropos LLC. All rights reserved.