Cloud Services Austin, TX
This past week I got a call from a company needing help with Citrix. Their entire environment was down and they had no idea why nor did they have the expertise to fix it.
8:30AM – Initial Phone Call
Fortunately we have a great guy on staff available to engage with this company quickly to help them out of their difficult situation. He is Citrix certified up to his eyeballs, has 15 years of Citrix experience and has done some 35+ Citrix projects or engagements this year alone. Perfect match.
9:31AM - Engineer Engaged
Once engaged, our Citrix engineer realized that something was amiss but what he was seeing did not make sense in terms of how Citrix functions. He sought out a second opinion internally to try and reach a resolve, another 15+ year Citrix Admin.
10:45AM - 2nd Citrix Engineer Engaged
Now both were equally perplexed as to the root cause of the issue after utilizing all of the tools at their disposal designed specifically to help them arrive at root cause.
11:15AM - Hmmm.
11:30AM - The problem was isolated to being a Group Policy issue. However, there was no group policy enforcing the behavior they were seeing.
12:00PM – Ghost Group Policy identified that was applying policy but not otherwise visible.
12:10PM – Offending Group Policy eliminated.
12:15PM – Normal order restored.
Ultimate cause of the outage: A client-side IT Admin did not finish creating a Group Policy but got far enough along that incorrect policy instructions were indeed being enforced across the Citrix environment.
Productivity cost: 60 employees unable to work for 4+/- hours at an average wage of $30/hour = $7,200. + $20,000 in lost revenue for that 4 hour period based on annual revenue earned per hour numbers.
Ballpark business impact and cost of this one issue: $27,000.
It should have gone something like this:
8:30AM – No Call, no issue. Business continues as normal.
…but, let’s just say there was an issue.
8:30AM – Initial Phone Call, issue identification
9:00AM - Engineer Engaged, VM restore initiated
9:05AM – Normal Order Restored
Productivity cost: 60 employees unable to work for 30 minutes +/- hours at an average wage of $30/hour = $900. + $2,500 in lost revenue for that 30 minute period based on annual revenue earned per hour numbers.
Utilizing cloud services, total cost for this one issue: $3,400.
Business impact and cost that could have been prevented: 88% or $23,600.
Technology is supposed to be is a business multiplier. Technology should by its existence allow a business to find savings and be more productive, to do more with less.
In this small example, technology was not a multiplier; it was a subtraction that did not have to happen.
This is one reason we are seeing Managed Services and Cloud Services adoption by business increase exponentially, because businesses are realizing that the “business by subtraction” model does not make a lot of sense in most instances.
Technology is not perfect, that much is clear, there will always be outages, but there is no reason for a business to accept the same technology limitations of 13 years ago, today. There can and will be outages, but there should not be many that the end users feel.
Utilizing cloud services in this instance, there would not have been an outage as there would not have been an inexperienced engineer deploying group policies in the first place.
In the unlikely event that this mistake did occur, recovery would have happened in less than five minutes as the VM was restored.
In the plus column, utilizing (public or private) cloud services, this company would have fully redundant data centers located in three time zones, with fully redundant power, fully redundant internet connectivity, and diesel generators at each location that they could never practically afford on their own.
Today they have multiple single points of failure starting with their one data center and singular office.
There would be no capital expense for hardware, no space requirements for a full data center, reduced electricity, heating and cooling costs.
Cloud services make sense for a lot of businesses today. Or, rather, paying a premium for inferior data center space, paying a premium to build, staff, and support IT infrastructure, while absorbing unnecessary capital cost makes less and less sense each day.
The fundamental point here is: Take a look around. Does it make good business sense for you to deliver and consume IT services the way you do today?
Are you, as in the case of this business, taking on unnecessary cost in the name of security and better productivity that are not really founded?
My guess is yes. Cloud services, like hosted email, servers/applications and desktops are not for everyone.
IT is, at its core, a service that should make the balance of a business more productive. Just like every other service, from legal advice to oil changes, you should be looking for the most cost effective, secure way to get the quality IT services you need.