Cloudy with a chance of issues?

CloudSwallowerFor a few years now, the technology meteorologist have been forecasting a heavy amount of cloud computing in our future.  The business side of the house has taken note, so the typical emerging technology questions have arisen from pundits and enthusiast alike.  Emerging technology is always challenged as it gains popularity and goes mainstream; great, these challenges often accelerate progress in response.

Over the course of the past 20 year, I’ve started a few technology based companies.  Almost all of them have offered at least some form of what is now called Software as a Service.  Up until recently, this always meant spending tons of precious capital on enough technology infrastructure for some number of production, staging, QA, performance and development environments.  Firewalls, load balancers, front end servers, databases, storage and backups, oh my.  And of course, there is the pleasure of the icy cold co-location environment – the cage assembly, power struggles and pipe installation delays – nearly as fun as short code provisioning.  Very expensive to do correctly, times the number of environments that we needed – sometimes I even needed more budget than marketing. Always a top 10 board meetings getting the buy in, often we had to settle on using less than sufficient gear for dual purposes – which to this day, I still content that we spent more on managing switch overs, fixing problems, recovering from mistakes and educating our technology teams on the rules of the jungle.

More recently, I used Amazon’s cloud environment for everything.  There were huge advantages and cost savings.  Most notably, we did not need to sink enough cash into technical gear to cover the peak traffic requirements of our product, especially since average requirements were almost always far lower than these peaks.  We built various types of environments, archived snap shots of the configurations, brought things up during transient needs and shut stuff down when we were finished  – saved enough cash for marketing programs.  The perfect example, Staging environments, are really only used as a final sanity check before we push product to production.  We saved cash, plus when the idle environment lay in steely cold snapshot storage, I rested assured that no midnight buckaroo is slipping quick configuration tweaks into our staging environment that never seem to make it all the way to our production environment.  I could go on and on about the advantages we reaped, the problems we avoided.  As a veteran startup guy, I field plenty of questions from others and I almost always rank using cloud computing as a must-do.

StabilityBut, I’ve fielded plenty of healthy challenges too.  The two big ones have always focused on security and stability.  Stability usually means, what happens when someone like Amazon goes down.  Oh, it happens.  Not often, but everyone has their special day – Google, eBay, etc… My teams, all cracker jack technologist were startup budget magicians with vested interest in maximizing shareholder value, trust me.  But, I remember quiet a few more stability issues when we built our own product clouds – certainly never less than I experienced while using Amazon.  Some were mistakes, but most were simply because all we could afford still had a single points of failure or two – it is really hard to avoid and is usually cost prohibitive.  One time, our co-location provider fried both their OC-48 metro cards; their on-site replacement card failed and the card they flew in within 4 hours failed – a fun day of downtime for all was had in Boston.

spyvspyIn addition to stability, folks want to be assured that cloud computing security risks pale in comparison to our own security.  Security is one of those things in life that has an exponential cost compared to reducing risks beyond a certain threshold – take a look at how much money the Pentagon or every major financial firms spends on security, it usually appears in the same story about how someone has successfully breached their guard.  But the scenario at hand seems to be that the hacker makes it through Amazon’s security to gain unusual access to our data.  Is this more likely than making it through our cloud’s security measures?  I’m certain that Amazon has top-notch security infrastructure and numerous dedicated security experts diligently fighting the good fight to protect their multi-billion dollar buisiness.  So, I’ve always spent more of my time paying attention to our own configurations – ssh keys, port openings, software stack vulnerabilities and most worrisome, our own product which always seems riskier and more vulnerable to the likely interested culprit.

So don’t cancel your outdoor plans – the clouds are the high wispy, non-threatening type and you don’t have to spend all your cash on sun screen, umbrellas and air conditioners.


About chipcorrera

Chip Correra View all posts by chipcorrera

2 responses to “Cloudy with a chance of issues?

  • Steve Straminsky

    Great post, can you share any Amazon costs?

  • chipcorrera

    We constantly changed our configuration profile – after all, that is part of the benefit. But to give you some idea, we ran approximate 16 EC2 instances – mostly medium sized. I believe the configurations included 2 Apache stacks (load balancers), 4 Tomcat/SOLR stacks, 2 Memcached stacks, 5 MySQL stacks, 2 side-tier stacks, and a command and control machine. We would also stand up a staging environment – probably 10 additional machines – once or twice a month for testing purposed, which lasted 2 or 3 days. So this probably adds up to a 20 machine/month average. We also allocated EBS storage for many of these machine, not just databases – for example, the fronts each had numerous SOLR instances because we shared the entire architecture from day one. Finally, we had not yet taken advantage of Amazon's cost savings via reservations – which has benefits beyond saving dollars. The average cost was approximately $3500/month and if we had reserved these same 20 instances for at least one year, it would have easily been less than $2500/month.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: