Springpad

I don’t have a very good memory – and like most people with a family of five, I have many things going on.  Some routines are repetitive – medical appointments, school sports, pet related chores, grocery shopping and home maintenance.   Some tasks are personal health, well being and organizational – exercise, diet and Getting Things Done (GDT).   We also have an occasional big family events that take a massive amount of work, months, to coordinate – Bat Mitzvah, graduation party and college application/selection to name a couple.  Coordinating with everyone in my family is a big challenge – there are many moving pieces, late breaking changes and necessary communication.

Fortunately, my entire family is very technology oriented – everyone has a cell phone, email account, online calendar and our own social network; we utilize many of the typical online services such as shopping sites, travel portals, media sharing and financial management tools.  Things quickly get complicated when we spread these services across five people.  While we’re able to gain some efficiency within some of these individual services, many of them lack coordination across our social networks and they fail to roll up across our real life events in any type of comprehensive, manageable container. Over the years, we’ve tried many tools and techniques – paper, Covey organizers, refrigerator calendars, spreadsheets, Microsoft Outlook, Google docs, Evernote and Cozi; all come with some number of short comings, silo issues and lack of integrate actionable data.  Stress builds and my brain hurts.>

These problems are exactly what Spring Partners set out to address with Springpad.  My entire family shares an account that we use to remember stuff, integrate actionable data across other online services and leverage our trusted social connections as we manage our real life events.  We’re able to aggregate “My Stuff” in meaningful combination and coordinate calendar and communications across multiple channels – TXT, email and mobile interfaces to Springpad itself.  There is a light social network integrated within Springpad – the usual “follow” other Springpad user and includes the typical “share” your stuff with your Facebook friends and Twitter followers.  Springpad also attempts to solve the empty notebook problem by offering many pre-built, pre-organized applications around many common life events like getting organized, meal planning, maintaining a home, parenting, traveling, celebrating, exercising, learning and working.

I’ll use a simple example.  My plan in life was to simply enjoy wine, not ever invest in learning the elaborate details and subtle nuances that yielded good wine and more importantly wine that I like.  Yes, I planned to simply follow others in this life pleasure – ride the coat tails of others, lean on my friend’s investment in wine knowledge.  In general, this plan was working well.  But I still had the memory gap problem; how do I remember the great wine at last Friday night’s dinner party.  Back in the day, there was only good old paper and pen – not often in my pocket at a party and a frequent washing machine victim.  More recently when I’m able to sneak the smart phone past my wife’s what not to wear review, I’m able to electronically note the wine.  Still a pain to type name, vineyard and year – but if the wine is good, this was my only shot at remembering later.  So this was a perfectly sufficient solution – but I realized that I really like almost everything that certain friends liked.  Unfortunately, I’m not always with my friends when they’re drinking wine, so I’m not there to note the ever expanding collection.  I’m off the coat tail – FAIL.  How do I keep up with this important endeavor?

One of Springpad’s nicest applications is a Wine Notebook sponsored by Gary Vanynerchuk.  Using Springpad’s Wine Notebook, I am able to collect, organize, share with followers, include in my own plans and act on the information.  I am also able to follow my other trusted friends, who also share their favorite wines.  I am even able to see what Gary is recommending.  I am able to “spring” a wine into my collection and use it in many ways.  I can reference that info on my phone while at the store and I can use it later in my party planning notebook where I am keeping track of shopping list and things to do.  Even better, my friends can see what wines I like and bring it along when they come to my house for dinner.  I able to add comments to my wine data, categorize, note vineyard and pricing information or even attach a video presentation by Gary.

The power doesn’t stop there.  While, it is great that Springpad lets me continue riding wine enthusiast coattails, I think the real power is that I’m now able to reuse “My Stuff” in managing my other life events.  In planning my next Napa visit, I can reuse all this wine data to organize my vineyard tour, using a travel planner notebook, and work diligently to verify everyone’s comments on nose and palate.  I can search and include information on my travel schedule, car rental, hotel stay, restaurent reservations and local area friend meet ups.  I can keep track of details on my trip, include picture, videos, comments and share with my followers so that they can plan a similar time later.

Finally, less stress and fewer brain cramps.

Droid Ups the Anti

I like Apple; they innovate and build some products that I like to use because they work well and help me get important things done in easy and intuitive ways. I hope they keep creating great products and I totally respect their efforts to protect their intellectual property. I don’t buy everything that Apple sells. I understand, but don’t like Apple’s walled garden approach; I don’t like that they gate keep their application store.

I used an iPhone a little over two year ago for about one year and I mostly liked it; I did not like AT&T’s network, many calls were dropped and I didn’t have service in places that I needed service. I’ve been using a Verizon Blackberry for the past two years; the network service has been outstanding and the phone has been rock solid. I didn’t like the browser or the application support.

I’m a huge fan of competition and free markets – I believe everyone wins in this environment. I’m thrilled by the great progress and promise of Android. I really wanted to get a Google’s Nexus One, but I need Verizon not T-Mobile. Last week, I bought a Verizon Droid phone and I love everything about it. I like it more than the iPhone because it runs multiple applications, it has a QWERTY keyboard as well as the virtual touch keyboard and it integrates beautifully with many of the Google products that I often use.

But my real excitement isn’t so much about the many cool Droid features or endless open application market place – I’m jazzed that this is the real deal, real competition. Android is going to push great companies like Apple to go further to win – the consumer is going to get great stuff out of this competition. I know that Apple is cranking up their patent suit machine, I get that they have to do that. But I don’t believe they’re DNA is about fighting battles in courts, they’ll fight in the market place and we’re going to see the pace of progress rocket.

Now we just need the carriers to open up.

Recruiting in Down Economies

I’ve been on both sides of the hiring equation for a number of years.  I earned my recruiting MBA on the streets of the dot com era when we were challenged to aggressively compete or simply loose.  It was a crazy seller’s market where companies did unusual things to attract talent.  Companies offered BMW signing bonuses and nearly everyone allowed employees to bring their dog to work.  Well, I’m sure that everyone knows how that story ended and thankfully those lopsided gold rush days are gone.  Ten years ago, the market was uniquely unbalanced and clearly today’s market is nearly as lopsided in favor of hiring companies.  Unbalanced marketplaces support all types of bad behavior, at least temporarily.  But I think more is at play here than just cyclical job market trends and the corresponding shenanigans, I think there are many companies that fail to recognize the importance of managing a top notch recruiting process even when they’re holding nearly all the cards.

I’m aware of many wise and savvy business people stating the enormous importance of hiring – most often they’re talking about attracting great resources, managing a diligent and complete inspection, closing the deal for “A players” only and then quickly realizing the newly added benefit.  While I agree that this happy path is spot on, I think that it stops short of many other important parts of hiring.  So I thought I’d share a few things that I’ve found critical in stellar recruiting and hiring execution.

I realized early on that recruiting and hiring is a team sport with many moving pieces, often including external resources like recruiters, agencies and outsource companies.  There is no single formula that works for all companies, but there is certainly one that fits each company.  Sure, I want to get the hiring done, but in most case it is a work flow that I have to repeat and scale with business growth.  So, I think that it is worth some significant investment and optimization.  One of the very first things I put into place is some type of tracking tool.  I’ve used whiteboards, paper, spreadsheets, MS-Exchange public folders and software packages to keep the work flow moving quickly.  I haven’t found a significant amount of difference with any of these, it usually depends on scale and if your team includes external recruiters.  This single tool offers sales pipeline like tracking that is easy to keep current; the entire team always knows where all the candidates are in the work flow, who owns next steps and how long are things taking.  Investing in this type of information has helped me identify execution flaws, operational bottleneck and recognize places that need focus.

I typically insist on phone screening – there are always a bunch of really easy show stoppers that can save everyone, including the candidate, time early on.  When using an external recruiter, I like to divide the screen criteria into things that the recruiter can scan for and things that our internal team must check.  Many recruiters will attempt to differentiate themselves by claiming they can handle more of the technical screening – I have not found this to be true when it comes to technical hires – so splitting the screen is important.  Some recruiters will push back on the screen and propose the interview day at their place – where they’ll line up 10 candidates in one day at the recruiter’s office.  I haven’t found this to be efficient either – the phone is best because I find it much easier to find 30 minute slots in the schedule and I don’t need to hide off site to get important work done in the office.

I invest heavily in team interview preparation – it pays big dividends when I get it working, but it usually takes a few iterations before the team is efficient.  So immediate post-interview group discussions are super important to running a smooth hiring machine.  If I’m using a professional recruiter, I definitely include them in the tuning process to make sure we’re targeting and screening correctly – we rarely get it right initially.  I like to divide up the interview agenda across disciplines and expertise on my team.  I find that some small overlaps is useful for consistency comparisons, but I usually have an agenda that needs to be mutually exclusive so that we can cover everything with sufficient depth.  I always include in-depth technical interviews, yes whiteboard exercises, puzzle solving, architecture walk through and deep dives on recent deliverables.  When I am the candidate, I easily eliminate companies that don’t have their act together – large overlaps, gaping holes and folks reading my resume for the first time during the interview quickly worries me about who else the company may have hired with such a poor work flow.

Once I am tracking the work flow and we’re efficiently interviewing, my interview team always follow up with every candidate, 100% of the time, no matter how big or small the role is or how well suited the candidate matches what we’re looking for.  It is a super small world, it is ignorant to believe that reputation won’t be communicated and bad news travels very fast.  I think this problem is much worse with the senior candidates because they’re the reference check network.  Great candidates will thoroughly check my company out and I definitely want five star backdoor references, even from candidates that we’ve rejected.  Plus, it is simply the right thing to do and it usually only takes a couple of minutes.

In addition to maintaining 100% follow up, I believe that it is in everyone’s best interest to terminate the interview process when there is a big showstopper.  I can’t tell you how many times folks on the interview team choose to waste everyone’s time, including the candidate, rather than politely and respectfully stop the interview.  I can almost guarantee that it won’t be long before the candidate adds up this bad behavior.  More bad karma to those that dodge the delivery of bad news and again, not the right thing to do.

Finally, I’ve found that reference checks are almost as important as the interviews.  I really have two goals during reference checks – the first is to verify some of the interview data.  The second is to get information that will be helpful in working with the candidate once on the team.  I rarely uncover out right lies, occasionally some creative historic accounting.  So, I spend nearly all my time on later goal.  I rarely rely on the references given to me by the candidate – they’re almost always, but not quite 100%, spectacular.  I’m frequently able to leverage my and my team’s network to find a connection to top-notch candidates who may not be on the candidate’s payroll.  With tools like LinkedIn, Plexus, Facebook, Twitter and good old Google there is no excuse for not finding the backdoor references.

I see many companies today abandoning these types of practices – maybe because the supply is so ample that they believe they can perform poorly.  Or maybe they didn’t experience the intolerance of candidates when they held all the cards a few years ago.  In either case, poor performance in an area that is well known to be a top priority points to deeper, potentially fatal, flaws.

Cache Poor

How much do you really know about managing high traffic web sites?  I was asked an interesting question recently.  Here it is:

You are operating a web site that receives about 4 million unique visitors per day from around the world seven days a week, 24 hours per day.  The load is spread evenly throughout each day.  Performance was starting to become unacceptable.  You’ve identified that the current bottle neck was your memcached machines, serving as a back cache to your web front ends.  Below is a diagram of your simple architecture.  What would you do about the problem without bringing your busy site down?

Apple Battles Android Tsunami

A mere tic of the patent clock ago during Apple‘s January 2009 earnings call, Tim Cook fielded questions about how Apple plans to stay competitive amidst challengers like Google and Palm.  I expected to hear Tim lean on the stock technology leadership and innovation response.  Tim took a different tact and ended with “if others rip off our intellectual property, we will go after them”.  I remember thinking that it seemed a bit fiery, even slightly mistimed and awkward.  I doubt Tim  suddenly remembered Apple’s multimillion dollar patent portfolio and got all fired up, but rather it felt like an attorney coached non-answer.  Or could it be the Palm Pre was Apple’s first real competitor and the device just happens to be co-developed by a former Apple employee, Jon Rubinstein?  I doubt it.

Major wars always have numerous key events that lead up to the first shot, so this plot thickens.  Remember Steve Jobs 2007 Macworld keynote where he introduced the multitouch on the iPhone including a slide reading “Patented!”.  Steve was referring to the now famous patent #7,479,949 which is routinely referred to as the iPhone patent.  Apple immediately orchestrated numerous media releases about being awarded the multitouch patent.  It is a common practice for large technology companies to build a patent portfolio that they’ll routinely revise and loop in features to target competitors in an attempt to cut them off.  Yet, I don’t think Apple is the big bad wolf and Palm, for example, has a pretty hefty patent portfolio of its own.  If you read some of Palm’s patents, it is very easy to think that Apple’s got some ‘splainin to do.  For example, Palm’s patent #7,007,239, almost exactly describes how the iPhone uses buttons for dialing, call history, contacts and speed dial that stays on-screen as you toggle between them.  In addition, it covers pulling up contacts by just typing in initials – exactly how the iPhone has implemented it.

Meanwhile, Google’s Android Tsunami is picking up momentum and support is swelling.  While certainly not a sneak attack, perhaps the Nexus One invasion was the aggressive trigger point that motivated Apple’s patent office to abandon diplomacy and fire the first shot.  Yesterday, Apple has announced that it has sued HTC for, in Steve Jobs’ words, stealing Apple’s patented technology.

Make no mistake, this is no angry, hot-headed, knee jerk reaction – this is a war, that Apple has prepared to fight and they’re going after Google now!   This doesn’t appear to be a full frontal attack either but rather some type of island hopping strategy that starts with the Asian Android alliance, then onto Motorola and keep whittling away at the alliance.  Apple has organized a very careful first shot in this battle.  They’ve split their suit and filed in two different courts.  On one front, the suit includes 10 patent infringements covering user interaction with a touchscreen, filed in U.S. District Court in Delaware.  The more telling suit includes 10 operating system level patents, some dating back to NeXT, filed with the U.S. International Trade Commission in Washington.

It should be very interesting to see how HTC handles this suit and whether or not Google is going to get involved now.  Eric Schmidt, another former Apple executive, and Google released a statement: “We are not a party to this lawsuit.  However, we stand behind our Android operating system and the partners who have helped us to develop it.”  Of course, it is never too late or difficult to draw others into wars.  As with all patent battles, HTC and probably Google’s patent lawyers will likely find “prior art” to demonstrate that Apple’s innovations were not new.  In this case, this task won’t be any D-Day sized task as Palm sold touch-based mobile phones for years before the iPhone.  Also, a quick look at Synaptics’ patent portfolio will show plenty of intellectual property around touch screens.

I doubt this war will be won with patent battles, they rarely ever turn out that way.  But clearly Apple is trying to buy some time in order to get to higher ground as the Android tide has made landfall.  Creative innovation will win the war and the consumer should enjoy the rewards.

Inside the Gates of Hell

Last week, I called out Dilbert for avoiding the dreaded load test – it is hard, expensive and takes too much time.  Yeah, but.  I use my stuff to make money and if it is slow then nobody will like it and I will die in the gutter, gasping for air.  So, I simply must enter through the gates of hell and get some testing done – no way around it.

So, I checked with NBC and despite the recent loss of Olympic coverage they’re still not desperate enough to cover my idea that testing directly on a production environment is news worthy.  They mumbled something about boring and too geeky.  Pfsst.  I’ll press on.

Beyond the obligatory warning, “Don’t blame me if you screw up and spend your weekend discovering that your disaster recovery plan has a few holes that you’ll need to fix on the fly”, let me bullet point the typical test in your production environment stuff here (I will try to add some value later in this post):

• Make sure you have a disaster recovery plan and have a backup or two handy
• Conduct your test during the lowest traffic periods
• (Optional) Redirect users to a “Temporarily Out of Service” page, if you don’t want to include any real users in your experiment
• Increase load volumes gradually to avoid complete system crashes

A quick side bar, perhaps a good subject for a later post – disaster recovery plans that haven’t been validated through execution are virtually useless.  Not that I think that combining these complicated and risky test scenarios is optimal, but I certainly think that validating a disaster recovery plan precludes any type of live load testing in your production environment.  So if you haven’t done this, stop here – now!

Many technology platforms today can be vertically subdivided front to back – in particular to enable maintenance and upgrades.  This is most often accomplished with a load balancer rule change that sends traffic down one path or another.  I can use this technique to send folks to the out of service page or maintain service while you use either all or part of your production gear to test.  Further back in the stack – caches, queues, indexers, databases – it gets a bit more complicated and depends highly on your architecture.

But this common, well covered approach only seems to solve the where do I get a test environment problem.  I still have all the data problems that I mentioned last post and I still have the load generation problems.

The technique that may be a little less obvious than just separating out a test environment within my production gear is what I’ll call load testing through reduction.  Basically, the idea is to test various parts of my infrastructure under load by removing a component member and measure the increase in load being served by the remaining production environment.

For example, of course I’ll use the easiest component here, lets say I want to measure the front end server performance under load and I have 10 front end servers.  Of course, I already know that under “normal” traffic that my array of front end servers are performing well within the acceptable range at 25% of their peak capacity – so removing one front end should not tip everything over.  The starting measurement is when all 10 servers are working and there is measurable “normal” traffic.  Removing one front end server should increase traffic to the remaining 9 servers.  It is also worth measuring the impact on other components in the stack – caches, queues, indexers, databases, etc…  Some components will have an initial reaction to the change in load and then should settle down to a new normal performance under load.  In a perfectly scalable world, I should notice approximately a 10% decrease in performance at the remain front end servers.  In the real world, mileage varies and knowing the details is super valuable in forecasting and planning capacity.

I can continue using this technique, gathering multiple data points, until I reach various stress points.  Usually a stress point is defined as the point in which performance becomes unacceptable or worse, failure emerge.   I typically stop before catastrophic failure and usually have plenty to work on well before many stress points.  There is tons of learning available using this technique – developing a deep understanding or my product’s scalability on a component by component basis is incredibly useful in guiding future development investment strategies, inspiring consensus building around “acceptable” performance metrics and even budgeting future infrastructure spend.

This technique allows me to avoid time consuming environment building, expensive and inaccurate approximations as well as the nearly impossible and risky data moving and cleansing  for testing purposes.

No data movement, cleaning, backups/restores, load generation needed.  Know thy product performance and scalability.