High level architecture and design goals

I come across many businesses that are not grounded in basic high level architecture and design goals.  Basically, they simply leap from features to building stuff, most often this happens because everyone is in a mad rush to just get something working as soon as possible.  The fundamental flaw in this approach is the belief that spending the time to build it “the right way” takes longer and is more expensive.  I’ve found the exact opposite to be true.  And for the record, the idea that you can just slop something together quickly for a minimally viable product and later insert a good architecture is pure non-sense – it never happens, ever.

Here are a few basic guidelines that apply to almost anything that you are building and I’m certain that there are plenty more that are specific to your particular challenge and some of the technology choices that you make along the way.

Support Hierarchical Configuration

Don’t cripple configuration automation and unnecessarily burden your operations team with configuration management.  Design configuration points in a hierarchical fashion such that all deployments derive from a base configuration and implement deployment specific configurations that override the defaults.  Wherever possible configuration should be modifiable at runtime and should be persistent across restarts.

Facilitate Production Troubleshooting

Don’t count on your development team’s access to production, it isn’t a good practice to allow a live debug session attached to your production environment.  When log messages are written to record exception situations they should include as much contextual information as possible in order to enable production support staff to recreate the conditions present at the time of the exception or undo data corruption that results from the error.

Fail Fast

Don’t unnecessarily retry what you already know won’t ever work.  If exceptional conditions occur the system will not be configured or coded in a way that directs it to retry the action that failed. This rule is particularly important where interfaces into 3rd party APIs are being configured.  If a 3rd party API is failing and we have no expectation that is should ever fail (say a pool API that provides us with database connections) there should be no attempt at reconnection.  The exception should be logged as a high priority exception (fatal) and messaged to a management system.

Automate Test Environment Management

Avoid accumulating manual test drag, there is a huge return on investing in test automation.  The design of test infrastructure will include a framework by which the overall test “suite” initial setup (configuration, code, results and data) is configured to a well known state and that individual unit test are otherwise atomic in their own setup and execution.

N+1 Design

Lots of stuff happens in the real world, stuff that you will never anticipate.  Ensure that anything developed has at least one additional instance in the event of failure.  There should never be less than two of anything.

Design for Rollback

Your release will fail, no doubt about it.  Any new design should be backwards compatible with previous releases.  Test your rollback before every release or you will get caught and the impact may be fatal.

Design to Be Disabled

Enable efficient maintenance and minimize outages – planned or unplanned.  Any system or service endpoint should be designed to be capable of being “marked down” or disabled.

Design to be Monitored

There typically are signs that a failure will occur soon, make sure you know that bad things are accumulating.  The system should be able to identify when it is performing differently than it normally operates in addition to alerting when it is not functioning properly.  An example of this principle is instrumenting the application to report performance statistics on page render times or query execution times.

Asynchronous Design

While it is nice to count on quick and efficient compute pathways, high scalability platform often benefit from offloading and distribution but this usually relies on asynchronous designs.  Wherever possible systems should communicate in an asynchronous fashion.

Atomic Compute and Stateless Systems

Don’t attempt to store state outside of your persistent data storage – you’ll unnecessarily create scalability obstacles and cripple your ability to build resilient platforms.

Scale Out Not Up

You’ll eventually not be able to buy a big enough server.  The system should be able to be horizontally split in terms of data, transactions and customers.


Outsource Unicorn

If I tell you that you can purchase a brand new iPhone for $5, what do you think?  Yet if I tell you that you can hire an outsourced developer for $10 / hour, somehow we set aside our dad’s advice that “nothing is free” and “you get what you pay for”.

Over the years, dozens of companies have asked me to join their company to help them “clean up their technology organization which they have unsuccessfully outsourced”.

Tell me if you’ve heard any of these before?

  • You should outsource that projects because engineering resources overseas cost about $10 / hour.
  • Our engineering team is buried for the next 12 months, just outsource that project.
  • That project is a one time project, don’t hire full time resources, just outsource it.
  • We outsourced all our engineering 3 years ago, it isn’t really working and now we need someone to come help us clean up.
  • You can hire a few outsourced resources to augment your core engineering team.

So while I believe that you may know of someone who has successfully outsourced a project, there are some real challenges to getting it to work well.  I’ve learned quite a few lessons along the way – here are some of these learnings.

As you know, communications is always a critical factor in any business – outsourced or not.  Communications in an outsourced relationship, especially offshore, is a very big challenge that is most often completely underestimated.  Not only are there usually language challenges, even if the outsourced team speaks english, but there are time challenges.  You should be prepared to manage workday offsets, sometimes by as much as 12 hours.  Outsourced companies will tell you that their resources will use technology to help communicate efficiently – Skype and Hangouts work poorly in many global scenarios – delay, echo, drops, etc…  Using Wikis, Basecamp, Slack, ticketing systems and email just like you probably use with your core team is helpful, but is not a great substitute.  Outsourced companies will also tell you that they will assign an on-shore resource that will manage the offshore team as a solution to communication and timeshifted work hours.  This approach helps, but again it is not great and at best it will add lots of cost.  It adds the cost of this onshore resource, but it also adds the cost of inefficiency in having a relay system in place.

In addition to communication, there is a very large issue of resource stability over time.  For whatever reason – maybe because they pay their resources poorly, outsource companies don’t seem to retain resources for more than a month or two.  The costs of changing resources is very high, as usual.  You will pay for the lost productivity of having to bring in a new resource and bring that resource up to speed.  Also, if the onshore manager, communication relay that I mention above happens to be the person who leaves your team then the impact is enormous.  Finally, keep in mind that if you reduce your demand for resources, of course the outsource company will move the resources and so starting another phase or even supporting your existing product will become very, very expensive – so manage your demand wisely or the supply will go away.

In my opinion, there is a big difference between engineers and coders.  Coders, simply implement – write lines of instructions to tell a computer what to do.  Understanding a problem and designing an approach to solving the problem using computer instructions is a totally different animal.  Designing an end-to-end platform that involved many, complicated interactions requires even more experience, education and skill – engineering is very, very different than coding.  I have yet to work with an outsource company that employs engineers and never an architect.  The results that I have routinely seen is giant pile of code, mostly undocumented and never, ever efficient.  By this, I mean there is never any use of modern object oriented constructs such as inheritance, if a routine is needed elsewhere – the code is copied and pasted elsewhere.  So when you have a bug, you probably have that bug in 10 places.  To make matters even worse, I’m just describing simple procedural routines.  I never, ever see good use of more advanced engineering concepts – for example, multi-threading or distributed processing.  All of this leads to a maintenance nightmare.  Sometimes you can lower the impact of this by hiring a full time employee as your architect, but even this is not foolproof.

The above are very serious landmines that are not easy to circumnavigate – you are not going to write an iron clad contract and secure a fixed bid contract to avoid all of this.  And while you may conclude that you should never use outsourced development, that is not really my mission here – you can use it, but it will be far more expensive than advertised and you will need to invest significantly in avoiding issues that will turn your project into a big problem.

Invest in your own health

Since I’ve been about 14 years old, I’ve been an entrepreneur.   I started mowing lawn, shoveling driveways and painting houses.  I bought a freaking office desk with my own money when I was 16 – ok I admit it that’s a bit weird.  I started writing software, in Pascal, on an Apple II+ in 1978.  I earned every penny to put myself through UMass/Amherst.  I became that legendary startup guy that worked 80+ hours.

Along the way, I lost track of the importance of eating right and exercising.  I was very aware of all the advice about taking care of yourself for all the right reasons.  I just never spent the time.  Until I was about 30, I completely got away with it – 5’9″ and 145 lbs – no matter what.  Then, 155 as soon as I turned 30, like it came in my birthday present – no big deal, 10 lbs.  As they say, onward and upward – working the long hours and on my way to 205 lbs by 50.  I wasn’t sleeping well, I spent several 3 to 4 days stints nursing a horrible back problem and my productivity was lower than years gone by.  I got really tired of feeling terrible and dragging ass all the time.

Starting in March, 2014 – no particular event or occasion – I started eating more reasonably, no crazy diet just smaller portions and better choices.  I also started going to the gym for an hour a day.  It was a big struggle to find an hour a day – how pathetic is that.  It was super boring too – headphones on an elliptical machine for a half hour and a half hour on some Nautilus equipment.  There were some strange mental wrestling matches about an hour a day, but in general, I stayed on track.  I lost the easy 15 lbs in the first 3 or four months and I started riding my bicycle in July of 2014.  The bicycle was far more interesting than the gym, so the rest of last summer was bit easier.  I still wasn’t convinced that I was committed, so I stuck with my 1985 Centurion LeMans RS 12 speed bicycle – that’s right, the 30 year old bike.  I used MapMyRide on my cellphone and my dashboard tells me that I rode about 600 miles that summer.  I went back to the boring gym over this past year’s record setting New England winter and weighted in at 170 lbs in March of 2015.

CycleBeing the geeky, data guy – I got a Garmin 810 GPS for my biking this summer.  It is awesome – it measures my heart rate, cycle cadence, power and mashes it up into a map overlay giving me all kinds of information about how I’m riding compared to previous rides and other cyclist in the area.  It also can track my ride live, so my wife can find out where I am along the ride.  I started riding in March, it was routinely 35 degrees and there wasn’t much daylight to squeeze in the rides.  My goal is to ride the same crappy bike 3000 miles before the cycling season ends – which is probably mid-November.  I’ve ridden approximately 2800 miles so far and here is some more data – 120 rides, 170 hours, 160,000 Calories burned with an average heart rate of 85% of the max.  Still 5′,9″ but now 155 lbs which is a loss of 50 lbs in the past 18 months.

So 170 hours was the hardest part, I’m still the crazy busy startup guy with a family and several other obligations and activities.  During these rides, I get to think about many things – a huge benefit to get away, uninterrupted.  I’ve thought about making this investment, not only for myself but for family, friend and my business.  I’ve done the math on the time against my typical work week and I believe I’ve cut my hours by approximately 10% over this past cycle season.  I’m certain that I’m in a totally better situation – I sleep far better, I have zero back issues, I enjoyed snowboarding all last winter, I’m awake and alert all day and far more productive throughout the entire day.  I’m certain that I’ve got far more than a 10% return on this time, not to mention the other benefits.  I’m not looking forward to going back to gym, I might try compu-cycling and use my rowing machine.

In any case, I’m thrilled that something happened and I changed my situation – it was worth it.

What this CTO does.

The role of a CTO is a topic that inspires many to ask so many questions.  What is the difference between a CTO and a VP of Engineering?  What to look for in a startup CTO?  There are literally hundreds of posts on the topic and deep comments threads explaining various experiences in various scenarios.  I think that is exactly the take away; the specific scenario matters.  The specifics being important and not unique to the CTO role.  A good sales leader in one business doesn’t necessarily work in all the other businesses.

My experience as a CTO has been in the area of building native cloud based, highly scalable platforms from the ground up.  The platforms have been built to process massive amounts of data, high transaction volumes and large numbers of concurrent users or device interactions.  When I say from the ground up, I mean that I run around with powerpoint slides and wireframes describing the business before a line of code is written.  The first goal is alway to sanity check the business idea.  The idea always changes, often in big ways (aka. the infamous “pivot”) – so it most likely is a total waste of time and money to write code to aid in describing the business value.  I’ve raised at least $20M with PPT and zero code.  Yet, I’m literally contacted two times a week by folks that want help or advice in writing prototype code so they can raise money.  What?  You might be talking with the wrong folks if they need a prototype or maybe you are not explaining your idea clearly.

Unfortunately, I don’t get to write code these days.  I’ve read all the comments about if the CTO doesn’t write code then go get someone else.  That advice is total baloney.  What is much more likely to be true is that an architect or lead developer is needed for that particular business and you should not being describing the role as CTO.

So what do I do as a CTO?  Thinking back over the past companies that I’ve been a CTO, I spend a lot of time translating business ideas to a platform concept or dear I say, vision.  I also spend time researching available technology to support the future platform architecture – I refuse to build things that already exist, I want to concentrate on building the new, non-existent stuff as quickly as possible.  I also spend time reverse translating platform capabilities to business use cases – in other words, how does the technology support solving the important business problem – in the end, that is the entire point.

I’m deeply involved in contract negotiations, especially in relationships that involve critical technology partners.  There are many important aspects of technology partnerships that will cto-areas-of-responsibilitymake or break the value of your platform as an asset and in turn, maybe even your entire business model.  For example, data rights and protection of intellectual property are critical.  Similarly, I spend a lot of time in helping sales and business development understand customer use cases and mapping it to our platform, or in identifying creative ways to fill gaps or in recognizing a general pattern that needs to be built into our platform.

I’ve been part of businesses that have raised hundreds of millions of dollars and have sold several of these businesses.  I spend a lot of time in the due diligence process – explaining our platform inside and out, describing our discipline and investments, forecasting future ideas and in general supporting the idea that we have a technology asset that is truly worth something and supports our business idea.

I’m responsible for recognizing intellectual property and doing something about it.  Notice that I’m not just leaving it as, filing a patent.  That is because filing a patent is just one small thing that you do to protect your IP; quite honestly, probably not the most important.  I believe that if you have something that is true IP, that you need to get it to market quickly and run with it as fast as possible because that is what makes it really defendable and it often is the key business differentiator that every business wants to include in their pitch.

I spend a good amount of time identifying and understanding emerging technology and trends.  I look for opportunities to use or integrate in support of solving problems or filling gaps; I’m also interested in potential threats or disruptors.

Finally, just beyond the first couple of weeks and months, there is a ton of team work investments and operational stuff that I need to make and this grows with the size of the business.  Having bought and sold a few business, I understand how companies evaluate a technology based business and what value is assigned upon acquisition but also how companies forecast future investment requirements.  In other words, buying a great technology idea running in my basement with no documentation run by cowboys is far less valuable than a well documented platform, running in a fault tolerant architecture cared for by a top-notch team of professionals with verifiable workflows.  Building that isn’t free, but well worth paying those investment taxes along the way.

See no code, sadly for me – but then, I’m not the lead developer or architect or even the engineering manager.


Avoid email bankruptcy

Email is useful but can be evil if you let it manage you.  Friends and colleagues often tell me that they are overrun by email, missing important communications and failing to execute.  I’ve even seen people declare email bankruptcy where they’ve lost control and stop trying to “keep up”.  My inbox has exactly one message in it as I write this post.emailInbox  I get plenty of email – Google lets me know that I have approximately 10G / year.  Below are some of the ways that I avoid email bankruptcy and have a reasonable relationship with email.

I read email 2 times per day, one time first thing in the morning and one time at the end of the day for a total of 1 hour per day.  Email is for asynchronous communications – in other words, things that can wait until the reader gets a chance to read it on their schedule; it is not 911.  There are plenty of better tools for simple, quick back and forth communications – txt, twitter, Skype, etc…  It isn’t my problem if folks don’t understand what email is for and how to use it.  But it becomes my problem if I train folks that I’m on email all the time and I will reply back in 1 minute.

When I do read email, I avoid the practice of skimming with an intent to spend more time on it later.  I think that it is important to understand context as well as message content, a re-read forces you to re-invest in both.

I don’t use email as my task list.  Currently, I’m using Producteev as my task list – works across all devices and does all the good stuff that I like – tag, schedule, subtask, share, assign to others, follow, note, etc.  So, when I’m reading email that includes something that I think I need to attend to, I create a task in my task list with everything that I need from the email – attachments, dates and other important information – and archive the email.  I literally do not need the email any more.

I don’t sort email.  I use to sort email, but two things dawned on me.  First, I had to come up with a fixed and static organization to sort message into.  I tried sorting by sender, by topic, by importance, etc…  As things changed for me over time, I was constantly adjusting and re-sorting.  It also occurred to me that I was spending time on preparing for a search that may never come.  At about the time that I cut over from Outlook to just using native Gmail, I started just archiving – no use of tags to simulate folders.  If and when I need to find something, I just use the built in search.  Admittedly, ever once in a blue moon, I have to spend some time in the advanced search criteria trying to remember approximately when a message occurred, looking for attachments, guessing at words that may have been included in the message.  But I always find what I am looking for and I’m totally confident that if I added up all the time that I spend on actual searches, it is far, far less than the cost of sort and resorting.

I make extensive use of Gmail filters and automation.  If you spam me, I mark the message as spam and Gmail does a decent job of not showing me any more messages from you.  If you spam me again, referring to the earlier spam that you sent me, I mark the message as spam and I call you a name.  I’m an active “un-subscriber” – meaning, I do not just let some service continuously send updates that I’m not interested in – I just cut the cord.  Finally, I make extensive use of filters to bypass my inbox and go directly to archive when email is routine status updates and system type message that are good to have as references but I don’t need to waste time reading or managing messages about all systems being healthy.

I use Boomerang to help me follow up on communications that might otherwise go into a blackhole.  I don’t need to create a task to follow up with so-and-so when I send an email asking someone to do something.  In most cases, my team automatically knows to follow up with me, but Boomerang helps me remember without having to keep the message in my inbox as my reminder system.

Service Platforms not Product

For some time now, technology appears to be trending away from selling products and towards selling products as a service.  Netflix lets you stream all the movies you want, Amazon delivers all the books you want.  Movies and books are migrating from nouns to verbs. 5fc2ba31a62a343 Amazon has migrated from a product to a platform for selling products and they are selling services that they did not create.  Apple, Microsoft, Facebook and Google all have platforms that enable others to build services.  API are extensive so that ecosystems can evolve and support various businesses – even competitors.  Trying to build all the services in an ecosystem or network doesn’t scale, so you must build an open platform. Heck, at Linkable Networks, I’ve been building an open platform that changes printed coupons to digital coupons and enables partners to build their loyalty platform using my API.

The consumer side of this is that product ownership is migrating to access.  Consumer’s don’t need email servers anymore, we use email as a service.  We subscribe to cable and cell services.  We’re quickly heading towards not owning any movies, books and music.  ZipCar has pioneered transportation as a service.  AirBnB is offering hotel as a service.  How much longer before education, games and many other things are delivered as services?

Beyond the MVP

One of the most important product management concepts is getting to market sooner rather than later with the Minimally Valuable Product (MVP) and iterating or pivoting as you gather feedback and learn, especially if you are building new, innovative products.  This is simply the 80/20 rule applied to product management.  Ignoring this concept and you risk over investing before you truly understand market requirements and you are very unlikely to get a return on your time and money.

MinimumViableProductBut what happens after you’ve validated that your MVP is on-track and valuable?  How do you decide how much of the remaining 20% improvement is worth your time and money?  Information is the missing ingredient and often it doesn’t exist when you’re building something new.  The Imitation Game movie is about how Alan Turing crack the enigma code by building what many folks believe is the first computer.  The key was reducing the breadth of the problem, reducing it again and finally searching for a pattern to build into your product.  You can always expand the scope later.

How do you get the non-existent information?  In years past, the only way to get non-existent data is to gather the information yourself – often this is slow and very expensive.  All the map product guys employ many people to drive around and gather information on roads and buildings.  Startups never have the resources that Google has so alternatively today, crowdsourcing is an outstanding method and alternative to gathering your own data.  For example, Waze designed a product that encouraged their users to provide feedback and they crowdsourced information that would have cost many millions of dollars.

Finally, when you’re in the trenches of reducing the problem scope and searching for patterns, be on the lookout for technology game changers – i.e. internet, mobile, cloud, etc. – that change assumption and unlock new pathways.