You know why I’m here

Just some blogging fun here…

In the world’s biggest noise factory, the SuperBowl, Marshawn Lynch continues to straight arm the circus performers as he responds to every question with “You know why I’m here”. He defies the NFL’s mandates, wearing his unapproved beast mode gear while trending on twitter. Genius media stunt or pampered starred athlete who is breaking his contract? Marshawn Lynch
Meanwhile, “All-Pro Stanford Graduate”, Richard Sherman attempts to redirect the mob to more important matters. Mission accomplished, I’m entertained by the entertainment (and remember, sports entertainment is the NFL’s business) and I will survive without the philosophical gems laid down by NFL players.

I will admit that I like the phase “You know why I’m here”; it got me thinking. Not about checking the box, conforming to the rules, avoiding the penalty – but about strongly believing in something and sticking to it. Cutting through the noise, facing adversity and staying on a mission is difficult and when you take this path, you’ll stand out which will make it even more difficult. I see many examples of following the pack, getting into a routine, doing the comfortable stuff – all ok, not bad. You definitely need to take care of some basics, but remembering why you are here and pushing is super important.

Full disclosure, I’m a Patriots fan.  Patriots 30, Hawks 20.

Go Patriots!

Now I have to go find a t-shirt with this quote.


My Bash Profile

In the past couple of weeks, several folks have asked me to share my bash profile.  I’m not sure why, maybe better bash profiles are high on everyone’s new year’s resolutions. 🙂 This post should also give my daughter a good chuckle and reassure her that I’m still a geek.

In any case, these days I mostly work with Macs and Ubuntu AWS instances and the bash profiles are a bit different.  I’ll document my Mac profile, I’m sure you can tweak to your OS.

Finally, these are the configurations that I have found personally useful in my work. I’ve accumulated this profile from many folks that I’ve worked with and several online resources – sorry if I’m not properly giving full attribution, I assure you nearly none of these configurations have come from my own thinking.


# -------------------------------------------------------------------
# Description: This file holds all my BASH configurations and aliases
# Sections:
# 1. Include other sources
# 2. Environment Configuration
# 3. File and Folder Management
# 4. Searching
# 5. Process Management
# 6. Networking
# 7. System Operations & Information
# 8. Development
# --------------------------------------------------------------------

# --------------------------------
# 1. Include other sources
# --------------------------------

# Source any base profile
[[ -s "$HOME/.profile" ]] && source "$HOME/.profile" # Load the default .profile

# Source Bash base aliases
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases

# -------------------------------
# -------------------------------

# Change Prompt
# ------------------------------------------------------------
export PS1="______________________________\n| \w @ \h (\u) \n| => "
export PS2="| => "

# Set Default Editor
# ------------------------------------------------------------
export EDITOR=/usr/bin/vi
export SVN_EDITOR=vi

# Set default blocksize for ls, df, du
# from this:
# ------------------------------------------------------------
export BLOCKSIZE=1k

# ------------------------------------------------------------
export GREP_OPTIONS='-D skip --binary-files=without-match --ignore-case'

# Maven
# ------------------------------------------------------------

export M2_HOME=/usr/local/maven
export M2=$M2_HOME/bin

# ------------------------------------------------------------
export MYSQL_HOME=/usr/local/mysql

# ignore .svn in filename completion
# ------------------------------------------------------------
export FIGNORE=.svn

# ------------------------------------------------------------
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home

# EC2 Tools
# ------------------------------------------------------------
export EC2_HOME=/usr/local/ec2

# EC2 Chip's Environment
# ------------------------------------------------------------
export AWS_ACCESS_KEY="yourkeyhere"
export AWS_SECRET_KEY="yoursecrethere"
export EC2_CERT=~/.ssh/yourcerthere.pem
export EC2_PRIVATE_KEY=~/.ssh/yourkeyhere.pem

# My Scripts
# ------------------------------------------------------------
export SCRIPT_HOME=/usr/local/scripts

# ------------------------------------------------------------
set -o vi

# Set Paths
# ------------------------------------------------------------
export PATH=".:$PATH:/usr/local/sbin:/usr/local/mysql/bin:$M2:$EC2_HOME/bin:$SCRIPT_HOME"

# Command configs
# -----------------------------
alias cp='cp -iv' # Preferred 'cp' implementation
alias mv='mv -iv' # Preferred 'mv' implementation
alias mkdir='mkdir -pv' # Preferred 'mkdir' implementation
alias ll='ls -FGlAhp' # Preferred 'ls' implementation
alias less='less -FSRXc' # Preferred 'less' implementation

# ls family
# ------------------------------------------------------------
alias ls='ls -la' # Standard
alias lx='ls -lXB' # Sort by extension.
alias lk='ls -lSr' # Sort by size, biggest last.
alias lt='ls -ltr' # Sort by date, most recent last.
alias lc='ls -ltcr' # Sort by/show change time,most recent last.
alias lu='ls -ltur' # Sort by/show access time,most recent last.

# cd family
# ------------------------------------------------------------
cd() { builtin cd "$@"; ll; } # Always list directory contents upon 'cd'
alias cd..='cd ../' # Go back 1 directory level (for fast typers)
alias ..='cd ../' # Go back 1 directory level
alias ...='cd ../../' # Go back 2 directory levels
alias .3='cd ../../../' # Go back 3 directory levels
alias .4='cd ../../../../' # Go back 4 directory levels
alias .5='cd ../../../../../' # Go back 5 directory levels
alias .6='cd ../../../../../../' # Go back 6 directory levels
alias ~="cd ~" # ~: Go Home

# misc
# ------------------------------------------------------------
alias f='open -a Finder ./' # f: Opens current directory in MacOS Finder
alias c='clear' # c: Clear terminal display
alias ducks='du -cks *|sort -rn|head -11' # ducks: List top ten largest files/directories in current directory
alias which='type -all' # which: Find executables
alias path='echo -e ${PATH//:/\\n}' # path: Echo all executable Paths
alias showOptions='shopt' # showOptions: display bash options settings
alias fixStty='stty sane' # fixStty: Restore terminal settings when screwed up
alias cic='set completion-ignore-case On' # cic: Make tab-completion case-insensitive
mcd () { mkdir -p "$1" && cd "$1"; } # mcd: Makes new Dir and jumps inside
trash () { command mv "$@" ~/.Trash ; } # trash: Moves a file to the MacOS trash
ql () { qlmanage -p "$*" >& /dev/null; } # ql: Opens any file in MacOS Quicklook Preview
alias DT='tee ~/Desktop/terminalOut.txt' # DT: Pipe content to file on MacOS Desktop

# lr: Full Recursive Directory Listing
# ------------------------------------------
alias lr='ls -R | grep ":$" | sed -e '\''s/:$//'\'' -e '\''s/[^-][^\/]*\//--/g'\'' -e '\''s/^/ /'\'' -e '\''s/-/|/'\'' | less'

# mans: Search manpage given in argument '1' for term given in argument '2' (case insensitive)
# displays paginated result with colored search terms and two lines surrounding each hit.
# Example: mans mplayer codec
# --------------------------------------------------------------------
mans () {
man $1 | grep -iC2 --color=always $2 | less

# showa: to remind yourself of an alias (given some part of it)
# ------------------------------------------------------------
showa () { /usr/bin/grep --color=always -i -a1 $@ ~/Library/init/bash/aliases.bash | grep -v '^\s*$' | less -FSRXc ; }

# -------------------------------
# -------------------------------
zipf () { zip -r "$1".zip "$1" ; } # zipf: To create a ZIP archive of a folder
alias numFiles='echo $(ls -1 | wc -l)' # numFiles: Count of non-hidden files in current dir
alias make1mb='mkfile 1m ./1MB.dat' # make1mb: Creates a file of 1mb size (all zeros)
alias make5mb='mkfile 5m ./5MB.dat' # make5mb: Creates a file of 5mb size (all zeros)
alias make10mb='mkfile 10m ./10MB.dat' # make10mb: Creates a file of 10mb size (all zeros)

# cdf: 'Cd's to frontmost window of MacOS Finder
# ------------------------------------------------------
cdf () {
currFolderPath=$( /usr/bin/osascript <<EOT
tell application "Finder"
set currFolder to (folder of the front window as alias)
on error
set currFolder to (path to desktop folder as alias)
end try
POSIX path of currFolder
end tell
echo "cd to \"$currFolderPath\""
cd "$currFolderPath"

# extract: Extract most know archives with one command
# ---------------------------------------------------------
extract () {
if [ -f $1 ] ; then
case $1 in
*.tar.bz2) tar xjf $1 ;;
*.tar.gz) tar xzf $1 ;;
*.bz2) bunzip2 $1 ;;
*.rar) unrar e $1 ;;
*.gz) gunzip $1 ;;
*.tar) tar xf $1 ;;
*.tbz2) tar xjf $1 ;;
*.tgz) tar xzf $1 ;;
*.zip) unzip $1 ;;
*.Z) uncompress $1 ;;
*.7z) 7z x $1 ;;
*) echo "'$1' cannot be extracted via extract()" ;;
echo "'$1' is not a valid file"

# ---------------------------
# ---------------------------

alias qfind="find . -name " # qfind: Quickly search for file
ff () { /usr/bin/find . -name "$@" ; } # ff: Find file under the current directory
ffs () { /usr/bin/find . -name "$@"'*' ; } # ffs: Find file whose name starts with a given string
ffe () { /usr/bin/find . -name '*'"$@" ; } # ffe: Find file whose name ends with a given string
ft() { /usr/bin/find . -name "$2" -exec grep -il "$1" {} \;
} # ft: Find text in any file

# spotlight: Search for a file using MacOS Spotlight's metadata
# -----------------------------------------------------------
spotlight () { mdfind "kMDItemDisplayName == '$@'wc"; }

# ---------------------------
# ---------------------------

# findPid: find out the pid of a specified process
# -----------------------------------------------------
# Note that the command name can be specified via a regex
# E.g. findPid '/d$/' finds pids of all processes with names ending in 'd'
# Without the 'sudo' it will only find processes of the current user
# -----------------------------------------------------
findPid () { lsof -t -c "$@" ; }

# memHogsTop, memHogsPs: Find memory hogs
# -----------------------------------------------------
alias memHogsTop='top -l 1 -o rsize | head -20'
alias memHogsPs='ps wwaxm -o pid,stat,vsize,rss,time,command | head -10'

# cpuHogs: Find CPU hogs
# -----------------------------------------------------
alias cpuHogs='ps wwaxr -o pid,stat,%cpu,time,command | head -10'

# topForever: Continual 'top' listing (every 10 seconds)
# -----------------------------------------------------
alias topForever='top -l 9999999 -s 10 -o cpu'

# ttop: Recommended 'top' invocation to minimize resources
# ------------------------------------------------------------
# Taken from this macosxhints article
# ------------------------------------------------------------
alias ttop="top -R -F -s 10 -o rsize"

# myPs: List processes owned by my user:
# ------------------------------------------------------------
myPs() { ps $@ -u $USER -o pid,%cpu,%mem,start,time,bsdtime,command ; }

# tm: Search for a process
# ------------------------------------------------------------
alias tm="ps -ef | grep"

# ---------------------------
# ---------------------------

alias myIP='curl' # myIP: Public facing IP Address
alias netCons='lsof -i' # netCons: Show all open TCP/IP sockets
alias flushDNS='dscacheutil -flushcache' # flushDNS: Flush out the DNS Cache
alias lsock='sudo /usr/sbin/lsof -i -P' # lsock: Display open sockets
alias lsockU='sudo /usr/sbin/lsof -nP | grep UDP' # lsockU: Display only open UDP sockets
alias lsockT='sudo /usr/sbin/lsof -nP | grep TCP' # lsockT: Display only open TCP sockets
alias ipInfo0='ipconfig getpacket en0' # ipInfo0: Get info on connections for en0
alias ipInfo1='ipconfig getpacket en1' # ipInfo1: Get info on connections for en1
alias openPorts='sudo lsof -i | grep LISTEN' # openPorts: All listening connections
alias showBlocked='sudo ipfw list' # showBlocked: All ipfw rules inc/ blocked IPs

# ii: display useful host related information
# -------------------------------------------------------------------
ii() {
echo -e "\nYou are logged on ${RED}$HOST"
echo -e "\nAdditionnal information:$NC " ; uname -a
echo -e "\n${RED}Users logged on:$NC " ; w -h
echo -e "\n${RED}Current date :$NC " ; date
echo -e "\n${RED}Machine stats :$NC " ; uptime
echo -e "\n${RED}Current network location :$NC " ; scselect
echo -e "\n${RED}Public facing IP Address :$NC " ;myip
#echo -e "\n${RED}DNS Configuration:$NC " ; scutil --dns

# ---------------------------------------
# ---------------------------------------

# cleanupDS: Recursively delete .DS_Store files
# -------------------------------------------------------------------
alias cleanupDS="find . -type f -name '*.DS_Store' -ls -delete"

# finderShowHidden: Show hidden files in Finder
# finderHideHidden: Hide hidden files in Finder
# -------------------------------------------------------------------
alias finderShowHidden='defaults write ShowAllFiles TRUE'
alias finderHideHidden='defaults write ShowAllFiles FALSE'

# cleanupLS: Clean up LaunchServices to remove duplicates in the "Open With" menu
# -----------------------------------------------------------------------------------
alias cleanupLS="/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain local -domain system -domain user && killall Finder"

# screensaverDesktop: Run a screensaver on the Desktop
# -----------------------------------------------------------------------------------
alias screensaverDesktop='/System/Library/Frameworks/ScreenSaver.framework/Resources/ -background'

# freq: Which commands do you use the most
# -----------------------------------------
alias freq='cut -f1 -d" " ~/.bash_history | sort | uniq -c | sort -nr | head -n 30'

# CMDFU lookup
cmdfu(){ curl "$@/$(echo -n $@ | openssl base64)/plaintext"; }

# easily scp a file back to the host you are connecting from and place on Desktop
mecp () { scp "$@" ${SSH_CLIENT%% *}:Desktop/; }

# ---------------------------------------
# ---------------------------------------
alias aEdit='sudo edit /etc/apache2/httpd.conf' # aEdit: Edit httpd.conf
alias aLogs="less +F /var/log/apache2/error.log" # aLogs: Shows apache errorlogs
alias aRestart='sudo apachectl graceful' # aRestart: Restart Apache
alias aTail='tail -n 1000 -f /var/log/apache2/error.log /var/log/apache2/access.log' # aTail: Tails HTTP error logs

alias hEdit='sudo edit /etc/hosts' # hEdit: Edit /etc/hosts file
httpHeaders () { /usr/bin/curl -I -L $@ ; } # httpHeaders: Grabs headers from web page

# httpDebug: Download a web page and show info on what took time
# -------------------------------------------------------------------
httpDebug () { /usr/bin/curl $@ -o /dev/null -w "dns: %{time_namelookup} connect: %{time_connect} pretransfer: %{time_pretransfer} starttransfer: %{time_starttransfer} total: %{time_total}\n" ; }

# AWS EC2 Functions
# ---------------------------------------
# Instance id info: pull the 2nd field which is instance id from ec2 info
ec2id() { ec2-describe-instances -H --filter tag:Name="$1"|grep -i instance | awk '/INSTANCE/{print $2}'; }

#ec2elip() { ec2-describe-instances -H --filter tag:Name="$1"|grep -i instance | awk '/INSTANCE/{print $2}'|ec2-describe-addresses --filter=instance-id= -; }
# grabs instance id from lookup and passes it to find associated elastic ips

ec2elip() { local awsid=`ec2-describe-instances -H --filter tag:Name="$1"|grep -i instance | awk '/INSTANCE/{print $2}'`; echo $awsid; ec2-describe-addresses -F instance-id="$awsid"; }
ec2info() { ec2-describe-instances -H --filter tag:Name="$1"; };

# awk or cut -f 3 would work
ec2volinfo() { ec2-describe-instances -H --filter tag:Name="$1"|grep -i vol | awk '/BLOCKDEVICE/{print $3}' | ec2-describe-volumes -;}
ec2volsnap() { ec2-create-snapshot "$1" "$2"; } # use ec2volinfo first to get vol id

# IP Address info: pull the 4th field which is ip address from ec2 info
ec2ip() { ec2-describe-instances -H --filter tag:Name="$1" |grep -i instance | awk '/INSTANCE/{print $4}'; }

Startup Timing

David Fisher once wrote “What often characterizes visionaries is their lack of vision. It’s a popular idea that people of genius see farther and clearer than other people, but perhaps the truth is actually the opposite. Visionaries often don’t notice the enormous and obvious impediments to realizing their technological dreams – roadblocks apparent to more practical people”.

Have you ever read The AltaVista Search Revolution?  Is there any argument that AltaVista’s search engine was a spectacular idea ahead of the market that Google later seized?

I am asked all the time for “What’s the one simple thing that makes a successful startup”.   I’ve written and tweeted about about passion, laser focus, speed, agility, leadership but to be clear there is no formula for startup success; Statistically, startup success is a rare and exceptional event.  Entrepreneurs are rightfully focused first on innovative ideas, but the genius is seeing the market timing and having the discipline to be ready for it.

greed-fear-marketWhen entrepreneurs are in the early stages of vetting their ideas, they are typically very focused on solving a problem that people with money very much want solved; In other words, do you need to use this and how much will you pay for it?  Often the pitch deck’s market slides describes broad, existing markets.  But pattern recognition using broad historic inspection and patterns are nearly useless for predicting rare future events.  Often sufficient conditions are just emerging and predicting scale is nearly impossible.  We all know that technology will be mobile, location-based, social and fully personalized – lots of successful startups will be built along the way, but when?

Again, the genius is seeing the market timing *and* having the discipline to be ready for it.  This means, manage your resources accordingly.  The basics are emerging trends usually take longer than the enthusiastic entrepreneur expects, so keep that burn low, lengthen the runway, work hard/smart and be patient.   If you’re entering a market that is already trending, you should ramp aggressively.

Finally, sometimes random, unexpected and uncontrollable events make significant changes to market conditions.  Sometimes these events are favorable, some will say “lucky” and sometimes these are catastrophic – for example, the events of 911 crushed the financial markets and nailed the dot-com coffin closed.

More than once, I’ve found myself significantly out in front of the market, then the real challenges and tough decisions take place.

Is a Personal Media Network Possible?

Is it possible to build a personal media network that allows you to collect, organize and stream video, pictures and music to all your devices – TV, phone, computer, game console?

There seems to be a couple of key components:

  • A media server – organizes your content wherever you keep it, transcode content appropriately for various devices and manage streaming tasks.
  • Content collection – content exists in many place – your personal library, streaming services (Netflix, Amazon, Hulu, YouTube, Pandora, etc.), NZB newsfeeds, Torrent sites and sprinkled throughout many other content sites (video news stories, blogs, sports sites, etc.).  While some of these services provide built in interfaces for searching and accessing, it would be nice to make use of some type of internet PVR technology to manage your interest and have your media network automatically collect content for you so that you can make use of it when you are ready.
  • Download agents – in some cases, you’ll need to download content rather than stream the media directly from a source.  Of course, downloading also means storing the content some place – hard drives, NAS devices and cloud storage are all options with various capacity, technical features and costs.
  • Utility – I don’t think there is all-in-one solution, so there are a few surrounding utilities, scripts and sites that might make a personal media network more automated and easier to use.

Some other features that might be valuable.  It would be great if the media network could:

  • Organize the stored media files – folders, file naming, etc.
  • Collect meta data – art work, descriptions, plot summaries, etc.
  • Present one simple interface for searching, recently added, view by artist, genre, year, etc.
  • Share will family and friends, inside your home and externally
  • Make your media reachable even when you’re offline – syncing to mobile devices.
  • Transcode for your viewing device’s capabilities – screen size, format, etc.
  • Support saving it now, watching it later
  • Manage media that you are interested in – watchlist, TV series, wanted movies, etc.
  • Automatically acquire or access media when available, including managing your media preferences – high definition (1080/720p), AC3, 320K bitrates, etc – and notify you when updates are available.

It might be possible to install and configure some free, open source products such as Plex Media Server, CouchPotato, SickBeard, HeadPhones, Transmission and SABnzbd to build a personal media network.

Laser Focus

A few years ago, I asked an experienced, technically-savvy friend to look at the product that we were building and objectivefocusly let me know his thoughts.  He quickly pointed out that while he was initially impressed with the array of features and power, after about an hour of use his head exploded.  He went on to point out that one of the great things about Twitter was that it basically does one simple thing with broad, mass appeal; a very difficult accomplishment derived directly from a clear strategy and laser focus.  Alternatively, lack of strategy and poor discipline often leads to building the kitchen sink in hopes that you hit on the one simple thing.  Ouch, the truth hurts sometimes.

Another guy once said: “That’s been one of my mantras – focus and simplicity.  Simple can be harder than complex: You need to work hard to get your thinking clean to make it simple.  But it’s worth it in the end because once you get there, you can move mountains.” – Steve Jobs.

Trying to do everything at once probably means that nothing will be done well, it is contrary to the fail-fast approach, it is counterproductive to being a nimble startup and it is nearly impossible to craft a message that will make your offering stand out.  Can you think of a single super successful company – Walmart, Apple, Google – that launched with a broad focus?  It’s great to dream big, but your startup needs a laser focus in the beginning to get market and investor attention. 

Here are the common sense reasons why a laser focus is more likely to lead to startup business success:

Lead the market with innovation first.  Market leadership isn’t a one-time event, continuous innovation is critical and time consuming.  Your competitors will innovate and blow right past you if you fail to invest significant amounts of energy over the long term.  It is easy to believe that pushing more products and features is innovation but it will dilute your focus, attention and depth of your expertise which is a sure fire formula for losing your market leadership position.

Time to market is critical.  Broad strategies within a resource limited startup environment takes too much time to build the supporting processes and products.  Seems obvious, but lack of strategy, fear of picking the wrong path, unwillingness to start small and build often lead to lack of focus.  Meantime, small focused nimble competitors will appear, cross the chasm and seize your business opportunities while you are off whale hunting.

Time to market’s cousin, be nimble.  Pick your cliche – fail fast, pivot, dodge and weave.  Don’t build a business in a lab.  Get into the marketplace quickly with the minimally valuable, high quality product, learn and go fast from there.  If you bring a big, do everything product to the market you will be crushed by its weight and cost.

Quality does matter.  I did not say quickly bring a total piece of garbage to the market and expect customers to use it enough to care to let you know what you need to do next.  The more you try to do in parallel, the harder it is to maintain quality.  There is a very old adage – time, quality, cost – pick two.  I pick time and quality.  Without focus, it will be expensive.  Startups can not afford expensive – so stay focused to keep your cost under control.  The dot com days of raising tons of money and spend like a crack monkey are way over.

Personal bandwidth is limited.  When a business doesn’t have focus, it quickly become complex – the team loses track of priorities, becomes tired and tends to lose motivation followed by giving up.  Laser focus is easier to manage, communicate and is more likely to result in quality results quickly.

Finally, focus on exploiting strengths and achieving success rather than resolving weaknesses and avoiding problems.  Conversely, focusing on the wrong things will result in a downward business spiral.  When applied appropriately, focus will result in rewards exceeding your expectations.  Most customers are only looking for one thing at a time.  After you saturate the market with your focused offering, then you will have the time and resources to broaden your offering.  Don’t give up your grand vision, since no investor will buy a one trick pony, but also don’t try to be everything for everyone on day one.

The Cryptocurrency Genie is Out of the Bottle

BitcoinI’m typically very interested in innovation, especially the disruptors.  A couple years ago in a bout of attention disorder, I ended up stumbling across Bitcoin while researching anonymous proxies and VPN.  At one point, I toyed with the idea of buying some ASIC machines and setting up a mining operation in Iceland where the electricity is inexpensive – just kidding.  In 2013, the Bitcoin exchanges documented crazy volatility and critics pointed out that Bitcoin is used for nefarious purposes – Silk Road.  I could easily document far greater use of cash for nefarious purposes and some government controlled, crazy volatile markets that are the origins of the word “Bubble”.

I believe Bitcoin, the first currency that removes the need for a trusted third party in financial transactions is an important disruptor and even if it fails, there will be successors; I have a difficult time seeing a future without Bitcoin or something like it (Ripple).  Analogous to Napster’s push on P2P file sharing, Bitcoin is blazing a trail for how currency can be separated from certain external factors; distributed, borderless, digital currency that can be micro managed and is inherently deflationary.  With no central source to control, regulation or manipulation, Bitcoin is free from political consideration that influences fiat money.  Bitcoin users get to send money anywhere in the world for a minimum fee, cutting out the middleman.  While I doubt it will entirely replace fiat currency, a viable alternative may force changes in the way other currencies operate.

What’s Big about Big Data – a C-Suite Challenge.

My twitter profile currently includes that I am “passionate about #BigData”.  Recently, someone spammed me with a direct message “What is the big whoop about #BigData”?  Typically, I dismiss DM spam but my brain kept whispering “This is a c-suite challenge”.

Today, cutting edge corporate superpowers are heavily investing top-management muscle into profoundly altering their business landscape by using the power of data MinerHelmetanalytics on the frontline.  They use big data to create new lines of business, improve existing revenue, cut costs and boost productivity by making significant changes in mind-set.  

Bootstrapping…  More than a few years ago, with the commoditization of big data technologies like cloud infrastructure, Hadoop, MapReduce, etc. plenty of forward leaning folks started buzzing about the technical horizon of collecting, storing and managing extremely large sets of data.  Aggregating multiple data sources, annotating data sets and deriving new views make the technical challenges even more exciting and interesting.  Lots of available data and tons of tools, cutting edge stuff is always exciting for the technical community.  Companies are emerging, smart folks are gathering and producing tools, products and platforms.

No kidding Chip, obviously you need more than a mountain of data – you need to mine some trends.  Who are my best customers?  What do they like to buy most?  How often do they visit my business?

This post is about more than turning on the miner’s helmet flashlight and breaking out the pickaxe.  There are plenty of posts, studies and surveys that are pointing out that many doomed companies are leaving these transformative matters to the IT department, building infrastructure and mining.  Designing an advanced analytic strategy and culture to earn a leadership position is more challenging than collecting data and mining.  Transformative change to business operations all the way to the front line requires authority during conflict, tough trade off decisions, company wide knowledge and consistent leadership commitments.  The c-suite challenges of “Big Data” include:

Perspective pivot: Typical of all new things, the entire senior team must acquire knowledge about what is possible with data analytics and embrace that data is core to the business so that significant improvements can be delivered from analytics.  Without this pivot in perspective by leadership, it isn’t reasonable to expect the organization to adopt a consistent change in behavior.

Data analytics strategy: Um, no strategy equals random results – so maybe you hit the lottery or maybe you dump large sums of money in the toilet.  And while even a blind squirrel sometimes finds a nut, if you want to improve your chances of your big data investment then define a data analytics strategy.  Seems obvious, but you might be surprised by the lack of course setting.  Again, not easy to do with new stuff – you can’t copy your friend’s homework.  You’ll need the usual strategy inputs – priorities, expected results, leadership, decision making, etc…

Acquire expertise: The rapidly changing data analytics landscape currently includes cloud infrastructure and open products supporting large data sets that combine internal and external sources in easy to use forms.  The c-suite needs to prioritize creatively hiring and retaining talented statisticians who can create and optimize predictive models.

Build or Buy:  Collecting large sets of data, building advanced analytic models and rolling out change to the front line can be a very large resource commitment.  Typical to many build vs. buy decisions, do you expect results or have proprietary intellectual property that justify the time and cost of development and ownership or is your strategy better served by partnering with external vendors that can quickly build out infrastructure, data, tools and expertise.  This is the single challenge that most companies start with and end with when investing in big data.

Rapid Iterative Mobilization: To lower the failure rate, a company’s leadership must commit to breaking down the usual barriers to change and mobilize cross functional support to help front line manager in making use of advance analytic models.  It is super important to involve the frontline team while identifying the opportunity to impact performance with data analytics and to quickly iterate the entire workflow of collecting data, modelling, mining and rollout to the frontline.  The senior team must be held responsible for sustained frontline change by prioritizing and measuring that the sophisticated analytics solutions that statisticians and technologist devise are embedded in frontline tools in simple and engaging ways so that frontline employees will be eager to use them daily.

In the end, I’m thinking that I should amend my profile to say that I am passionate about data analytic strategy and culture in a big data world.  Why?  Because admittedly, big data is just the trendy tip of the iceberg, the buzzword, a misnomer and doesn’t really capture the real story.