Data is the new black

People are interesting.

Every once in a while it seems like you go from something being rare and no one is doing it to everyone is doing it all over the place.  Today that kind of dynamic, explosion of interest and activity is happening around “analytics” and “Big Data.”  This is actually a great sign for Microsoft.  Given that tech companies thrives on analysis and data, it has been rather surprising that we haven’t had more focus and activity on the science of data as an industry.  Today, everywhere you turn someone is working on a big data project or implementing analytics. 

Reminds me of the Yogi Berra quote about Ruggeri’s restaurant in St. Louis – “Nobody goes there anymore because it’s too crowded.”

One thing that strikes me quickly is how easy it is to get started – send some data in to a REST api and you have started (btw, send it in loosely structured or unstructured for bonus points!)  Pretty soon your table storage is overflowing… you might try to load some of that “analytics” in to a DB and run some queries…. You are pretty much doing BI now.  Well almost. 

The second thing that strikes me is that then it starts to get a lot harder.  Having data is not the same has being able to do data mining.  Having a pretty chart or a report is the not the same thing as doing analytics.  Running a query does not necessarily provide deep insight.  Unfortunately, insight and analysis and utility require work, more thinking. Turns out thinking is some of the hardest work there is to do.

Don’t fret.  It is a great sign that we are starting this journey.  Our DNA is evolving.  Some groups — Facebook, Google, Bing, Twitter, Netfilx — are creating a beach head.  So companies like SAS have been preaching this for a long time. My only ask of us is that we keep going… don’t let these efforts stall. We are building muscle and capability and experience a step at a time.  It is hard work and will be worth it.  There is a tremendous value computer science and engineering can create by getting good at this.

We are at a cross roads. Data is seeping in everywhere.  We can double down and get great at it.  Invoking Mr. Berra again…

“When you come to a fork in the road, take it.”