In this weeks edition of The Economist there is a fourteen page special report entitled “The Data Deluge.”  It’s an illuminating read about the ways that data mining can be used to predict everything from disease before the onset of common symptoms, to tracking what you buy so that a computer can suggest other purchases you might like. It also discusses the problems of storing the overwhelming amount information being generated and how that problem will only worsen with the passage of time.   Finally, the article also speaks volumes about how most of the data currently available for use is not reliable; why many of the largest collectors of online data (like Google, Amazon, and Microsoft) remain silent in regards to how they actually use and protect it; and the future of both governments and society when the amount of information available to the average person is literally on a scale so vast that it can only be compared to all the previous centuries of available data in terms of what the ocean is to a drop of water.

In reading the article there was one old adage that just kept popping into my head: “One man’s garbage is another man’s treasure.”  Two quick examples from the article will suffice to illustrate: first, up is Walmart.  When the company applied its algorithms to figure out what it sold in stores just prior to a hurricane, a surprising item that made the list was Pop Tarts. It’s completely logical, as they are fairly tasty (albeit in a cardboardesque kind of way), can be eaten without any cooking, and are relatively cheap.  The trash in this case, was just the years of records they kept and sifted though.

What sifting though all that data really feels like!

Second, and I found this one way more interesting, is that Google accessed European Commission documents that had already been translated into twenty languages, as well as a myriad of other books that had already been translated so that they could break the translation down into a problem of the probability of one word matching another in a different language; English acts as a bridge if there is no direct correlation.  This overcame a difficulty that had thwarted (I love that word….TWAR-T-ED!) some of the best and brightest minds at companies like IBM, who had been trying to program translators to incorporate the grammar of languages, along with all the exceptions, into a program and then move on to the words. This latter method never worked, but Google’s method yielded results within two years of getting the programing up and running.

This concept of using old data to find new correlations relates to too many applications to be discussed by the likes of me, but what the hell!  The beauty of being human is that I can do things the computers say I should not do! The irony of course is that they could probably predict it if they knew my reading habits. One more reason not to read the Economist on-line and stick to hard copies! But that is both my point, and may be the greatest acknowledged escape of the “Big Data Age.”

As time goes by, more and more companies will begin funneling and channeling information in both useful and harmful ways, and it will be ever more difficult for individuals to escape the fact that they can be profiled by a computer in ways that are both wickedly accurate, but at the same time leave out that all important human X factor. This factor, I think, involves a good bit of disregard for logic, intuition, a dash of vim and vigor, the ability of the human brain to weigh ethical and moral gray areas in ways not quantifiable or measurable, and those gut feelings that could either be God or indigestion speaking to you.

I’m appalled and almost indecently turned on by the fact given a large enough data set, the proper metrics and analytics, and time, data can be mined to yield hidden connections in our world.  But I defiantly get hot and bothered by the fact that as the computers that cooked up all those crazy financial deals prove, you can have all the data in the world and it can point you to all the hidden connections you care to find, but at the end of the day, there still has to be a human to see what’s coming down the tracks.  Now, if we could just find some humans that were good at that!