jun 4


"When information moves at electric speed, the world of trends and rumors becomes the 'real' world." -Marshall McLuhan

Data, data, data, and more data.

We inhabit a moment in time that obsesses about data -- too much data, yet never enough data; fast data, yet never fast enough; smart data, yet never smart enough. But talking about data today is like talking about air -- it pervades our lives in such ways as to render us almost blind to its existence. Like a Borgesian map where one mile equals one mile, data is suspended over us, with little stick pins that demarcate the boundaries and info packets of our lives. If we were to get especially sauced in metaphysical hyperbole, perhaps we'd even say existence itself has become a synonym with data -- the ways we quantify and visualize our lives expose our true self.

I would like to talk about two broad side-effects of the surfeit of data:

1) Manipulation. More data equates more exploitation of data. The era of data manipulation has propelled everything from hack culture (Make magazine, Lifehacker, etc.) to the advance of APIs (Facebook, Digg , Flickr, etc.). For my purposes, a more interesting outcome of data manipulation has been the advance of game culture. "Gaming the system" has become the meta-narrative of our time. (I will expand on this idea later in a separate essay on this topic for publication.)

2) Prediction. With more data creation, we can build more hypotheses on the future. Predictions are what I briefly want to talk about here today.

The Rise of Prophetic Clients

What are we talking about when we talk about data? Data is effectively the outcome of measuring something. When we hear about the increase of fluorocarbons in the atmosphere, we're really talking about a form of data measurement. When we talk about "the long tail," we're visualizing measurable data in a new way. If data is effectively the measurement of something, then looking at data over time is trend analysis. And when you analyze trends, you make predictions. More data, more predictions.

Finally, we're coming to my thesis point: the most exciting advances happening online right now are in the area of trends and predictions.

Trend analysis is the direct outcome of the explosion of data. Just look at Farecast, a site that accumulates information on previous flights and predicts when purchasing a plane ticket will be cheapest. Similarly, Inrix is a forecasting engine for traffic data. It is able to portend gridlock because previous traffic trends are now more measurable than ever before.

More examples? In news, there's Trendio; in sports, there's Pickspal; in technology there's Yahoo' Tech Buzz Game; in culture, there's Buzzfeed; in celebrity, there's the various forms of the Celebrity Death Pool.

In most cases, some wisdom of the crowds is aiding this new class of prophetic client. When an information shortage occurs, these applications ask people to supply more data. Even industries that are predictive by nature have been turning to the masses to generate more data. Metaweather is a site that evaluates its own predictions based upon previous success, as gauged by the public. A bevy of stock prediction sites -- Social Picks and Motley Fool CAPS among them -- aggregate user stock picks to prognosticate the market. Even a site as intrinsically data-centric as Digg has its own prediction layer in the form of Digg Trends. Meanwhile, Google tries to capture all online activity with Google Trends.

If you think about it, the new breed of recommendation engines are also attempts at playing soothsayer. Last.FM and iLike aggregate music from your friends (more data!) and build a trend analysis which suggests that you will probably like the new Mims single. Your TiVo tries to augur that you'll enjoy Desperate Housewives based up on data aggregated from your prior viewing.

Predicting the News

This is the point where I back up some of this thinking with a new product launch. Last week, I officially launched a news prediction application on MSNBC.com called iPredict.

msnbc ipredict


iPredict allows you to make predictions around news events. Although we tend to think of news stories as ways to inform us of the past, I contend that every news story has an interrogative kernel hidden inside -- a question about the future. Twelve people killed in Iraq? That story makes me wonder about the future of the war. LeBron James scores 25 straight points? That story makes me wonder if he really will be the next Jordan. Although news is immediately a historical account, it is also implicitly a gamble on the future, a suggestion of where things will be.

iPredict attempts to capture those implicit questions in every news story. The application visualizes the public's votes against your vote, storing your previous predictions with a cookie. When you return to the prediction later, you can change your vote. You see how your vote has changed over time, graphed out in comparison to everyone else's prediction.

Some example predictions you can currently vote on: Roger Federer Will Not Win The French Open, Gas Prices Will Hit $4 A Gallon By Labor Day, and Harry Potter Will Die In The Final Book. Each of them, with their answers visualized over time, is itself a unique story. The prediction, the question, the trend -- these are the new narratives.

I predict the future is in predictions.

Rex Sorgatz is the Executive Producer at MSNBC.com, where his colleagues tease him about quoting McLuhan like a smart-ass. He makes outrageous predictions at a regular basis here at Fimoculous.com


Good to see the return of the requisite McLuhan quote

posted by Jim Ray at 1:24 AM on June 4, 2007

Ha, this uses the same method I'm trying to apply editorially in one of my shows-in-the-making.

posted by Nick Douglas at 1:34 AM on June 4, 2007

Very cool idea, Rex.

posted by Mathew Ingram at 10:28 AM on June 5, 2007

"Although news is immediately a historical account, it is also implicitly a gamble on the future, a suggestion of where things will be."

I see this vote more as a poll of what people think would happen, rather than a suggestion of where things will be. It might not be a magic ball, but it gives us a general idea of what people feared/wanted at a given time.

posted by CH at 5:29 PM on June 7, 2007

What does it mean that almost everyone of these is near the middle right now?

Great work Rex. This is very nice.

posted by Gavin at 9:45 PM on June 7, 2007

NOTE: The commenting window has expired for this post.