If you talk enough, eventually you'll say something smart, or at least interesting. Jason Calacanis on what media companies should do to Google:
The idea is that publishers could use their robots.txt as a ransom note, selling it to the highest bidder -- Bing or Google. (I suspect this idea takes fire and gets repeated a lot over the next couple months.)
Google composes, what, 70% of search referrals to news sites? More? For a business that relies on advertising and social distribution, it sounds like suicide to me, the same effect as putting all your content behind a registration wall.
posted by Andy Baio at 7:19 PM on November 9, 2009 
Disagree!
It's probably 40%, and the online revenue is probably 5% of overall revenue. (In other words, for all the heckling that Murdoch has gotten for his blocking Google quote yesterday -- including me! -- it really doesn't matter what the hell they do from a revenue perspective... at least in the short term.)
But the numbers aren't what matters. What matters is that no one has every really thought of their search traffic in these terms. We've always just put the search engine in the position of power, and no one has has ever positioned robots.txt as an asset.
posted by Rex at 7:41 PM on November 9, 2009
robots.txt is not an impermeable legal wall. It's just a webmaster's wish of how he would like to be treated.
Thus far, the good engines have respected it, but if you're being a dick, they can ignore your suggestions and crawl right through.
posted by Andy at 9:36 PM on November 9, 2009
Oh cmon, you know it would be a PR catastrphe if Google ignored it. That sounds like a Republican slippery slope argument!
posted by Rex at 9:44 PM on November 9, 2009
No, the slippery slope argument probably starts with Bing stealing the top 10 publishers by doing face-to-face agreements. Then, it works well enough that they open a self-serve model: "All publishers who block Googlebot, register here and we'll send you a revshare check in the mail." Then Google does the same thing. Then Yahoo does the same thing, but it doesn't work because everybody blocks MSNbot, which powers Yahoo's results. Yahoo dies. Then there are two left, Google and Bing. Each have half the world's content and search is now retarded. I build an aggregator engine that puts the results of the two parties back together again, and I steal all their marketshare before anyone can block me.
This should happen.
posted by Andy at 10:03 PM on November 9, 2009
I still don't get how this would cut into google revenues/market share all that much. I get to the WS Journal/CNN/NY Times more often than not through blog links. Google would still index these right? Wouldn't this essentially just funnel more of their revenues into the hands of rebloggers/aggregators.
Am I missing something?
posted by r l at 10:36 PM on November 9, 2009
To put it another way:
Isn't this akin to covering your eyes and shouting "I'm Invisible!"
posted by r l at 10:40 PM on November 9, 2009
Disallowing robots from crawling is not the same as restricting indexing. When Calacanis talks about blocking via robots.txt, he probably means using a meta tag like this on all pages:
posted by Andy at 10:50 PM on November 9, 2009
Anyway:
meta name="googlebot" content="noindex"
posted by Andy at 10:52 PM on November 9, 2009
years ago "they" set out to kill gawker by posting erroneous celeb sightings.
just saying.
posted by steve at 10:22 AM on November 10, 2009
hmm... if use wordpress engine, we can setting it...
posted by Lowongan Kerja at 4:36 PM on November 11, 2009
Calacanis, at 1:43 "I don't think the publishers are aggressive and smart and gangster like that. Rupert Murdoch probably is..."
This is EXACTLY what Rupert Murdoch said in his interview with David Speers on Sky News. And I quote: "We'd rather have fewer people coming to our sites, but paying."
What a concept- paying for content on the Internet! Murdoch is about to throw a major spanner in Google's Less Than Free business model.
posted by buck at 5:15 PM on November 11, 2009
bing could steal NYT and washington post... and still have no market share. this would just translate into more page traffic for cnn, intl. tribune. etc.
posted by andy at 5:33 PM on November 11, 2009
They can't do it as a group due to anti-trust laws. Bing would have to approach each inevitably.
posted by John Hall at 7:27 PM on November 11, 2009
I feel like there is Nash equilibrium stopping this from happening. i.e. if wash post, la times, chicago tribune, all take the microsoft bribe, nyt would significantly increase its traffic by not opting out, they would anticipate that and ask for a much higher bribe than the value of the current revenue stream.
posted by John at 9:40 PM on November 11, 2009
Again, I need to repeat this. As much as we like to think the new currency is "pageviews," this is still a meaningless concept without a monetization strategy.
A billion pageviews or a million dollars -- you pick.
posted by Rex at 10:01 PM on November 11, 2009
Rex is right ... the scenario might play out exactly like Andy paints it, but the probability of that isn't enough amo to keep the media from jumping on the randsom/opportunity, if presented. The only downside'd be that the Cory Doctorows of the world would H8 you forever.
posted by alesh at 11:52 PM on November 11, 2009
blocking google does nothing without hiding everything behind a paywall. If a bing referral was the golden key and opened up the paywall then yeah it could definitely steal market share.
I subsribe to the notion paywall or not, information is liquid and will find the smallest of cracks to flow through.
posted by ryanl at 5:58 PM on November 13, 2009
think its more than 70%. This is EXACTLY what Rupert Murdoch said in his interview with David Speers
posted by bukmacherzy at 7:08 AM on November 15, 2009
I'm with ryanl on this - I don't really see how this could ever be foolproof.
posted by John at 5:21 PM on November 16, 2009
Why not, blackmail was always good business. http://lifehacker.com/5409467/youtube-will-soon-block-access-from-set+top-devices
posted by T-Zombix at 6:19 PM on November 20, 2009
NOTE: The commenting window has expired for this post.
