Report: Microsoft may help News Corp. delist sites

Maybe Rupert Murdoch was serious about wanting to go without Google.

Murdoch's News Corp. has initiated discussions with Microsoft over a plan to have the media company's Web content essentially delisted from the world's largest search engine, according to a report Sunday in the Financial Times that cited a person familiar with the situation. Microsoft, which owns rival search engine Bing, has also reportedly approached other media giants about having their content removed from Google search results as well.

Microsoft representatives did not immediately respond to a request for comment.

The two companies have been linked discussing a Web-search partnership in the past. During Microsoft's failed bid for Yahoo in 2008, the tech giant was reportedly in "serious" talks with News Corp. to make a joint bid for Yahoo.

Murdoch, the chairman of a newspaper, TV, and Internet empire that includes The Wall Street Journal, The New York Post, 20th Century Fox, Fox News, and Hulu, warned earlier this month that his sites may soon disappear from the search engine's listings. Murdoch accused search giants of "stealing" his company's content during a recent interview with Sky News Australia. When he was asked why he just doesn't pull his Web sites from Google's search results, he said: "I think we will. But that's when we start charging."

Murdoch and other News Corp. execs have said that they intend to charge readers and viewers for access to the company's content, forsaking the ad revenue model.

For several months, executives at some of the nation's most influential news sources, including The Wall Street Journal and the Associated Press, have been blaming Google and similar Web services for at least some of their deepening financial troubles.

Google sells ads tied to the news blurbs it "scrapes" from news sites. It links back to the Web sites from which it acquired the content but doesn't share ad revenue with them.

"Publishers put their content on the Web because they want it to be found," Google said in a statement earlier this month. "Very few choose not to include their material in Google News and Web search. But if they tell us not to include it, we don't."

Critics of the media companies' bashing Google point out that if media companies were serious about not being indexed by search engines, they could accomplish the feat on their own by adding a robots.txt file to the root of their Web site containing a simple code that would prevent bots from indexing their pages.

[LandMineHare]

So, Microsoft is bribing Fox to not have their garbage show up on Google search results?<br /><br />I didn't expect MS to want Google to have more quality results and less BS...

[FutureGuy]

This is not any worse than what Google did to derail the initial deal between Yahoo and MS. They bribed and frauded Yahoo and after the deal was off they moved to the sidelines and watched Yahoo and now its former CEO take a beating. Don't let the shine from the "Do no evil" badge blind you.

[eadeguzman]

How could you interpret "Murdoch's News Corp. has initiated discussions with Microsoft" to be Microsoft bribing News Corp?

[TotallyMadeUpName]

"Microsoft, which owns rival search engine Bing, has also reportedly approached other media giants about having their content removed from Google search results as well."<br /><br />Yeah. I'm a company trying to make a living selling to as many customers as possible and I'm going to delist myself from the search engine that 85% of the people use?

[Random_Walk]

If Murdoch was serious about getting off of Google, it's as simple as editing those little files called robots.txt on his websites... somehow, I'm not seeing him (or his company) as being all that technically literate. :/<br /><br />The rest is just begging for a DOJ investigation.

[cosuna]

Two points:<br /><br />1. I believe that not enough technical detail was lost on this piece. You cannot simply "delist" from Google, since you don't own or control Googles' database. The only way you can avoid searchability by Google is to use the robots.txt file. My take is the following. The report found out that Microsoft is pushing an alternative robots file or some other standard. They will call it something like SearchOptimizer.xml with a proprietary standard. This combined with a closed or almost closed Robots.txt will in practical terms "delist" News and other companies using the standard. Also, this "standard" will help Microsoft with Search as it will in some way "target" or as they will call it "optimize" the search results, that is custom fit the search to the needs and intentions of the site owner. My premonition: Just as what happened with SenderID, Passport, HTML+Time and OOXML, in the end, most Microsoft sponsored "standards" tend to fail mostly by the fact the incumbent tends to ignore it.<br /><br />2. @eadeguzman: I think the reporter misread, or interpreted the above as an intent to block Google, which in practice it is. As elegant as you can put it, in the end this is a type of bribery from one company to another. This is not to say it is illegal. Companies hide true intentions inside PR and PC jargon, but in the end the result is the same.

[eadeguzman]

As mbenedict below mentioned robots.txt might not help for those News Corp content that are syndicated to other websites. <br /><br />I still don't see how it's bribery. What does Microsoft have to bribe Fox with? I'm more likely a negotiating ploy by Fox and possibly other media companies to force Google to share in the profits Google is making of its content.

[Random_Walk]

"As mbenedict below mentioned robots.txt might not help for those News Corp content that are syndicated to other websites."<br /><br />he was also incorrect - you can rig robots.txt as a whitelist, a blacklist, or both. News Corp. can rig their contracts to enforce robots.txt usage (if the language isn't already there to allow said enforcement).<br /><br />Long story short, this is a play by Murdoch to squeeze some cash out of Google, and apparently it isn't working to well for them.

[meh100]

Wait, so News. Corp, a global media conglomerate, needs Microsoft's help to make a robots.txt fle? I guess Murdoch is too cheap to hire web developers. And does Microsoft think that they'll remove the search results from Google but not Bing?

[Lerianis3]

Exactly right..... if Murdoch cannot make a robots.txt at the upper level of his site..... he deserves to have his stuff still be listed. Heck, I can make one of them in 10 seconds and post it on my private website.

[mbenedict]

If you guys think this is about robots.txt, then I have a bridge to sell you guys.

[Lerianis3]

Yes, it is about robots.txt. I know that Murdoch is also just trying to get publicity, but the online newspaper should HAMMER IN AT EVERY TURN that it is very easy to delist your sites from Google, and make this man look like the fool that he is.

[SteveW928]

@ meh100 -<br /><br />"And does Microsoft think that they'll remove the search results from Google but not Bing?"<br /><br />I think that is the point. M$ can't compete with Google on a technological basis... so they will start cutting deals with content to be exclusively found on Bing. If they succeeded with enough big players in Internet content, it would kind of force users to use Bing, or at least make it enough of an annoyance of having to use both to ensure you find what you are looking for, that people will just move to Bing.<br /><br />It's a pretty typical M$ move really. They've never really won any business by having something better than the competition.<br /><br />As for Murdoch.... these big media people are simply lost and doing anything to shift blame and grab headlines. The average person off the street could do a better job because they are more in touch with reality. It will be hard for these companies to recover now. They spent the last 10-15 years denying the Internet, and are now just scrambling to get on board in some way, but they don't really understand it.

[mbenedict]

News Corp stories are syndicated to thousands of 3rd-party news sites around the world, such as those belonging to local &#38; international newspapers, etc. I mean, virtually every major newspaper in the world carry news feed from Dow Jones. <br /><br />E.g., if you go to DenverPost.com or LeMonde.fr or SeattlePI.com or wabcradio.com (New York) you'll likely find feed originally coming from one of the gazillion News Corp source. But none of these sites are owned or controlled by News Corp. So how will your robots.txt file block Google from indexing those feeds? <br /><br />But again, this has NOTHING TO DO with technology.<br /><br />Many of the comments here reflect why some people are low-end techies, while Murdoch counts his billions. People just don't get the bigger picture.

[solitare_pax]

The news media like to think that if they are the 'first' with a major story, then they are the 'best' and therefore they should get the advertising revenue as if it were a prize.<br /><br />It is currently within Murdoch's control to prevent any search engine from viewing his websites, but that would be as useful as pulling his publications off the newstands.<br /><br />What Murdoch is looking for is a source of revenue from these search engines for the honor of displaying his wares. Since Google's argument is that the media wants to be seen, he won't get any money from them. And Microsoft's Bing is only looking for a reason to exist, so it is willing to pay for the content.<br /><br />The question is, will it work - or will it fall flat on its face? Either way, I can see Google deciding to remove any advertising they provide from Murdoch's sites if he signs a deal with Microsoft's Bing.

[aMUSICsite]

Wouldn't it be better to work with Google, after all they did say if News Corp wanted to take down all their sites Google would help them. <br /><br />Is MS going to help them unlist from Bing to or is Bing still going to carry links?

[Super2online]

You guys might consider that News Corp may be looking for something a little more sophisticated than a plain vanilla robots.txt file. Something that would prevent Google but not others, or something that would prevent certain articles but not others. There could be a whole host of ideas going on here behind the scene that you don't know about.

[klor5]

@mbenedict<br />I agree 100%

[Random_Walk]

"So how will your robots.txt file block Google from indexing those feeds? <br /><br />The syndications come with contracts. It doesn't take much to stipulate that the contract terms include protecting the content from, oh I don't know, "unauthorized replication" (you know, basic stuff, like requiring member sites to edit their own respective robots.txt files as a condition).<br /><br />Do it with every new syndication renewal (or enforce it now since the contracts likely carry a close enough stipulation to enforce it, FFS), and problem solved. 'course, the implementation is a bit more complex, but certainly more simple than colluding with your target's business rival in a way that puts said rival at a legal risk.<br /><br />But of course, it's funny that it took a "low end techie" to come up with such a basic, quiet, yet damned effective solution, but a billionaire business gent like Murdoch has to go flirting with antitrust laws to get the job done. Then there's you, feverishly defending such dumb business practices (mostly because your idol is involved... or are you a fan of Fox News, perhaps?) ;)<br /><br />==<br /><br />"Something that would prevent Google but not others, or something that would prevent certain articles but not others"<br /><br />The file can be selective as to which bots it allows and which ones it does not. <br />This isn't rocket science; it's scripting 101.

[eltoro2827]

Google can't be trusted. Down with google!!!!!!

[Turtle9598]

No down with Microsoft do you know how many years they held the web back with IE6 NOT AGAIN!!! Now amagin that with Google in there grasp my it makes me sick!!!

[eltoro2827]

no, down with google X2... thats a pathetic company that lies when it comes to their cheap products. <br /> <br />free comes at a price. im not willing to give my identity to google.

[winstein]

FUD! People who don't read instructions will always complain the wrong things.

[FutureGuy]

@Turtle9598, you may have been peacefully sleeping in your cave but IE6 is now almost almost 5 years old, the latest version is 8 with 9 on its way.

[Perry_Clease]

"@Turtle9598, you may have been peacefully sleeping in your cave but IE6 is now almost almost 5 years old, the latest version is 8 with 9 on its way."<br /><br />It is still being used a lot and because of that a web designer must, well should, accommodate it.

[rcrusoe]

GOOGLE can't be trusted? Have you ever read your Microsoft Windows EULA? IMO, they basically own your computer and everything on it.<br /><br />As I read it, you give them the right to add software to your computer, change software on your computer, disable software on your computer, remove files, share data from your computer with third parties, etc.<br /><br />At least Google doesn't have anything of mine I don't specifically give to them.

[loose_screw]

No, idiotic anonymous comments on the internet can't be trusted.

[badasscat]

If News Corp. were really "serious" about removing their content from Google, they'd DO IT. It's not rocket science; a 5 year old can do it in 2 minutes. Here:<br /><br />User-agent: Google<br />Disallow: /<br /><br />Save as --&gt; robots.txt<br /><br />Done. Can I now have the $500 million you promised to MS for this service, Mr. Murdoch?<br /><br />Clearly, this is just smoke-blowing to try to blackmail Google, which is just as clearly not going to work.

[FutureGuy]

@badasscat, you are such a genius. <br /> <br />On a side note, I have 100 million stuck in a bank, can you help? We can share it 50 - 50.

[Super2online]

A simple solution for a simple mind. Do you seriously think thats all they want?

[Random_Walk]

"Do you seriously think thats all they want?"<br /><br />Murdoch wants money - hold the news content hostage, and Google will talk. <br /><br />Geez - it's not like there's only one goal going on...

[Noneyabeeswax]

This is a publicity ploy. What they're really after is for Google to say, "Hey wait a minute we'll share revenues with you. Your trash uis worth it. Out of all my searches News Corp sites make up less than 1% od my clicks. If they take it offline, how the hell do they think they're going to sell subscriptions? People won't be able to find them.<br /><br />As for MS, they're just loking for a way to loosen Google's grip on the market place, especially with a cloud based OS on the horizon. Yahoo has already slipped behind the pack.

[Hooligan1971]

no one will pay for it. It?s a good thing that you?ll also delist your sites off of Google too. I don?t want anyone to waste their time with your stupid advertisements either. It?ll be funny though to see you go crawling back to Google after Bing leads you to nowhere. Bing sucks and no one uses it.

[Turtle9598]

Agreed!!!!*Infinity

[sph001cnet]

Of course Murdock could put the robot.txt in place.. but there is not much publicity in that. he wants to monetize his content... and Bing is a willing suitor... <br /><br />this sets a bad precedence... now we will have fragmented search engine content? no longer will any one search engine contain a majority of content? this isn't a good road to go down... <br />[CNET editor's note: Prohibited spam deleted.]

[ThinkBeforeYouPost]

"this sets a bad precedence... now we will have fragmented search engine content? no longer will any one search engine contain a majority of content? this isn't a good road to go down... "<br /><br />It isn't a good road to go down for consumers and users of search engines. MS, on the other hand, has everything to gain from a fragmented search engine content, not just because it would help Bing, but mostly because it would weaken Google at a time when Google is threatening Exchange, Office, and ultimately Windows.

[Super2online]

@ThinkBeforeYouPost - The threat you speak of is only in the minds of people who read articles telling them there is a threat and they blindly jump in line. So far that supposed threat has amounted to very little.

[littleM]

If Murdock wants to charge for Fox News, he needs to adopt a programming format like this -- <br />http://www.nakednews.com/ .

[sph001cnet]

Murdock paying MS... I would think it would be the other way around. That MS would pay Murdock for the right to KEEP his content on BING... unless of course, Google pays 600M. sounds more like content hi-jacking. <br />[CNET editor's note: Prohibited spam deleted.]

[ertem0]

Woo hoo!!! This is great news. I've been asking Google to remove any FoxNews links from my iGoogle homepage for months, but they have still not implemented a -site: filter. Can't wait for this to happen... Good riddance mate.

[SwissJay]

Wow, I prefer my news from non-partisan sources any way. It's a lot more objective that way. Way to go Microsoft, I'm all for it, LOL :)

[Sausagebiscuit]

I find stories like these funny: http://techdirt.com/articles/20091120/0223387019.shtml<br /><br />Double standards for the win. Murdoch is just another person who doesn't "get it".

[dwinks]

Wow, this Murdoch guy is a complete idiot. Go ahead and remove all of your content from Hulu and the like. Instead of getting some money (from me viewing ads) you will get nothing and I will just get the content elsewhere, such as torrents.<br /><br />Heck, take away Hulu and I likely won't find as many shows to watch, and will also cease to bother watching them on TV too.<br /><br />Way to go, moron. Instead of getting a small amount of money from me via advertising, you'll get jack-****, because I will NOT be handing over my credit card numbers to you, EVER...no matter what you are "selling".

[Super2online]

Tech people - always viewing the world through the eyes of what they percieve everyone outside their world will think.

[n3td3v]

Nobody is going to pay to view news web sites, what a ridiculous suggestion.

[makryger]

Unless of course every news agency started charging. Let's be frank: Google is the reason newspapers across the country are going out of business. THEN where will you get your news fix?

[winstein]

The real reason newspapers are going out of business is when readers realized that most "news" in the local newspapers are from the same place: Reuters.

[Random_Walk]

"Unless of course every news agency started charging."<br /><br />...just like it would never rain if all the clouds decided to stop dropping water, and would be just as likely. :)

[Renegade Knight]

@makryger <br /> <br />You are wrong about that. The internet in general is why folks have stopped with the paper and gone online. The local news has the best bet of anyone to provide a local "first thing in the morning" home page for each community to have as their home page. That they don't just means they are missing a market. <br /> <br />Google is no substitute for what's going on in my own community. Some small timers are trying this. If the new folks don't get off their butts and do it, in time the small timers will hire away the writers and add local content and be the new news medial.

[CBSTV]

Murdoch would be doing me a favor by filtering News Corporation content from my computer.

[gerrrg]

If this is true, and Microsoft engages in this backdoor attack on Google's search results, the backlash will be severe and widespread.<br /><br />Ballmer, don't be an idiot. People reward companies that innovate and punish companies that try to hold everyone back. If you work with News Corp, you will not only lose market share in the search market, but your actions will severely cripple the rest of Microsoft's markets.

[Super2online]

According to the article News Corp was seeking Microsoft's help, not the other way around. Microsoft has everything to gain when results aren't as complete in Google, especially if this is being done at News Corps request. The article didn't mention whether or not News Corps content would continue to find it's way into Bing though.

[Renegade Knight]

Backlash? I simply will find the other sources for something that would have come up in a search. I use Google to search. I go to the site to read. If your not on Google I likely won't find you. Just as likely I won't miss you either.

[supoman]

Good riddence! He can take his junk disguised as news and stuff it!!

[nps_ca]

Mr. Murdoch aka News Corp has lost the plot for years...<br /><br />Even PAID newspapers are dying... The Evening Standard in the UK which COMPETES head on with News Corp's publications now GIVES AWAY the print edition - advertising pays for the print edition and the online edition.. No stupid premium subs.<br /><br />News Corp however is trying to screw print subs and online subs - NO concept to have advertising even at a VERY low rate per impression pay for the content... <br /><br />Mr. Murdoch has been out of it for years... Buying MySpace? What the heck for? The only guys who seem to have a clue at News Corp are his sons - but they seem to not want to confront "Daddy" on how out of touch he is...

[wshwe]

The best way to guarantee that Google can't crawl Murdoch's sites is to take them all down. The world would be a much nicer place.

[Endbringer]

Umm...aren't you commenting on a News Corp site?

[ssg13565]

News Corp does not need Microsoft's help delisting from Google. Microsoft is paying money to News Corp to delist from Google. I thought such commercial interference on the part of Microsoft with the relation between News Corp and Google was an illegal anti-competitive practice.<br /><br />Our justice deparment should jump in this like a cat on a mouse. Google ought to sue Microsoft for interference. The European Union ought to slap a huge fine on Microsoft.<br /><br />Finally, Microsoft ought to be cut up into little pieces to stop their outrageous, monoploistic practices.<br /><br />We don't need Microsoft to set the world of technology back decades. They have already done enough of that. Microsoft has become the giant it is through very sharp (not a good thing) business practices, not because of superior technology.

[winstein]

+1

[Super2online]

You are missing the point. Microsoft is being approached by News Corps to do this. Not the other way around like your comments insinuate.

[knowles2]

News corps. Microsoft who cares who initiated the contract. <br />It take two to tango, an if it got leaked it obvious the Microsoft have not broken off potentially illegal talks. An the EU will just see it as a way to double the money they get from fines.

[t8]

+2