October 6, 2005 2:15 PM PDT

Blackout shows Net's fragility

A correction was made to this story. Read below for details.
Since early Wednesday, Phil Bradham, the network engineer at Boston's Museum of Fine Arts, has been cut off from the parts of the Internet he needs the most.

He can't reach his Web hosting company to update his site. Critical e-mails aren't going through, and some aren't reaching him. He can't get to some important sites on the Net, such as the popular Wikipedia encyclopedia.

The source of Bradham's difficulties is a feud between two big backbone Internet companies--the long-haul networks that most consumers and even most businesses ordinarily have little to do with. One of these companies, Level 3 Communications, has cut off direct communications with rival Cogent Communications, causing many of each company's customers to lose access to potentially significant swatches of the Net.

"We've been working with both (companies), but neither one will do anything until the other one budges," said Bradham. "It's very frustrating that two top companies would try to resolve this with a standoff like this."

In theory, this kind of blackout is precisely the kind of problem the Internet was designed to withstand. The complicated, interlocking nature of networks means that data traffic is supposed to be able to find an alternate route to its destination, even if a critical link is broken.

In practice, obscure contract disputes between the big network companies can make all these redundancies moot.

At issue is a type of network connection called "peering." Most of the biggest network companies, such as AT&T, Sprint and MCI, as well as companies including Cogent and Level 3, strike "peering agreements" in which they agree to establish direct connections between their networks.

That means that when a Cogent customer wants to visit a Web site hosted by Level 3, the data can take a short, fast path, instead of winding its way around the broader Internet.

Typically, peering agreements are made without any money changing hands, since each company expects to hand off a roughly comparable amount of traffic. Smaller network companies buy what are called "transit" agreements with larger companies, in order to hand off their customers' traffic to the big networks.

Peering gone wrong
These collegial peering relationships among big companies allow traffic to flow efficiently across the Net without most customers knowing anything about the under-the-hood relationships. But when these relationships go sour, the feuding parties' lack of flexibility can result in blackouts like the one that occurred this week.

In this case, Level 3 says that it believes it is substantially larger than its rival, and told Cogent as long as 90 days ago that it was planning to sever the direct connection between the two networks. The connection could be re-established if Cogent were to pay Level 3 access fees for use of its network, the company says.

For its part, Cogent contends that it is similar in size to Level 3, and that it makes no sense to pay for the kind of peering relationship that it maintains with many other companies. Cogent is offering any Level 3 user who can't get to Cogent sites free Internet service for a year, in an attempt to attract its rival's customers.

"Our goal is to have this problem go away, whether through Level 3 reconsidering, or their customers coming to us," said Dave Schaeffer, chief executive officer of Cogent.

As of mid-Tuesday, both sides said they were committed to their position, showing no willingness to budge, despite complaints from customers on both sides around the Net that they can't reach Web sites or can't send e-mail to some addresses or receive it from others. This means that there is no immediate fix ahead, unless customers (or their ISPs) find an alternative or auxiliary network provider.

The scale of the problem
It's impossible to say precisely how many people are affected. Many customers of the two companies, and customers of the ISPs

 

Correction: This story incorrectly stated the day Phil Bradham began having difficulties accessing his Web site. The problem began Wednesday.

CONTINUED:
Page 1 | 2

See more CNET content tagged:
Cogent Communications, Level 3 Communications Inc., network company, relationship, agreement

82 comments

Join the conversation!
Add your comment
Level-3 is bad -- Cogent is good
I've done business with both Level-3 and Cogent.

Cogent is a positive force on the Internet. They are bringing down bandwidth prices. The quality of their service is excellent and their technical support shines too.

Level-3 on the other hand is a brutish, old-fashioned, monopoly-wannabe. They are trying to control the internet. They have always pushed around other companies. Their arrogance is matched only by their business incompetance.

It's important the public come out against Level-3 and for Cogent.

Cogent can make our lives better. Level-3 will make our lives worse.

Level-3 is the enemy of the internet and the people.

Level-3 also mistreats their own employees. They treat their workers like numbered slaves.

Please get involved in the fight.
Posted by (3 comments )
Reply Link Flag
Cogent is bad -- Level-3 is good
Cogent's network is nothing more than porn, warez and spam. I
frequently am inundated with spam from Cogent's network. I hope
they are forced to actually pay to distribute their crap to other
networks like Level-3.
Posted by testtest (20 comments )
Link Flag
B.S. - Level 3 did exactly what it should have done before
That's BS. Level(3) depeered Cogent because cogent probably did not meet its in:out ratios. Cogent can go and BUY transit to get to Level(3) or it can stop dumping IP well below its cost.

Cogent decided it was not going to do that. It bet on Level(3) not pulling the peering. It bet wrong. Now it got caught with its pants down. It seems now it is also rather clear who has larger network - it is Cogent's customers that are ******** about the issues a lot more than the Level(3) customers.
Posted by Alex Yuriev (9 comments )
Link Flag
Level-3 is bad -- Cogent is good
I've done business with both Level-3 and Cogent.

Cogent is a positive force on the Internet. They are bringing down bandwidth prices. The quality of their service is excellent and their technical support shines too.

Level-3 on the other hand is a brutish, old-fashioned, monopoly-wannabe. They are trying to control the internet. They have always pushed around other companies. Their arrogance is matched only by their business incompetance.

It's important the public come out against Level-3 and for Cogent.

Cogent can make our lives better. Level-3 will make our lives worse.

Level-3 is the enemy of the internet and the people.

Level-3 also mistreats their own employees. They treat their workers like numbered slaves.

Please get involved in the fight.
Posted by (3 comments )
Reply Link Flag
Cogent is bad -- Level-3 is good
Cogent's network is nothing more than porn, warez and spam. I
frequently am inundated with spam from Cogent's network. I hope
they are forced to actually pay to distribute their crap to other
networks like Level-3.
Posted by testtest (20 comments )
Link Flag
B.S. - Level 3 did exactly what it should have done before
That's BS. Level(3) depeered Cogent because cogent probably did not meet its in:out ratios. Cogent can go and BUY transit to get to Level(3) or it can stop dumping IP well below its cost.

Cogent decided it was not going to do that. It bet on Level(3) not pulling the peering. It bet wrong. Now it got caught with its pants down. It seems now it is also rather clear who has larger network - it is Cogent's customers that are ******** about the issues a lot more than the Level(3) customers.
Posted by Alex Yuriev (9 comments )
Link Flag
Poor TimeWarnerCable and Roadrunner
yea when Rita hit , TimeWarner Cable/Roadrunner was cut offline for 3 hours because of generator failed in houston DataCenter and this dispute.. makes me glad i have SBC. i think it time for Congent to go For Profit and let Level 3 do it own thing.
Posted by googlefan33 (36 comments )
Reply Link Flag
Poor TimeWarnerCable and Roadrunner
yea when Rita hit , TimeWarner Cable/Roadrunner was cut offline for 3 hours because of generator failed in houston DataCenter and this dispute.. makes me glad i have SBC. i think it time for Congent to go For Profit and let Level 3 do it own thing.
Posted by googlefan33 (36 comments )
Reply Link Flag
Multinational wars and the Internet
This could be a prelude to a new kind of war: One between the multinationals. Big companies who don't have any loyalty to any country, but only to the quest to make as much money as possible even though it means the customer gets the shaft.

In the book "Friday" by Robert A. Heinlein, practically all the violence in the book was caused by these companies and factions within them, not by nations at war with each other for territory or dwindling natural resources (those kinds of wars we'll likely be seeing later, as we run out of oil and clean water in the next ten to twenty years as even a certain well-known oceanographer stated that they'll be gone in twenty years back in 1995). Interesting reading if you want to try interpreting the intentions and reasoning behind corporations' facades. LOL But now, the masks are slipping, and we catch glimpses of what they really care about.

"Sometimes, it's all about the gold." The main motivations for humans is power and money (which in many circles is one and the same), fame, and the other one, which I won't name, but leads to increases in population.

The Internet was supposed to be set up so that it could survive in the event of nuclear war, like a fishnet of sorts, back in 1969, when it started as the ARPAnet. One spot gets taken out, and stuff goes around it. Kind of like guerilla warfare in a way, you don't give the enemy a central position to target (if you can be seen, you can be hit, and if you can be hit, you can be killed). But, with control of the Internet in the hands of corporations, we are playthings to their whims. People don't get access to information, their e-mails never get sent to where they sent them, non-profits like the one in the story as well as small businesses suffer... We all lose. Not that we should allow governments to control the Internet, either, or else they'll likely create a neck-crushing grip on what people can say and do on the Internet, attempting to make it so that no single person of any culture gets offended.

As U.S. District Judge Stewart Dalzell wrote: "As the most participatory form of mass speech yet developed, the Internet deserves the highest protection from governmental intrusion." (More quotes here, if you're interested, and no, it's not a real commercial site: www.cyberwolfman.com/quotes.htm)

Oops. I'm ranting again, aren't I? LOL

- CyberWoLfman
Posted by CyberWoLfman (47 comments )
Reply Link Flag
Multinational wars and the Internet
This could be a prelude to a new kind of war: One between the multinationals. Big companies who don't have any loyalty to any country, but only to the quest to make as much money as possible even though it means the customer gets the shaft.

In the book "Friday" by Robert A. Heinlein, practically all the violence in the book was caused by these companies and factions within them, not by nations at war with each other for territory or dwindling natural resources (those kinds of wars we'll likely be seeing later, as we run out of oil and clean water in the next ten to twenty years as even a certain well-known oceanographer stated that they'll be gone in twenty years back in 1995). Interesting reading if you want to try interpreting the intentions and reasoning behind corporations' facades. LOL But now, the masks are slipping, and we catch glimpses of what they really care about.

"Sometimes, it's all about the gold." The main motivations for humans is power and money (which in many circles is one and the same), fame, and the other one, which I won't name, but leads to increases in population.

The Internet was supposed to be set up so that it could survive in the event of nuclear war, like a fishnet of sorts, back in 1969, when it started as the ARPAnet. One spot gets taken out, and stuff goes around it. Kind of like guerilla warfare in a way, you don't give the enemy a central position to target (if you can be seen, you can be hit, and if you can be hit, you can be killed). But, with control of the Internet in the hands of corporations, we are playthings to their whims. People don't get access to information, their e-mails never get sent to where they sent them, non-profits like the one in the story as well as small businesses suffer... We all lose. Not that we should allow governments to control the Internet, either, or else they'll likely create a neck-crushing grip on what people can say and do on the Internet, attempting to make it so that no single person of any culture gets offended.

As U.S. District Judge Stewart Dalzell wrote: "As the most participatory form of mass speech yet developed, the Internet deserves the highest protection from governmental intrusion." (More quotes here, if you're interested, and no, it's not a real commercial site: www.cyberwolfman.com/quotes.htm)

Oops. I'm ranting again, aren't I? LOL

- CyberWoLfman
Posted by CyberWoLfman (47 comments )
Reply Link Flag
I Agree. Cogent is Doing Good.
Look, Cogent has problems before 2003, we all know this. It's not their fault for this contract dispute. Level 3 did this to Cogent, and several other providers.

Cogent has a comparable Network, and Level 3 wants money.

I have never, had a problem with Cogent, as a CUSTOMER, not as a whiney complainer to their net op's.

2 years and running 1 connection issue that got resolved in 10 minutes. Now that is service. TimeWarner? Lets wait 2 days. Level 3? Lets not talk how long it takes them.

When i called Cogent about the situation, they explained down to detail. Level3 kept claiming it was a business decision, and when i told them that they need to rethink their position, i was told to give my information for submission to their legal counsel. I hope, for the sake of the Internet, Level 3's stock price plummets, they get delisted, and declare bankruptcy, so Cogent can buy them out. Then the end of the problem.
Posted by killerpenguinz (19 comments )
Reply Link Flag
They should have learned from past mistakes.
Why am I not surprised that it happened again? Cogent should have learned from it's past mistakes, like the 2002 outage with AOL/TimeWarner.

And being pre-warned about it, makes it even worse.

Two thumbs down for Cogent.
Posted by (8 comments )
Link Flag
I Agree. Cogent is Doing Good.
Look, Cogent has problems before 2003, we all know this. It's not their fault for this contract dispute. Level 3 did this to Cogent, and several other providers.

Cogent has a comparable Network, and Level 3 wants money.

I have never, had a problem with Cogent, as a CUSTOMER, not as a whiney complainer to their net op's.

2 years and running 1 connection issue that got resolved in 10 minutes. Now that is service. TimeWarner? Lets wait 2 days. Level 3? Lets not talk how long it takes them.

When i called Cogent about the situation, they explained down to detail. Level3 kept claiming it was a business decision, and when i told them that they need to rethink their position, i was told to give my information for submission to their legal counsel. I hope, for the sake of the Internet, Level 3's stock price plummets, they get delisted, and declare bankruptcy, so Cogent can buy them out. Then the end of the problem.
Posted by killerpenguinz (19 comments )
Reply Link Flag
They should have learned from past mistakes.
Why am I not surprised that it happened again? Cogent should have learned from it's past mistakes, like the 2002 outage with AOL/TimeWarner.

And being pre-warned about it, makes it even worse.

Two thumbs down for Cogent.
Posted by (8 comments )
Link Flag
The Blackout
How incredibly stupid. The government and the nation have too
much invested in a smooth operating internet to allow this petty
BS to continue. If they want the Gov't to seize them and take
over, let them keep this blackout going.

There is no such thing as excess profits if you are in business
but stealth beats crude tricks like this. I'd rather have $.0001 or
less on every internet connection than risk losing the whole
thing.

Unbelieveable!

SEW
Posted by (2 comments )
Reply Link Flag
The Blackout
How incredibly stupid. The government and the nation have too
much invested in a smooth operating internet to allow this petty
BS to continue. If they want the Gov't to seize them and take
over, let them keep this blackout going.

There is no such thing as excess profits if you are in business
but stealth beats crude tricks like this. I'd rather have $.0001 or
less on every internet connection than risk losing the whole
thing.

Unbelieveable!

SEW
Posted by (2 comments )
Reply Link Flag
Not the first time for Cogent!
It is not the first time Cogent has been in peering trouble. Couple of years ago, same thing happened with Cogent and AOL/TimeWarner. At that time it took Cogent weeks to restore service back to their customers.
Posted by (8 comments )
Reply Link Flag
Not the first time for L3 either.
Look around on the web. This isn't the first time L3 has done this either.
Posted by katamari (310 comments )
Link Flag
Not the first time for Cogent!
It is not the first time Cogent has been in peering trouble. Couple of years ago, same thing happened with Cogent and AOL/TimeWarner. At that time it took Cogent weeks to restore service back to their customers.
Posted by (8 comments )
Reply Link Flag
Not the first time for L3 either.
Look around on the web. This isn't the first time L3 has done this either.
Posted by katamari (310 comments )
Link Flag
Articles about prior Cogent blackouts
Searching Google for Cogent AOL turns up quite a few interesting articles.

One of them are:

<a class="jive-link-external" href="http://legalminds.lp.findlaw.com/list/cyberia-l/msg42080.html" target="_newWindow">http://legalminds.lp.findlaw.com/list/cyberia-l/msg42080.html</a>
Posted by (8 comments )
Reply Link Flag
Articles about prior Cogent blackouts
Searching Google for Cogent AOL turns up quite a few interesting articles.

One of them are:

<a class="jive-link-external" href="http://legalminds.lp.findlaw.com/list/cyberia-l/msg42080.html" target="_newWindow">http://legalminds.lp.findlaw.com/list/cyberia-l/msg42080.html</a>
Posted by (8 comments )
Reply Link Flag
Level 3 and Cogent
I don't believe I quite understand the views from Level 3 here... Cogent sends traffic across level 3's network. There's also the fact that Level 3 sends traffic accross Cogents network. Why is Level 3 asking for compensation where Cogent has not?
Posted by (2 comments )
Reply Link Flag
Peering goes two ways!
They are asking for compensation because the traffic ratio between the two networks is way off.

There would probably be no problem if there was an equal balance in the exchange of data over the two networks.
Posted by (8 comments )
Link Flag
Level 3 and Cogent
I don't believe I quite understand the views from Level 3 here... Cogent sends traffic across level 3's network. There's also the fact that Level 3 sends traffic accross Cogents network. Why is Level 3 asking for compensation where Cogent has not?
Posted by (2 comments )
Reply Link Flag
Peering goes two ways!
They are asking for compensation because the traffic ratio between the two networks is way off.

There would probably be no problem if there was an equal balance in the exchange of data over the two networks.
Posted by (8 comments )
Link Flag
Level 3 VS Cogent
Current score:

Level-3 3-5 Cogent




<a class="jive-link-external" href="http://ahseng.blogsome.com" target="_newWindow">http://ahseng.blogsome.com</a>
Posted by toaaron (11 comments )
Reply Link Flag
Level 3 VS Cogent
Current score:

Level-3 3-5 Cogent




<a class="jive-link-external" href="http://ahseng.blogsome.com" target="_newWindow">http://ahseng.blogsome.com</a>
Posted by toaaron (11 comments )
Reply Link Flag
Personally, I'd like to sue those involved
My customers have been complaining they cannot reach our website to complete their orders and emails automatically generated to our shipping hub are being undelivered. This started yesterday and with no warning. I'm basically screwed.
Posted by ericnn24 (7 comments )
Reply Link Flag
Personally, I'd like to sue those involved
My customers have been complaining they cannot reach our website to complete their orders and emails automatically generated to our shipping hub are being undelivered. This started yesterday and with no warning. I'm basically screwed.
Posted by ericnn24 (7 comments )
Reply Link Flag
Class action should be taken.
I'm a small ISP with a national presents. I'm trying to grow my company so I can make a life for myself and my family. Over HALF of my user base is screwed because of this stupid dispute. I personally feel that both companies should be held responsible for business loss for those of us that have contracts with them. They are interrupting my business and causing me to loose money because of customers leaving thinking that other ISP's are better. Sure, they will leave, but when they find out that it's the same problem with the other ISP, they aren't going to simply swap back. That is lost revenue for me. Other questions I have is why does this mesh internet networking not work? Who controls the routing on the peering points? Why the hell do all these hosting companies support L3 and Cog after they decide to pull the plug on the net? If not a law suit, then boycot them for childish actions. These write up's I've read is all about who is bigger? Give me a break. It's about money? Oh please. You here's a business 101 for ya... You pull the plug on your customers, you loose money. Simple as that. Now who has such great wisdom to dispute that? I'm already taking steps to make sure my T1's, T3's are MOVED from L3 and Cogent. Neither one of them will ever have my business again. As for the dialup side of things, ALL customers that are using L3's dialups are being moved as well. THAT is the power I have. So SCREW BOTH OF YOU L3 AND COGENT!
Posted by (12 comments )
Reply Link Flag
I hope...
I hope your business and technical prowess is superior to your spelling and grammar skills.
Posted by DoohanOK (51 comments )
Link Flag
Class action should be taken.
I'm a small ISP with a national presents. I'm trying to grow my company so I can make a life for myself and my family. Over HALF of my user base is screwed because of this stupid dispute. I personally feel that both companies should be held responsible for business loss for those of us that have contracts with them. They are interrupting my business and causing me to loose money because of customers leaving thinking that other ISP's are better. Sure, they will leave, but when they find out that it's the same problem with the other ISP, they aren't going to simply swap back. That is lost revenue for me. Other questions I have is why does this mesh internet networking not work? Who controls the routing on the peering points? Why the hell do all these hosting companies support L3 and Cog after they decide to pull the plug on the net? If not a law suit, then boycot them for childish actions. These write up's I've read is all about who is bigger? Give me a break. It's about money? Oh please. You here's a business 101 for ya... You pull the plug on your customers, you loose money. Simple as that. Now who has such great wisdom to dispute that? I'm already taking steps to make sure my T1's, T3's are MOVED from L3 and Cogent. Neither one of them will ever have my business again. As for the dialup side of things, ALL customers that are using L3's dialups are being moved as well. THAT is the power I have. So SCREW BOTH OF YOU L3 AND COGENT!
Posted by (12 comments )
Reply Link Flag
I hope...
I hope your business and technical prowess is superior to your spelling and grammar skills.
Posted by DoohanOK (51 comments )
Link Flag
Customer Service
You can obviously see that there is issues with the customer service thats being presented by Level 3. It seems as though Level 3 doesn't care about it's customer's, let alone the frustration and monetary loss by smaller companies due this greediness!

Essentially the internet isn't supposed to rely on any one company. The sites that are being affected here are sites that are hosted by these two companies.

Is there grounds for reprimand to Level 3? I think not. Can you sue Level 3? I think not. Somewhere buried deep in both Congent and Level 3 networks are terms of use for there networks.

Essentially these two companies are helping out the internet by agreeing to connect there rather large networks. Does it matter who's network is larger? NO.

Bandwidth split between these two companies is menial and the true cost is in maintenance of the networks.

I'm done.
Justin
Posted by OneWithTech (196 comments )
Reply Link Flag
Customer Service
You can obviously see that there is issues with the customer service thats being presented by Level 3. It seems as though Level 3 doesn't care about it's customer's, let alone the frustration and monetary loss by smaller companies due this greediness!

Essentially the internet isn't supposed to rely on any one company. The sites that are being affected here are sites that are hosted by these two companies.

Is there grounds for reprimand to Level 3? I think not. Can you sue Level 3? I think not. Somewhere buried deep in both Congent and Level 3 networks are terms of use for there networks.

Essentially these two companies are helping out the internet by agreeing to connect there rather large networks. Does it matter who's network is larger? NO.

Bandwidth split between these two companies is menial and the true cost is in maintenance of the networks.

I'm done.
Justin
Posted by OneWithTech (196 comments )
Reply Link Flag
Fix this or face regulation!!
I don't think these folks realize that if this "battle" goes on much
longer, many of those affected will ask their Congressman to fix it
for them.

If these two company's wish to avoid being heavily regulated, they
need to resolve this issue by some other means. Shutting down
access is not the right approach.
Posted by m.meister (278 comments )
Reply Link Flag
Too late...
Congress is already starting. Read today's headines.
Posted by TheReaperD (189 comments )
Link Flag
 

Join the conversation

Add your comment

The posting of advertisements, profanity, or personal attacks is prohibited. Click here to review our Terms of Use.

What's Hot

Discussions

Shared

RSS Feeds

Add headlines from CNET News to your homepage or feedreader.