Network feud leads to Net blackout

Two major Internet backbone companies are feuding, potentially cutting off significant swaths of the Internet for some of each other's customers.

On Wednesday, network company Level 3 Communications cut off its direct "peering" connections to another big network company called Cogent Communications. That technical action means that some customers on each company's network now will find it impossible, or slower, to get to Web sites on the other company's network.

William Steele, a senior network engineer for Syncro Services, said his company noticed the problem Wednesday morning.

"There are some people I can't send an e-mail to," Steele said. "At home, I have Road Runner as an ISP, and wasn't even able to remotely connect in order to manage our servers."

The Internet's wobbly legs
Blackout shows Net's fragility

A dispute between major Internet backbone companies has selectively shut down e-mail and Web sites for many online.

"Peering" arrangements are maintained by network companies that agree to connect their networks directly together to exchange traffic more efficiently. When the companies are of roughly equal size, money rarely exchanges hands.

Level 3 contends that its arrangement with Cogent is no longer financially viable, since it is larger than the other company. It has asked Cogent to seek other arrangements, possibly including paying for the traffic exchange, a Level 3 representative said.

Cogent CEO Dave Schaeffer contested that claim, saying that its network is at least as big as Level 3's, and that it makes no sense to pay for the connection. Cogent is offering any Level 3 user who can't get to Cogent sites free Internet service for a year, in an attempt to attract its rival's customers.

"Our goal is to have this problem go away, whether through Level 3 reconsidering or their customers coming to us," Schaeffer said.

The Level 3 representative said the company was unlikely to reconsider its position, however.

The problem is likely to affect only a small number of each company's customers. Many network company customers have several connections to the Internet and can use an alternate connection to reach a site that might otherwise be blocked.

A similar Net blackout happened in 2001, when Cable & Wireless and PSINet were embroiled in a billing dispute.

Load of Bull

I've been having a problem with this too...when is all of this going to be fixed? I contacted RoadRunner, and all they said were that they were "restructuring" their network and all of that, and they have no clue when it's all going to be said and done. Personally, I think this is all a bunch of bull, and a bunch of bull that is costing me, and a lot of other's I know, money.

Agreed.

So have I, and I'd like to know myself. I got in touch with a RoadRunner tech, and he told me that it was a known problem, and that "Unfortunately, at this time there is no estimated time of repair." Thanks a ton.

I love watching my money just shrivel up into nothingness as the time ticks by.

Corporate idiocy

This something like I see little kids indulge in on a playground.

"Im going to take my ball and go home."

Corporate idiocy

LOL
they're a bunch of Babies!
dont they know we ALL pay for the internet, it's NOT free... and when something goes down, it hurts everyone, including themselves...
and the little man is stuck right in the middle of it all..

[killerpenguinz]

This is Just Horrid

It took me 5 hours on the telephone to get an answer out of TimeWarner / Brighthouse, and i ended up telling my district what was going on.

I called Cogent, and Cogent said it was Level 3. I call level 3 and demand to know why they wont turn it back on and stop costing me money, and they pretty much said "We want money now" so apparently one of their CEO's needs a pay raise.

Too bad... Our company is thinking about filing legal action against Level 3 for their actions. We have been down since 5:44 AM EST and counting. Someone will have to pay for our business interruption, and it sure as much won't be us.

At least Cogent isn't backing down, and as a customer of Cogent (Servers in Dallas, i Work them from Orlando RoadRunner Lit Location) i respect their stance. I think Level 3 is just mining for money at this point.

[nottlv]

Save your money on the lawyers...

...and buy a multi-homed connection instead. These are private agreements between two parties, and Cogent has already admitted that L3 was within their rights to terminate their peering arrangements. L3 gave their 90 days notice that they were going to depeer Cogent, and whatever SLA you have with Cogent won't cover this. Here's a simpy analogy. Say two people say "you scratch my back and I'll scratch yours" (peering). All is great, until you realize the you're scratching the other guys back more than he's scratching yours (traffic exchange ratios), and it's causing wear and tear on your back scratcher (network maintenance). So you say, hey, you need to pay me for the extra back scratching I'm doing (paid transit) otherwise there will be no back scratching between us (depeering). The other guy says no, I'm not paying for what I used to get for free, so you stop scratching each other's back.

Info please...the bush is dead, stop beating it

What I'd also like to know is HOW we are supposed to get the alternate connections. Our RoadRunner, I believe, is set up on one path. I'm not sure how to get it to go on the "alternate" path, at least not from where I am. If anyone knows, let me, and anyone who reads this, know.

[pedershk]

That's Roadrunner's job, not yours.

Roadrunner will have to set up alternate connections in their network. I doubt their customers (i.e. you) will have to do anything.

Maybe This Information Will Help?

In regards to find another alternative path for Internet access, from customer premises to Internet backbone. Check with the Local Exchange Carrier of your city, which is the first physical link for Internet connection(fiber, cable or copper wire) to a customer premises, to see about connecting to another carrier or Inter-Exchange Carrier, which will link you to the Internet backbone for services. This option should be available, because all IXCs interface their networks with LECS in order to provide Internet& voice services.

The solution would be to order another temporay circuit(T-1,Frac/T-1) or business line(DSL,Dial-up)on a month to month bases, until the disbute is resolve and services are back to normal.

Another solution would be a wireless connection to the Internet, this provides a back-up access connection, and a different geograghical pathway.

These temporary solutions will help put your Internet service back to noraml, well the disbute is resolved and working legal matters.

Also, that is way Cogent is offering free service, because their company and Level 3 are aware of these solutions. I am sorry the customer is not being put first, by offering temporary Internet access solutions.

Hopefully Level 3 and Cogent can let go of their proud-pride, and be humble, so each company can work togather for taking care of their customer, who are in need. The first step is going through humility and apolgizing, then forgive.

The end of the matter is always better then the begin or patience is better than pride, when there is a problem to fix.

[bbatsell]

BGP?

Um, isn't BGP4 supposed to make these kinds of things pretty
much completely unimportant (with the exception of a couple of
increased hops)? I don't understand how any Level3 customers
wouldn't be able to reach Cogent or vice versa - their packets
should simply route through a secondary carrier in between the two
networks.

[JB_Smith]

Level3 has blackholed all packets from Cogent

In forcing Cogent to cave in and pay for their peering, Level3 has
blackholed all packets to/from Cogent CIDR Blocks. Thanks for
letting anyone know beforehand Level3! BGP has no means to
bypass without Cogent doing some form of NAT which would be
completely impractical, or doing a complete renumbering which
L3 could just further exclude very quickly. L3 is playing with
tactics akin to ........ I won't even say it. In addition L3 just
forced nearly every major IP subscriber to actively pay DOUBLE
for access simply to achieve redundancy that should not be
needed were it not for this kind of behavior. More than likely,
most subscribers will be planning a way to avoid having L3 in
their lineup of providers.

RE:BGP

You are correct however not only did level 3 disconnect their peering with cogent, but they also are blocking all announced routes that originate from cogent. It is simple, Level 3 just put in some nice route filters on all of their routers.

[OneWithTech]

A lesson to be learned

If there is any lesson to be learned here for both companies, it is this:

Don't ever think you can relie on a joint partnership like this to last. You first mode of defense is a good offence.

Neither of these companies had though that the other would try to do them wrong. The fact is that this is a money driven society.

Level 3, in the act of business, has decided that this partnership has reached a turning point and it's is not an even partnership, but rather an uneven partnership. Hence the need for change.

The lesson:
Always have a backup plan. Both of these companies should of never relied on there partnership a %100.

Having a plan B, a backup plan, doesn't just pertain to football! A backup plan would of ensured a smooth transaction in this case. Instead you have to publicly announce that you want the competitors clients because they cut off your line.

Maybe both of these managers should head back to the classroom. Business 101 maybe!

Justin

Childish Enough to Not Have a Written Contract?

Where are these people - who's in charge - there should be something in writing - - - OR is this whole thing just a system to add another layer of funding for someone to "collect" and not do anything.

Those in the middle getting screwed

I own a web design business and an Internet Radio station. I have clients for the web design business that have been calling for the last 20 hours demanding to know why they can't see their own website from their home or work. I've got listeners sending me mail from yahoo or hotmail about not being able to listen. I've got advertisers screaming at me for not being able to deliver.

I don't even have RoadRunner nor is it available here in Rochester, MN or Madison, WI where I just moved from.

So, we are caught in the middle of a turf war that does not even affect us from an ISP standpoint.

Class Action lawsuits are nice and all, but I need resolution NOW and not in 2 - 6 months when things go to trial.

There is no way I can contact the 3,000 or so listeners I get an hour to tell them to please change their ISP and find a backbone that works. There is no way I can make all my clients get their customers to switch to a ISP that has Level 3.

Perhaps things like this can be handled via the US Gov't in the same way the railroad strikes in the past were handled because it's got a deep impact on commerce and while things are flowing normaly there are discussions with a goverment mediator involved. I know the gov't has no ownership but then again there was no ownership with the railroad companies or airlines when PATCO did a walkout that had crippled the air travel industry in 1981.

Thanks,
Chris J. Popp
9412 - The Rock Station
www.9412.com

new economy business model

Chris sounds like he is operating out on an ewxciting frontier. I think he's cool and all, but I'm not ready to swallow his analysis comparing webhosting and web based media transmision to air trafiic control.
Sometiems there's lawlessness on the frontier.

The marketplace requires you to kick provider ass.. take your friends with you.

[killerpenguinz]

Re: Those in the middle getting screwed

I agree man, i'm in the same boat as you are. Gotta love how all this just falls apart so easily.

I already contacted my Representatives, Senators, and Govenor. I suggest all try that. Maybe we can light a fire under someones ass.

Christopher Clai
Killer Rock Radio - www.krradio.com

Blackout

Did I understand you correctly? Can I solve this blackout problem by just changing my ISP? I'm more worried about getting back online than which company makes more bucks.
I have Roadrunner now. Who would you suggest?
Thanks for your input.

[cgamboak]

UN should be the administrator.

today a company tomorrow a city a country. The network must be free to all

UN control?

Cristian Gamboa writes:

"UN should be the administrator. today a company tomorrow a city a country. The network must be free to all"

Oh yeah, the UN has such a terrific track record of managing things, right?

No thanks. I'll take my chances with companies working things out.

[Txporchhound]

UN never the answer

If you think its bad now wait until a multi-national/multi-theological committee begins controlling the internet. What has the UN EVER administered well?

UN should be the administrator.

The UN isn't able to take care of itself. Control of the Internet needs to remain right where it is.The US developed the internet as part of a Pentagon program some 30 odd years ago and has since turned over the running of it to a private company who is handling it far far better than the UN could even dream of doing.

As for the Internet being free to everyone. Then so should phone service, gas & electric, water ... ... Get my drift? At birth is every child to be assigned an E:Mail address, laptop and a wireless ISP? Please!

In this world you pay for goods and services. You need earn what you want.When things are given to people, such as wellfare and subsidized housing, it is not appreciated. If I have gone to the trouble to establish a business that will provide you with a service why should I not get paid for that?

UN should be the administrator

boy that's funny...!!!
they cant even control theirselves much less the internet! lol...

RE: UN SHOULD BE THE ADMINISTRATOR

Are you nuts?

What would the UN do to resolve a dispute between two providers?

If the UN were involved the network would be free to no one as they bend over backwards to fullfill requests by nations like China to regulate it.

[DrakeLoneStar]

No way

The U.N. is useless as an organization.

The Europeans veto everything anyways with a 'consequence' clause rendering non-execution with resolutions meaningless.

The U.N. should be dissolved.

You'll never find an unbiased organization to control the Internet. It's been working fine under U.S. control so far. Besides, the U.S. created the Internet with government money.

I don't have a solution for this, but the U.N. would be worse than the U.S.

Maybe an Internet body with one person from each major country that actually pays for the infrastruction. Such as one guy from the U.S., one guy from Asia, one guy from Europe, who knows.

But *not* the U.N.

No UN should not be.

Forget that, it would degrade the integrity of the net.

[nicmart]

UN totalitarians

How many of the UN member countries could reasonably be
described as free? How charming it would be for Robert Mugabe
and Hugo Chavez to have a say in how the Internet works. Better
idea: let's get a divorce from the third world country known as the
United Nations.

[jane colorado]

Earthlink Affected?

Does anyone know if Earthlink has been affected by this dispute? I have Earthlink DSL and it has been exceedingly slow the past two days. Can't get on to their support line, as the decision tree just disconnects callers.

[naomibigelowbooks]

Earthlink Affected Yes & No

I have Earthlink dial-up (rural area with no cable or DSL). Everything works fine EXCEPT my own website; I can't access my site or any other FreeWebs site, and haven't been able to since Wednesday. I fail to see how these people's dispute can be permitted to interrupt untold other entity's service and production. I've filed my complaint with the FCC, for whatever that's worth. The tech support at FreeWebs is truly caught in the middle, as are all our ISPs.

[RU_Kidding_Me2]

Level 3 Communications Phone Numbers

This post might get me in trouble, but I'm sick and tired of
companies with acting irresponsibly. This is why the web
should be treated as an utility. Please Call Level 3, I'll bet
they don't answer. These numbers are all in Colorado.
1. 303-410-7294
2. 720-888-1000
3. 925-398-3000

Peering/transit

"peering" is what two providers do then they exchange equal amounts of Internet traffic between each other. There are standard guidelines around qualifications to peer with each other so neither peer needs to pay for the exchange. "Transit" is when there is not enough traffic exchange from one provider to justify the "free" connectivity. The traffic exchange between level3 and cogent is not enough to merit a peering relationship, and for level-3, who is struggling like every other Internet Carrier, they can longer provide Cogent free circuits that can be better used otherwise. Cogent's whining is BS. Having working in Internet serviecs for many years, I'm quite certain Cogent recieved plenty of notification that peering would be discontinued. It appears Cogent chose to ignore it and screw their customers rather than pay for what they actually use.

i made a call

i called the first number you gave.and someone did answer a woman on the line said she doesn't no what this is all about.so i redialed the number again and the same woman answered.i told her she would probably get a lot of calls and thanked me for letting her no.i have road runner all in one which consist of my cable service my internet service and my phone service.is the phone service also effected or did you give a incorrect phone number ?

Yes, Earthlink is effected!

I have Earthlink Dial up, and one of my friends have Earthlink DSL and both are effected... I can view some webpages, and some I cant...
this is really bothersome and I wish they would hurry up and stop acting like babies and get it worked out!
Thank you,

[Mister C]

Ah Yes!

The free market, ain't it grand. Especially when applied to critical public resources! But have no fear, our government will come through and the golden rule will prevail. Them that's got the gold will end up making the rules and those who can pay will play and those who can't will be left out. J P would be so proud!

Lawyers will get there first

The government will be slow to respond but the lawyers will be all over it in a flash.

i applaud level3

ive have complained many times to cogent communications regarding porn spam originating from off their system and they never do anything about it leading me to the conclusion that they are pro or soft on porn and spam in general

as far as iam concerned cogent should be blackholed from off the internet until they clean up their spam/aup policy or at the least enforce same

Use some freaking spamblocker

This isn't at all about spam...Level 3 didn't disconnect the peering because of an influx of porn....just because you get porn and spam, doesn't mean that thousands other people should suffer business loss because you don't have proper filtering.

[c|nuts]

Spam could be a factor

Cogent does have a long reputation for willingly hosting
spammers, hackers, frauds, and criminals. Level3 might not
want Cogent's traffic clogging their networks and attacking their
own customers using high bandwidth direct peering.

Who would you email?

I'd love to know who to email on this situation... I'll have everyone in my groups email them, if we boycott, then something has to give!!!
Please someone reply with website or email please.

[Arbalest05]

Both of these companies may be under financial stress

Both of these companies are publically traded and both have pretty ugly charts (COI & LVLT). It sounds like Level 3 is trying to squeeze a few dollars out of their rival.

Maybe, but...

Even if Level 3 is trying to squeeze a few bucks out of Cogent by doing all of this...if in the end they end up losing customers because of all of this BS, they did more harm than good.

Spamblocker

if you have a good ISP, then you have Spam blocker... I suggest you dont want Spam/Porn.....use it....

Birns Telecommunications is affected as well

It appears as though Birns Telecommunications in Manhattan is also being affected by this feud, as my companies internet connection has been extremely slow. We get our T1 line from Birns, who in turn gets there connection from Level3. Hopefully Birns be able to find another service provider to help restore connection speeds.

Level 3

Level 3 has been in financial trouble from the start--I have some of their rotten stock (thought the company would get straightened out)and now have gotten screwed by them AGAIN!!!

Message has been deleted.

who is e.t.?

lol whatever this guys on i'll take two!

Spam crap here...

This is a thread from someone spaming multiple discussion talkbacks. I've made a complaint, and I suggest you do so as well. It's not needed here and does not belong.

solution

Should Roadrunner ISP clients such as me get a backup ISP that isn't blocked from sites routed through Cogent, or simply await a resolution? Seems like having to have vhs and betamax for those as old as I!
I couldn't get into Drudge yesterday but can today. Did he re-route?

I emailed Level 3, donno whether it did any good...

but I emailed them.. lol.

This is reeking Havoc on a lot in the internet world.
I cant understand why you guys cant get past the problems and resolve this Quikly.
I have Earthlink Dial up, and it's causing a lot of problems for Earthlink Customers!!
I will/am in the process of sending a Boycott to all of my Yahoo groups to Boycott Level 3 because of this.
I'm not affiliated in any way with cogent, neither do I own Stock in either company, but I do know people who own stock, and will let them know to sell their stock out of your company.
Like I said, it's causing a Lot of problems in the internet world.
Some websites I cannot even veiw because of this!!
Please resolve this problem.
Thank you.

[drewbyh]

Call the FCC

If the issue is affecting VoIP then it's affecting your 911 service and that is illegal. 1-888-CALL-FCC

re: Call the FCC

how would it effect our 911 service?
and Why would it, if it's a local service?

oh yeah get the govment

involved maybe they will send brownie to straighten it out ~rolls eyes~

911 and VoIP

VoIP suppliers even tell you that you should have a regular phone line for 911 in case of outages. I don't think there's anything the FCC can do for this.

Yes, 911 is local, but for those who don't know about VoIP, 911 calls have to go through a relay system to get to your local emergency office.

My Company's Bank is now 1.1.1.1

We can't connect to our bank to do wire transfers and all the stuff that our company does all day basically. When I did a WHOIS on the bank, I get a 1.1.1.1 as their IP. *** is going on? Are they gonna fix this situation or what? Do I just wait until the DNS fixes itself by propogating the net?

[rnmina]

HUH?

I want whatever it is;)