How Pakistan knocked YouTube offline (and how to make sure it never happens again)
This graph that network-monitoring firm Keynote Systems provided to us shows the worldwide availability of YouTube.com dropping dramatically from 100 percent to 0 percent for over an hour. It didn't recover completely until two hours had elapsed.
(Credit: Keynote Systems)A high-profile incident this weekend in which Pakistan's state-owned telecommunications company managed to cut YouTube off the global Web highlights a long-standing security weakness in the way the Internet is managed.
After receiving a censorship order from the telecommunications ministry directing that YouTube.com be blocked, Pakistan Telecom went even further. By accident or design, the company broadcast instructions worldwide claiming to be the legitimate destination for anyone trying to reach YouTube's range of Internet addresses.
The security weakness lies in why those false instructions, which took YouTube offline for two hours on Sunday, were believed by routers around the globe. That's because Hong Kong-based PCCW, which provides the Internet link to Pakistan Telecom, did not stop the misleading broadcast--which is what most large providers in the United States and Europe do.
This is not a new problem. A network provider in Turkey once pretended to be the entire Internet, snarling traffic and making many Web sites unreachable. Con Edison accidentally hijacked the Internet addresses for Panix customers including Martha Stuart Living Omnimedia and the New York Daily News. Problems with errant broadcasts go back as far as 1997.
It's also not an infrequent problem. An automatically-updated list of suspicious broadcasts created by Josh Karlin of the University of New Mexico shows apparent mischief--in the form of dubious claims to be the true destination for certain Internet addresses--taking place on an hourly basis.
So why hasn't anyone done something about it? False broadcasts can amount to a denial-of-service attack and, if done with malicious intent, can send unsuspecting users to a fake bank, merchant, or credit card site.
To understand why this is both a serious Internet vulnerability and also difficult to fix requires delving into the technical details a little.
How to pretend to be YouTube.com
When you type a domain like "news.com" into your Web browser, it uses the Domain Name System to cough up a numeric Internet address, which in our case is 216.239.113.101. That IP address is handed to your router, which uses a table of addresses to figure out the next hop toward the news.com server.
Network providers--called autonomous systems, or ASs--broadcast the ranges of IP addresses to which they'll provide access. One of the functions of the Internet Corporation for Assigned Names and Numbers is managing the master list of AS numbers, which it does by allocating large blocks of 1,000 or so at a time to regional address registries.
Kim Davies, ICANN's manager of route zone services, says ICANN isn't able to revoke the AS number of a misbehaving network provider. "It's best to think of them as similar to post codes or ZIP codes," Davies said. "We maintain a registry of them to ensure that they aren't conflicting."
If the address information provided by AS is reliable, all is well. But if an AS makes a false broadcast, because of a configuration mistake or for malicious reasons, all hell can break loose.
This is what happened with YouTube, which Pakistan's government ordered blocked because of offensive material, apparently a video depicting the cartoons about Muhammad that had been posted in a Danish newspaper. Some reports have said the video featured several minutes of a film made by Dutch politician Geert Wilders, an outspoken critic of Islam.
A spokesman for the Pakistani embassy said on Monday that the order to block access to YouTube came from the highest levels of the government. It would have been passed along to Pakistan's Electronic Media Regulatory Authority and then to Pakistan's telecom authority, the spokesman said, which in turn would have issued the formal order to the Internet providers.
Pakistan Telecom responded by broadcasting the false claim that it was the correct route for 256 addresses in YouTube's 208.65.153.0 network space. Because that was a more specific destination than the true broadcast from YouTube saying it was home to 1,024 computers, within a few minutes traffic started flowing to the wrong place.
A timeline created by Renesys, which provides real-time monitoring services, says that it took about 15 seconds for large Pacific-rim providers to direct YouTube.com traffic to the Pakistan ISP, and about 45 seconds for the central routers on much of the rest of the Internet to follow suit.
YouTube took countermeasures within minutes, first trying to reclaim its network by narrowing its 1,024 broadcast to 256 addresses. Eleven minutes later, YouTube added an even more specific additional broadcast claiming just 64 addresses--which, under the Border Gateway Protocol, is more specific and therefore should overrule the Pakistani one. Over two hours after the initial false broadcast, Pakistan Telecom finally stopped.
How could this have been prevented? First, Pakistan Telecom shouldn't have broadcast to the entire world that it was hosting YouTube's IP addresses. Second, Hong Kong-based PCCW could have recognized the broadcast as false and filtered it out.
An employee of PCCW, who wished to remain anonymous because he is not authorized to speak for the company, said that as soon as the false broadcast occurred, PCCW started receiving a flurry of phone calls from global ISPs wondering what had gone wrong. A YouTube representative also called.
Even Pakistan Telecom contacted PCCW. "I don't think they understood what was going on," the employee said. A spokesman for PCCW's U.S. operations, based in Herndon, Va., declined to provide details.
At the moment, large network providers tend to trust that other network providers are behaving reasonably--and aren't intentionally trying to hijack someone else's Internet addresses. And errors that do arise tend to be fixed quickly by manual intervention.
But as the number of suspicious broadcasts grows, and the potential for fraud increases, so does the justification for more aggressive countermeasures. (Besides, some government will eventually order its network providers to broadcast false information about the Internet addresses of "offensive" Web sites. We've already seen domain name blocking in Finland and Web page blocking in the United States, both supposedly enlightened Western democracies.)
One way to handle this is for network providers to be automatically notified when the virtual location of an Internet address changes, which is what some researchers have suggested in the form of a "hijack alert system." Another is to treat broadcasts with changes of addresses as suspicious for 24 hours and then accept them as normal. Simple filtering of broadcasts may not always work because some networks provide connectivity to customers with thousands of different routes.
Probably the most extensive countermeasure would be a technology like Secure BGP, which uses encryption to verify which network providers own Internet addresses and are authorized to broadcast changes. But Secure BGP has been around in one form or another form since 1998, and is still not a widely-used standard, mostly because it adds complexity and routers that understand will add additional cost.
At least that's been the conventional view. A high-profile incident like YouTube being knocked offline may accelerate this process, said Steven Bellovin of Columbia University. "I know there are serious deployment and operational issues," Bellovin said. "The question is this: When is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done."
News.com's Greg Sandoval contributed to this report.
Declan McCullagh, CNET News' chief political correspondent, chronicles the intersection of politics and technology. He has covered politics, technology, and Washington, D.C., for more than a decade, which has turned him into an iconoclast and a skeptic of anyone who says, "We oughta have a new federal law against this." E-mail Declan.
%20%7C%20News%20Blog%20-%20CNET%20News&srcUrl=http://news.cnet.com/8301-10784_3-9878655-7.html){kind=small}
There isn't enough research on stories now-a-days in the online journalist scene.
Its good to see some refreshing journalism again.
I've been reading Declan McCullagh over the years and he is without doubt one of, if not the best journalist who works with Cnet News.
When you see Declan McCullagh at the top of news articles on Cnet News, you know you're going to get a quality indepth read on whats going on.
His stories are particularly good on government issues and wiretapping.
Well done Cnet for original refreshing journalism, which lacks in the American-media scene all too often.
Usually I need to goto British news sites to get real news and real facts, but this proves me wrong that America's media outside of the NY Times can produce quality news.
There are issues so alarming, underlying this, and significant assumptions that are not being questioned.
I'll bet that no one in Pakistan thought this would happen, I'll bet they thought that no one outside of Pakistan would notice this at all.
There is a much easier way to cut of access, than this, the method chosen appears to be one of a very suspicous nature. A method that enables the local Pakistan NOC to find out who (in Pakistan) was connecting to YouTube, not just block it.
Your best bet for real news is still media sources outside the US, because ours is quite well owned and controlled....
"We've already seen domain name blocking in Finland and Web page blocking in the United States, both supposedly enlightened Western democracies.)"
just report the news, if I want snark I'll go to Huffington post or moveon.org
I don't think script kids can do a whole with this at the moment.
They would need to wait for the real hackers to release some kind of technique for them to use.
However, cyber terrorism is a possibility.
If a script kid gets a job at a NOC or SOC then there is real potential for damage to be caused.
Proper vetting of course is needed to stop the cyber terrorists getting jobs which work with technology and telecoms, where there is critical infrastructure at the hands of staff members.
Even human error can be a cyber terrorism, even if the error was unintentional.
It seems this was a "blue on blue" cyber attack...
Remember, your computers don't need to be connected to the internet for them to be exploited with zero-day code or a malicious user.
If a staff member can do badness by mistake or intentionally, to the end-user it doesn't matter how it happened, the result is still the same as a full scale malicious attack.
I'm sure Youtube don't care if it was a mistake, an outage is an outage and has the same affect as a malicious attack.
The hackers aren't going to go into the NOC /SOC centers, they are gonna pay the script kids to do it.
There is a pecking order that is maintained... when hackers want to target computers not connected to the internet, they will send in human resources to work at key jobs. Those human resources are the script kids, directed by the big hackers. The big hackers won't risk jail.
Remember folks, if you have computers in critical ifnrastructure buildings, they need to be patched from zero-day that appears on the security mailing lists, even if those computers will never be connected to the internet, because even with really good employee vetting and physical security checks into the building, there is always a chance the cyber terrorists will get in and get access to these not-connected-to-the-internet computers.
Yeah, that's how serious that threat actually was.
And now they seem sure every script kiddie with a cable modem will find a magical way to redirect all of the Web's traffic.
How did that saying go? "Fool me once, shame on...uh, won't get fooled again"?
- CNET is not a monolithic entity. We employ hundreds of writers and editors, each with his or her own view. There's no party line. (A few weeks ago, my colleague Michael Kanellos and I debated capitalism here on these pages.) So because one person was excited about pharming doesn't mean much.
- It was a quote. Did you actually read the full article?
Well. This is Internet's largest weakness. And I'm sure that countries in the future, in an event of a war will try to sabotage Internet for the country it is in war with and we will end of with several different Internets that cant connect. I'm sure this will be debated heavily over the coming weeks by media and security experts and be an example that maybe it's time to try to do something about this weakness.
My question is - can an individual do anything ? What if your ISP decides to hijack the IP address of one or other website ? Can an individual in Pakistan circumvent what Pakistan Telecom did ?
I know all hackers will now concentrate on how to replicate what Pakistan Telecom did, but I hope some IT wizard would come up with a way for individuals to fight back, and not having to rely on irresponsible businesses such as PCCW.
The below indicates that YouTube's 208.65.153.253 IP address is in or near San Jose, which makes sense.
eriador-cnet-cnwk:~ declan$ traceroute 208.65.153.253
traceroute to 208.65.153.253 (208.65.153.253), 64 hops max, 40 byte packets
[snip]
5 ggr6.n54ny.ip.att.net (12.122.86.101) 94.928 ms 93.830 ms 93.987 ms
6 64.212.107.97 (64.212.107.97) 93.317 ms 95.230 ms 98.577 ms
7 youtube-llc.po1.401.ar2.sjc2.gblx.net (64.212.108.162) 171.381 ms 184.317 ms 170.837 ms
8 youtube.com.hk (208.65.153.253) 170.291 ms 170.851 ms 171.321 ms
This would let you get around DNS spoofing by hackers or governments.
I am running on a negative amount of sleep, so I don't know if this is a good idea. Please comment.
I have a great deal of respect for a master counter fitter who can hand draw a $100 bill that will pass inspection. I have no respect for a 12 year old kid who has a color copier.
Not all criminals are created equal.
http://youtube.com/watch?v=5S3OA3nJRBQ
http://youtube.com/watch?v=HIHDqZLTK5Y
multiple "Danish Cartoon Parody" and "Geert" or "Forbidden Trailer" searchable.
http://thepiratebay.org/tor/4047508
http://thepiratebay.org/tor/4047509
We'll see how filterable a specific migratory stream / http url is... It takes a VERY large packet filter device to pull that scale of censorship off on multihoned isps.
Mirror FLV stream backup http://wikileaks.bluenorway.org etc
http://BlueNorway.Org
bluenorway@gmail.com
;)
the United Nations. This type of thing would become rampant.
2. BGP filters are common on smaller ASN's peering sessions, but on customers who are viewed as bigger due mainly to the number of prefixes and/or company size there is an assumption of clue fullness and implicit trust as you say and thus filters are often omitted. This is true with the big American users as well. PCCW buys service from Global Crossing as an example and nobody is complaining that Global crossing didn't filter PCCW, but since PCCW transits about 15,000 routes, this would be a bit odd (15,000 routes of the total 245,000 or so that make up the internet).
Is its proximity to gov intelligence installations a coincidence?
Is not this same mechanism able to snoop upon packets, by routing through 'forged routes', without interupting service?
Why would anyone do this to block a website when all that is needed is for the Pakistan telco to remove the route, and DNS entry of YouTube from its local NOC? (this is the safe easy way.)
Do we even know for sure that the supposed material being censored was what was claimed....cartoons just don't seem to justify such a drastic action?
There is little argument that taking over hosting of those addresses was done just to cut off local access, its a strong indicator that it was done to find exactly who (IP and MAC addresses) was attempting access to YouTube at that time.
Perhaps to trace/prevent uploading of?????
This again is a very strong argument for physical addressing and routing using GPS Signals, which cannot be forged, nor duplicated. (without fake GPS Satelites anyway).
Indiviuals cannot do much to affect the inetent, but when network engineers are ordered to do stupid stuff by the goverment, what can you do. Anybody with an authorized AS number and a BGP router can affect the stablility of the Interent.
Who is upset with them?
I learned a lot from it and the links referenced useful sites. To echo another poster, it should be an article, not just a blog post.
- Pakistan Telecom
-
by TheGreatOn
March 13, 2008 1:16 PM PDT
- Pakistan Telecom is 26% privately owned by Etisalat which also controls the company, so although you would be accurate to say it is a state-owned company I don't think it paints an accurate picture.
-
Reply to this comment
-
(32 Comments)