A real-world Internet test reveals that "intelligent" routing of peer-to-peer traffic can drastically reduce network utilization and speed up downloads for subscribers, according to a new study.
Verizon Communications, which participated in the study headed by researchers at Yale University, plans to release the data on Friday at the Distributed Computing Industry Association's P2P Market Conference in New York City.
Using network topology data from Verizon and Telefonica, Yale University tested a software enhancement to the peer-to-peer protocol that it developed with software developer Pando Networks.
What the researchers discovered was that when using the so-called P4P software they were able to reduce the impact of peer-to-peer traffic on Verizon's network by more than 50 percent. This is significant because peer-to-peer traffic makes up roughly half of all traffic traveling over Verizon's network.
The P2P protocol, which is used to distribute large data files, works by requesting pieces of a single file from different hosts all over the Internet. The technology has become popular for distributing high-definition video.
But applications that use P2P eat up a lot of bandwidth, which some service providers say is a problem. Cable operator Comcast has slowed down certain kinds of peer-to-peer traffic in an effort to manage its network. And Time Warner Cable is experimenting with a tiered usage model to deter people from sharing P2P files.
Traditionally, the P2P protocol has requested bits and pieces of content randomly, without considering the physical location of the data. This often results in some pieces of the content traveling over long distances across the network. For example, a user in New Jersey downloading a movie might get some bits of the file from New York and others from China or California.
The P4P software enhancements add intelligence to this process so that the bits are served from local hosts.
Douglas Pasko is Verizon senior technologist and co-chair of the P4P Working Group, which was formed by Verizon, Pando Networks, and the university to develop P4P. He said that when the P4P software was used on the Verizon network it found that 58 percent of its peer-to-peer network traffic stayed local. Using regular P2P technology, only 6 percent of the traffic stayed local.
Reducing the number of hops is key
Pasko said that keeping the traffic local is important because every link that a bit passes through costs the operator something. This means that if a Fios subscriber in New Jersey can get bits of content from Verizon customers in New York City instead of getting them from Singapore or Taiwan, Verizon can save money.
The key is reducing the number of routers or hops the traffic has to go through to get to its destination. On average, Pasko said that regular P2P traffic makes 5.5 hops to get its destination. Using the P4P protocol, those same files took an average of 0.89 hops.
Reducing hops means that Verizon can cut its network costs. Exactly how much the company saves depends on the individual links, but Pasko said the savings are significant.
Verizon broadband subscribers also saw a benefit when the P4P protocol was used. Customers using Verizon's all-fiber network called Fios saw movies downloading on average twice as fast as when they used the traditional P2P software. Some customers saw as much as a 6x improvement in download speeds, Pasko said.
For customers on regular DSL service, the improvement in download speeds wasn't as great because these customers don't have high bandwidth connections anyway.
This real-world field trial validates the value of P2P content providers working closely with Internet service providers to provide the most efficient service for customers, Pasko said. There are already 50 members in the DCIA's P4P Working Group, including some cable operators, such as Comcast, Cablevision, and Time Warner, he said.
"We hope this shows that using P2P in an intelligent way can benefit everyone," he said. "It allows us to use fewer resources on our network and get better performance for our customers."
Continuing an aggressive campaign to defend his copyrights, pop star Prince is preparing to file lawsuits within the next few days in three countries--including the United States--against The Pirate Bay, CNET News.com has learned.
John Giacobbi
(Credit: Web Sheriff)One of the world's best-known BitTorrent indexing sites, The Pirate Bay has defiantly linked to pirated copies of films, TV shows, music videos, and other content while often boasting that it ignores Hollywood's requests to remove them. The Pirate Bay does not host any unauthorized content, but the service is internationally famous for being a highly effective file-sharing tool.
Prince will file similar suits against The Pirate Bay in the U.S., France, a country with laws favorable to copyright owners, and Sweden, where The Pirate Bay is based. In addition, Prince is preparing to take civil action against companies that advertise on The Pirate Bay, many of which are headquartered in Israel, according to John Giacobbi, Web Sheriff's president.
Prince has hired Giacobbi and Web Sheriff, a service that protects copyright materials from Internet piracy, to coordinate the legal challenges against The Pirate Bay and others who the singer believes has violated his copyright.
Giacobbi said Web Sheriff is also helping to launch an investigation into The Pirate Bay's off-shore connections to determine whether the company is compliant with Swedish and international income and corporation tax laws.
The Pirate Bay has already weathered several attempts by the governments of Sweden and the United States to shut down the site. Yet, this is likely the largest civil challenge the Web site has ever faced.
At the core of Prince's lawsuits are his claims that the three founders of The Pirate Bay are profiting from the work of artists without compensating them. The Pirate Bay earns $70,000 a month in advertising revenue, Giacobbi alleged. The site's founders have previously denied that the operation makes money.
None of the three founders of The Pirate Bay could be reached for comment.
The Pirate Bay Founders, Peter Sunde, Fredrik Neij, Gottfrid Svartholm
(Credit: Pontus Alexander/Fabian Landgren)
Prince, who Giacobbi said has the backing of the International Federation of the Phonographic Industry, or IFPI, the group that represents the recording industry worldwide, is only adding to The Pirate Bay's legal troubles. The site founders also face criminal charges, according to a story published Thursday on the blog TorrentFreak.
A prosecutor in Sweden announced that he plans to press charges against five people involved with The Pirate Bay before January 31, 2008, the blog reported. The five are being accused of infringing on intellectual property.
The copyright battle that Prince has waged the past two months has not been without its costs. He was widely criticized this week when three unauthorized fan sites accused him of trying to violate their free speech rights when his handlers demanded that they remove several photos of him.
It was widely reported this week that Prince had begun suing fans. His representatives denied this.
"Prince is not suing his fans, is not looking to penalize fans and nor is he looking to inhibiting freedom of speech in any way," said AEG, Prince's promoter.
Prince began making headlines in September after lashing out against sites he believed were violating his intellectual-property rights.
In September, the singer said he planned to take legal action against The Pirate Bay, YouTube, and eBay. As of Friday, Prince's lawsuits appeared to be solely targeted at The Pirate Bay.
By suing The Pirate Bay in three different countries, Prince is hoping to put financial pressure on the service, Giacobbi said. Copyright laws in the United States and France would also make it nearly impossible for a site like The Pirate Bay to triumph, he claimed.
"There is no way that they will have any defense because it's blatant piracy," Giacobbi said. "They'll either have to come out and fight or just try and ignore it. In that case, we're going to win a default judgment against them. This could be a ticking time bomb for them. They can't outrun this. We are very confident."
While a class action lawsuit is definitely one way to get Comcast to behave, another perhaps more productive way to do so is to have politicians step in and regulate.
On Tuesday, I discussed the issue of Comcast's anti-BitTorrent "network management" with Rep. Rick Boucher, D-Va., who is a strong supporter of consumer rights and has led the battle to undo the damage caused by the Digital Millennium Copyright Act, or DMCA.
He was named Politician of the Year for 2006 by Library Journal, largely due to his efforts to protect the fair-use doctrine and expand Internet technologies to rural areas.
"Comcast has made a major mistake in attempting to hinder peer-to-peer file sharing as an aspect of its network management," Boucher said. "The inability of customers to (share files) significantly diminishes their ability to utilize the Internet for one of its most important applications, which is user-to-user content." He also noted that "file sharing is already being used for a wide variety of perfectly lawful and appropriate applications."
Net neutrality nightmare ad
(Credit: Rikomatic / Flickr)Discussing the realities of limited resources that the company faces, Boucher said, "Comcast obviously needs to engage in some aspect of network management. The company has limited bandwidth, and there are times when there is more demand for service than the infrastructure can support." However, the congressman stressed that "(the) management needs to occur in a more evenhanded way" and that "(Comcast) should not engage in a blanket disqualification of any category of lawful applications."
Until last month, the opponents of Net neutrality were doing just great. The issue, which had become one of national importance in 2006, had shrunk to a mere footnote in the annals of tech policy history.
CNET News.com's Declan McCullagh wrote about the death of Net neutrality last month, stating that "(the issue) went from being the political equivalent of a first-run Broadway show, with accompanying street protests and high-profile votes in Congress, to a third-rate performance with no budget and slumping attendance."
Luckily for fans of a free Internet, the telecommunications companies are extremely shortsighted. Thanks to a number of their boneheaded moves, Net neutrality has gone from being all but dead to a major news story--all in just a matter of weeks.
Respect BitTorrent
(Credit: Hetemeel.com)The first company to breathe life back into was Verizon Wireless, which decided in late September to block a SMS text message campaign by a pro-choice group.
Within one day (and after having its censorship techniques compared to those of the Chinese government in a New York Times article), Verizon quickly flip-flopped.
While Verizon should be commended for realizing that it needed to do the right thing, and quickly, the damage was already done. Net neutrality was back on the tech policy radar.
In mid-August, user reports began to surface alleging that Comcast was filtering the BitTorrent connections of its broadband cable customers.
While the story got a bit of press in some tech news outlets, it was ignored by the national media, primarily due to the flat-out denials issued by Comcast.
Fast-forward one month. This past Friday, the Associated Press and the Electronic Frontier Foundation both released investigative reports, documenting the fact that Comcast is actively engaged in anti-BitTorrent behavior.
In spite of Comcast's best efforts to yet again spin the story, the truth seems to have come out, and major news outlets have picked it up: Comcast is actively sending out false data onto its network, which impersonates its customers' computers and deceitfully convinces them to terminate BitTorrent connections. Not only does the company have a major PR disaster on its hands, but it has in a matter of days become the poster child for Net neutrality.
Comcast's name is surely to come up in any future discussion of Net neutrality - which has gone from a theoretical "what if companies did this kind of thing" debate to something more akin to "do you want every Internet company to start acting like Comcast?"
In my blog post on the subject this past Tuesday, I explored some of the potential legal risks that Comcast faces. I spoke to the Electronic Frontier Foundation's Fred von Lohmann, who revealed that "(the EFF has) already been contacted by attorneys, who are considering legal action against Comcast."
I asked Boucher what he would do if Comcast stuck to its guns and kept discriminating against BitTorrent. In particular, I asked him if he would propose legislation compelling the company to treat all traffic fairly.
Unfortunately for fans of Net neutrality, the congressman said he was not ready to go down this path and instead stressed market-based methods of fixing the problems. Instead of tinkering with packets, the congressman said that in the short term, Comcast should "simply tier their offerings and engage in a pricing structure that allocates more bandwidth to those who pay more, and less to those who pay less."
However, he said "the long-term answer is to deploy more capacity. That is what municipal broadband and other telecom companies are doing. Ultimately, the cable companies will have to deploy fiber to the house."
Columbia University cyberlaw professor Tim Wu recently pointed to a historical analogy regarding Verizon's SMS fiasco. He told The New York Times that in the 19th century, the telegraph company Western Union engaged in discrimination based on the political views of people who sought to send telegrams.
"One of the eventual reactions was the common-carrier rule," Wu said, which required telegraph and then phone companies to accept communications from all speakers on all topics.
Someone who believes in a market-based solution to this problem is Jim Harper, director of information policy studies at the Cato Institute. In a phone interview on Tuesday, Harper noted that one of the main problems is Comcast's lack of transparency--something that can be seen through the fact that no one yet knows, exactly, what Comcast is doing. He said "Comcast seems to lack the capacity to communicate terribly well. They should fix that."
Harper believes that competition is the key to fixing the problem and that if customers truly care about the issue, they will choose another Internet service provider that is more BitTorrent-friendly. He did, however, note that without transparency, "consumers cannot make smart choices."
He also rejected calls for Net neutrality regulation, stating that he believes that the problem can be fixed by promoting competition. While acknowledging that the state of the market is far from competitive for many rural consumers, he noted that customers in bigger markets often have the choice between multiple phone, cable and wireless companies.
Harper said that instead of "dividing the current pie through regulation, it is far better to grow the pie" by encouraging new companies to offer service. One example of this, he said, was allocations of additional spectrum to broadband, such as the upcoming 700MHz auction.
Finally, Harper was somewhat skeptical of the importance of this issue to most consumers. He noted that Comcast is not blocking BitTorrent downloads but rather only the sharing of files--something that is not viable to most users. "If customers don't care enough to vote with their feet" he asked, "then how important is it, anyway?"
It's long been an open secret that many major telecommunications companies, including Internet service providers (ISPs) and cellular data providers, impose specific limits on the volume and type of bandwidth consumed by their customers.
"Open" in the sense that these companies almost universally reserve the right to impose such limits, and occasionally make public statements defending their right to do so. "Secret" because the companies rarely reveal their specific limits, and because it seems like these limits are constantly being rediscovered by people who ought to know better.
It reminds me of people who claim to be shocked by new evidence of pork-barrel politics even though the practice has been common throughout history (though some politicians do refrain).
Two such shocking cases are before us in the news this week.
First, Comcast admitted it uses traffic shaping to reduce the bandwidth demands of peer-to-peer file-sharing applications such as BitTorrent.
Next, Verizon Wireless settled a dispute with the Attorney General of New York state stemming from its use of the word "unlimited" to describe its wireless Internet plans, which were very definitely limited by secret company policies to a total of five gigabytes per month.
In both cases, the primary issue is fairness...and the contradictory definitions of fairness held by customers and their service providers.
For ordinary customers, fairness means being able to use all the bandwidth they're paying for, continuously, forever, with whatever software they like. Providers apparently want to define fairness as satisfying the maximum number of customers while still making a profit.
I have to take the customer's side on the definition, but you can't argue with the business goal. Without profit, there's no business. Some ISPs don't care about profit, of course--chiefly the ones in foreign countries where the Internet has been subsidized by government for political reasons. But here in the US, Internet services have to make a profit, and that means bandwidth supply can never meet bandwidth demand.
On one hand, ISPs could build out their networks to provide the maximum rated bandwidth to all subscribers at once...but that would be extraordinarily expensive, and wasteful because that kind of load would never be seen in the real world. On the other hand, they could promise the lower throughput they truly can guarantee, then deliver more when possible...but that would look terrible as a marketing campaign. Verizon's 5GB limit, for example, works out to about 2 kilobits per second. Can you imagine advertising "2-kilobit high-speed Internet!"? Nope. Supply and demand isn't just a good idea--it's the law.
So ISPs use what's called "statistical multiplexing"--sharing bandwidth among multiple users on the theory that they won't all use it at once. A cable-modem service such as Comcast's might share some 30 megabits/s of bandwidth among dozens to hundreds of subscribers in a single neighborhood.
Although a mere half-dozen users could overload this shared channel, in practice, the natural load-sharing behavior of Internet protocols provides a reasonable experience for all the users online at any given moment.
Except when it doesn't. An active BitTorrent user might have dozens or hundreds of transfers in progress--multiple files, each with multiple partial downloads in progress. Each transfer competes for a share of the channel's total bandwidth, so a handful of BitTorrent users on one cable-modem service could consume the vast majority of the available bandwidth.
This gives us a third definition of fairness, the BitTorrent definition: fairness is being able to take everything you can get without regard to other customers on the same service.
So is it more unfair for Comcast to cut back on BitTorrent traffic, or for BitTorrent users to exploit the Internet's load-sharing behavior? I have to take Comcast's side on this one.
I was writing a reply to a badly-written article and subsequent ignorant comments about the Comcast situation over on Engadget earlier today when I realized this situation creates a market opportunity for some software company. So I figured I'd write this instead of posting the reply there.
To summarize: although I suspect most BitTorrent traffic consists of pirated software, music, TV shows, and movies, there's also some important, legitimate content on BitTorrent--Linux distributions, collections of classic e-books in the public domain, Linux distributions...wait, I mentioned that already. Actually, there probably isn't that much legitimate BitTorrent activity. But however much there is, it deserves to pass unmolested on Comcast and other Internet services.
It's unfortunate that these files are being subjected to Comcast's traffic shaping, but that's what happens when people put legitimate content into a distribution channel designed and optimized to facilitate piracy.
So clearly we need a separate public-access peer-to-peer system. How would it differ from BitTorrent? Well, the content would have to be legitimate, and probably so. That means a central authority and a master list of authorized content.
Of course, this is basically the Joost business plan. Joost is a peer-to-peer file-sharing service for video. It seems to me that either Joost ought to expand the concept to include other kinds of content, or someone else ought to get into the business. It'll have to be ad-supported, I think, because if anyone has to pay for the service they'll probably just keep using BitTorrent instead.
Such a service should also get support from ISPs since it would actually reduce their overall bandwidth load for legitimate file sharing. Peer-to-peer distribution is very efficient--that's why Joost uses it--so the providers could leave this new service unfiltered while BitTorrent gets quite legitimately squeezed out of operation.
No doubt such a plan would be highly unpopular with active BitTorrent users, but if that's the price of keeping ISPs profitable--and honest in their marketing policies-- I think it's a reasonable price to pay.
Peer-to-peer company BitTorrent is set to announce on Tuesday morning the availability of a new enterprise content delivery product, BitTorrent DNA. Designed for companies that use streaming video, large downloads or games over the Web, the launch of BitTorrent DNA marks yet another conscious move by the San Francisco-based software brand to move beyond its roots as the creator of file-sharing protocol that became nearly synonymous with digital piracy over the past few years.
BitTorrent described the new BitTorrent DNA product in a statement as "the ideal solution for publishers seeking ways to overcome the obstacles associated with centralized content delivery, such as slow downloads, choppy video streams, and inefficient use of network infrastructure." The inaugural client for the new content delivery network (CDN) is online video start-up Brightcove, which powers a number of large companies' broadband media operations.
BitTorrent DNA will be used to "accelerate" the delivery of the video hosted on Brightcove's platform.
With the rise of online video and large-scale media downloads, content delivery has become a crowded niche in the market. BitTorrent DNA will square off with industry leaders like Akamai Technologies--the force behind CBS' video distribution network as well as a host of others. BitTorrent is hoping, however, that its massive following (150 million downloads of its client, according to the company) will help give it an edge.
In addition, the peer-to-peer format has become increasingly popular in the streaming video space, with recent entries like Joost and Babelgum touting P2P technology as the backbone for their professional-quality video content.
In February, BitTorrent announced that it was creating a digital download store that would use that robust user base as a way to legally transfer large movies, games and other files. The company has also forged alliances with major movie studios for legal film downloads.
Meanwhile, the exhaustive battle over online piracy wages on.
If you use torrents frequently, then you'll remember December 7, 2006, as a day that will live on in infamy, a day that teh Interwebs broke in half--just a little. That was the day that BitTorrent, Bram Cohen's torrent progenitor, announced it had bought µTorrent, the free-but-closed-source torrent client that showcased innovative features in a surprisingly lightweight app.
Now, two years since their last upgrade, BitTorrent has released its first version that combines µTorrent code with its own open-source base, and µTorrent has also come out with a minor-point upgrade. Confused? Read on.
... Read more
The battle for net neutrality might be in its infancy, and already we're seeing casualties. There have been murmurs that Comcast has been capping bandwidth usage on its all-you-can eat high-speed subscribers that have simply used more of their connection than Comcast is comfortable with, along with evidence the ISP has been monkeying with people's BitTorrent usage. Last week GameDaily BIZ got in touch with Charlie Douglas, a spokesperson for Comcast Corporation who confirmed that the company was indeed capping monthly downloads of its "excessive" users.
The actual ballpark figure Douglas gave GameDaily BIZ was "30,000 songs, 250,000 pictures or 13 million emails in a month." This number might not seem like a more than you could use, but as Scott Gilbertson notes on Wired's Compiler Blog, that number amounts to about 90GB, assuming you're going by the typical industry-standard song size of about 3MB. Clearly this isn't a normal usage number, as most people aren't downloading 30,000 music files, but the fact that Comcast isn't providing a clear number that's been set in stone makes this mysterious cap all the more troublesome, especially for users of Web based back up and file transfer tools.
Related: Will Comcast cut you off?
Disclaimer: I am not a lawyer. I'm a cybersecurity Ph.D. student and take classes in the Indiana University law school, but this in no way makes me a legal expert. CaveatlLector.
Within the last few weeks, there have been a number of reports by Comcast customers claiming that their BitTorrent downloads and uploads have been capped--or worse, blocked. TorrentFreak recently reported that Comcast, a major U.S. cable company, is using an application from Sandvine to throttle such connections.
Comcast LolCat
(Credit: Comcast and LolCat Buildr)Many ISPs routinely filter the traffic on their networks. Many forbid customers from running e-mail servers or Web servers, and when the ISP detects that a customer's computer has been hacked, they often sever the Internet connection until the machine has been patched. Thus, the fact that a major ISP is now filtering yet another class of Internet traffic should not be major news--except for two factors: BitTorrent traffic accounts for upwards of 25 percent of U.S. Internet traffic, and the techniques used by Comcast are essentially the same as those used by the Great Firewall of China.
Before we get deeply into this issue, let us step back for a brief and high-level lesson in TCP/IP and Internet filtering technologies. Most Internet applications communicate via TCP, a protocol that uses a three-way handshake to establish a connection.
The very first step in a three-way handshake involves the client sending a SYN packet to the receiving party. Modern firewalls block this packet for banned types of traffic--that is, they prevent the recipient from receiving it, and as such, the connection can never be established. Your home firewall does this, as well as those used by Comcast and other ISPs to prevent you from sending millions of e-mail spam messages from their network.
Assuming that the SYN packet goes through, the three-way handshake is allowed to happen, then the two hosts will be able to begin communicating. Your ISP can still kill the connection later, should it wish to, merely by blocking the transmission of future packets.
According to TorrentFreak, Comcast is not doing this. They are instead sending a reset (or RST) packet to the Comcast customer, pretending to be from the host at the end of the BitTorrent connection. This RST packet is the TCP equivalent of stating "I don't want to talk to you anymore, please terminate the connection." It is extremely important to note that when Comcast creates and sends this packet, it does not identify itself as the the source of packet, but instead impersonates one of the parties involved in the BitTorrent connection. This is where things get rather shady.
Last year, researchers from Cambridge University analyzed the Great Firewall of China and found that it used falsified RST packets to terminate connections that matched keyword filters. They were able to determine that users could evade the Chinese government's censorship system by ignoring these reset packets.
Ok, so the Chinese government and Comcast are using the same censorship techniques. Why should we care? The Chinese government doesn't have to pay attention to U.S. law, but Comcast, being a U.S. company, does.
Many states make it illegal for an individual to impersonate another individual. New York, a state notorious for its aggressive pro-consumer office of the Attorney General, makes it a crime for someone to "[impersonate] another and [do] an act in such assumed character with intent to obtain a benefit or to injure or defraud another." (See: NY Sec. 190.25: Criminal impersonation in the second degree). I do not believe that it would be too difficult to prove that Comcast obtains a benefit by impersonating others to eliminate or reduce BitTorrent traffic. Less torrent data flowing over their network will lead to an overall reduction in their bandwidth bill, and thus a huge cost savings.
New York is not the only state with such a law. Several other states including Connecticut and Alabama have similar laws on the books. Should any state AG's office decide to go after Comcast, it is quite possible that Comcast could be looking at a world of regulatory pain.
Comcast is perfectly within its right to filter the Internet traffic that flows over its network. What it is not entitled to do is to impersonate its customers and other users, in order to make that filtering happen. Dropping packets is perfectly OK, while falsifying sender information in packet headers is not.
Yeah, yeah, I know. Harry Potter this, Harry Potter that, OMG-does-Harry-die, OMG-OMG-it's-on-BitTorrent... We're so ready to move on to the next cultural obsession from across the pond. Regardless, here's some video from Friday night of some of the people who showed up at the official New York book launch event hosted by Scholastic, the publisher that handles Harry Potter in the States.
The wizardly confab was held in New York City's shopper-friendly SoHo neighborhood, on a stretch of Mercer St. between Prince St. and Spring St. Conveniently, not only is Mercer St. right around the corner from the Scholastic bookstore, but it also happens to be paved with cobblestone--a rarity in Manhattan. Folks, welcome to Diagon Alley.
P.S.: In case you were wondering, this video is spoiler-free, unless you consider weekend book sales data to be spoilers.
- prev
- 1
- next

