A group of researchers have found a way to quickly transfer huge amounts of data over computer networks, paving the way for greater global scientific cooperation and future commercial uses.
By shuttling data at a sustained rate of 101 gigabits--the equivalent of three DVD movies--per second between Pittsburgh and Los Angeles, the so-called High Energy Physics team shattered a record for data transfer and won the Supercomputing Bandwidth Challenge, a contest geared toward improving network connection speed for grid computing.
The team sustained the 101-gigabit speed for only a few minutes during the 90-minute demonstration, peaking at just above the 101-gigabit mark. But "we are confident if we did it again...we would sustain the 100+gbps throughput for hours," Harvey Newman, professor of physics at the California Institute of Technology and the head of the team, wrote in an e-mail. Transfer rates of 130gbps to 140gbps are likely possible, he added.
The research also points the way toward future applications for much faster transfers of audio, video and other data. "There are also profound implications for how we could integrate information sharing and on-demand audiovisual collaboration in our daily lives, with a scale and quality previously unimaginable," Newman said.
The team is made up of computer scientists, physicists and network engineers from Caltech, Fermilab, CERN, the University of Manchester, and universities from Korea and Brazil, among other places.
The old data transfer record, set by the same group a year ago, was 23.2gbps, less than a quarter of the current record. The amount of data transferred under the new record is also greater than the sum of all of the other marks in the contest in the previous two years combined.
Put another way, the data transfer speed is equivalent to transmitting the entire Library of Congress in 15 minutes. On the Internet, the record for data transfer is about 4.23gbps, while the record on Internet2 stands at 6.63gbps.
The data transfer rates were achieved in part though the Fast TCP protocol developed by Caltech professor Steven Low, which prevents congestion better than standard TCP. Standard TCP gauges congestion by the rate that packets of data get dropped. FAST TCP observes the delay that packets experience as they travel through the network.
This technique "provides a more accurate and more timely measure of congestion than packet drops--i.e., the traffic source can react to congestion before it builds to such a severe level that buffers at routers overflow and packets are dropped," professor Low wrote in an e-mail interview.
Greasing the skids A beefed up hardware infrastructure--including several 10-gigabit links, four dedicated wavelengths of
National LambdaRail, an all-optical network that links U.S. universities, Web services software and a vast array of other technology--also helped boost overall speeds.
The goal of the experiment is to create technology that will enable physicists across the globe to cooperate on massive, data-intensive projects that will involve computers located around the world.
CERN, for instance, will begin to conduct experiments in 2007 to search for Higgs particles, believed to be responsible for mass in the universe and other phenomena. The search will ultimately involve more than 2,000 scientists from 160 institutions exchanging terabyte-size data samples in an effort to look for unusual particle interactions.
Because many users will make requests worldwide for data, the transfer of massive files will have to occur within a few hours, not days. Data from the project will generate several petabytes of data fairly rapidly, researchers on the project said.
Besides particle physics, the networking technology will benefit researchers in bioinformatics, astronomy, global-climate modeling and geosciences.
Reducing network latency on global computing is also the focus of PlanetLab, an initiative led by Princeton University, Intel and the University of California at Berkeley. Others are looking at ways to create an Internet link to Mars.
To demonstrate its technology, the group also transferred simulated physics data to CERN, the University of Florida, Fermilab, Caltech, U.C. San Diego and Brazil for processing. The results were then aggregated in Pittsburgh and transformed into a visual display of the data. In another demonstration, the organization transferred large data sets between Pittsburgh and Manchester.
Private companies that contributed to the project include Cisco Systems, Hewlett-Packard and Newisys, which makes servers based on Advanced Micro Devices' Opteron processor.
I can't do the math in my head but. The Gigabit speed they are talking about needs to be converted into the Gigabytes they are actually moving. It's 8 bits to a byte so assuming the math is correct it would be approx 3 DVD's worth of data.
Even if it's only 3 dvd's per second, what home user can supply that kind of data? Even linked raptor drives can only manage bursts of around 200kbytes (1.6 Megabits) per second, compared to the claimed speed of 100,000 Megabits per second. I don't believe even the fastest home memory can run anywhere near such fast speeds. I'm wondering what was used to supply so much data, so fast.
True, the average home user could not supply 3 DVDs worth of data every second, but how about a neighborhood of 10,000 people connected by one of those lines?
Okay, I just learned that you can get 16x dvd readers, but the numbers are still well within an order of magnitude ... this was just for fun
-Network Bandwidth: 101Gb/s = 12.625GB/s -Mapquest Distance for travel from CMU to Caltech: 2426.95 miles -Mapquest Estimated Automotive Travel Time: 36 hours, 39 minutes -Maximum Capacity of a Dual Layer/Dual Side DVD: 17.1 GB -DVD Dimensions: Width = 1.2mm, Diameter = 120mm -The most common data format on CD holds 650 megabytes of data - about 12 billion bytes per pound weight Since a DVD holds a 1024MB * 17.1 = 17510.4 MB 17510.4 MB / 650 MB = 26.94 So, Dual Layer Dual Side DVD's have a (12 GB/lb) * 26.94 = 323.27 GB/lb data density -A Dodge Sprinter 3500 High Roof Van (a very large and popular commercial van) can carry a maximum 4769 lb of payload Thus, carrying Dual Layer DVD's, it can haul, (323.27 GB/lb * 4769 lb) = 1,541,674.63 GB of data -In 36hours, 39 Minutes (131940 seconds) the Network would move 1,665,742.5 GB of Data from CMU to Caltech
So, the new network technology can move 124,067.87 GB more data from Pittsburgh, PA to Pasadena, CA than the largest Dodge Sprinter Van full of dual layer DVDs in the same time period.
However, on a 10 mile trip across town, the Dodge Sprinter has about 250 times more bandwidth than the network, but the fastest DVD readers (12x) only transfer 5.4MB/s.
With 1,578,674,821.12 MB of Data on the DVD's and the necessity of flipping the DVD's over (at least 10 sec on each of the 90,157 dvds) since they're dual sided, it would take at least 901,564.1s x 3 (flip time ,load time, unload time) + 292347189.s (data transfer time) = 2,704,962s (flip time ,load time, unload time) + 292347189.s (data transfer time) = 295051881s = 81958.6 hours or 9.36 years to transfer the DVD's into RAM once they arrived.
If you had 90,157 DVD Drives it would still take almost an hour to transfer the data to RAM. Assuming one person could possibly load approximately 20 drives by themselves you'd need a room large enough to house a staff of approximately 4500 people and the 7.24" x 1.65" x 5.83" = 69.64 cubic inch per drive = 6279000.49 cubic inch = 73,604 cubic feet worth of drives arranged in a fashion that allows for easy access to the front of the drives.
I'm not even going to get into the time it would take to load and unload the DVD's. Another interesting number to chew on is that a Dual Layer DVD written at full capacity on both sides in the fastest available writeable format takes (@ the industry maximum of 2.4x = 1.085 MB/s) 5.25 Hours to write. If the discs are already written, this doesn't matter so much though :)
0.1 terabit is still a great achievement, and worth talking about, but this article gets one thing completely wrong -- it claims that the Caltech team "won" the bandwidth challenge. They didn't. Multiple awards were given, but if you have to say that one team "won", it was San Diego Supercomputing Center, with less bandwidth, because they had a real application (as opposed to iperf).
Web giant is spending $120 million to beef up its Mountain View, Calif., headquarters, according to filings with the city reviewed by the San Jose Mercury News.
The Samsung Galaxy Mini 2 S6500 could make its debut at the Mobile World Congress in Barcelona later this month, according to a leaked promotional image.
MIT creates a simulation to celebrate the 50th anniversary of Spacewar. A relic of the early days of minicomputers, it was one of the first computer video games and set the stage for many others, including Asteroids.
so isnt that like 10 or 11 dvds?
Imagine if the piracy community got their hands on those speeds.
-Network Bandwidth: 101Gb/s = 12.625GB/s -Mapquest Distance for travel from CMU to Caltech: 2426.95 miles -Mapquest Estimated Automotive Travel Time: 36 hours, 39 minutes -Maximum Capacity of a Dual Layer/Dual Side DVD: 17.1 GB -DVD Dimensions: Width = 1.2mm, Diameter = 120mm -The most common data format on CD holds 650 megabytes of data - about 12 billion bytes per pound weight
Since a DVD holds a 1024MB * 17.1 = 17510.4 MB
17510.4 MB / 650 MB = 26.94
So, Dual Layer Dual Side DVD's have a (12 GB/lb) * 26.94
= 323.27 GB/lb data density
-A Dodge Sprinter 3500 High Roof Van (a very large and popular commercial
van) can carry a maximum 4769 lb of payload
Thus, carrying Dual Layer DVD's, it can haul,
(323.27 GB/lb * 4769 lb) = 1,541,674.63 GB of data -In 36hours, 39 Minutes (131940 seconds)
the Network would move 1,665,742.5 GB of Data from CMU to Caltech
So, the new network technology can move 124,067.87 GB more data from Pittsburgh, PA to Pasadena, CA than the largest Dodge Sprinter Van full of dual layer DVDs in the same time period.
However, on a 10 mile trip across town, the Dodge Sprinter has about 250 times more bandwidth than the network, but the fastest DVD readers (12x) only transfer 5.4MB/s.
With 1,578,674,821.12 MB of Data on the DVD's and the necessity of flipping the DVD's over (at least 10 sec on each of the 90,157 dvds) since they're dual sided, it would take at least
901,564.1s x 3 (flip time ,load time, unload time) + 292347189.s (data transfer time) = 2,704,962s (flip time ,load time, unload time) + 292347189.s (data transfer time) = 295051881s
= 81958.6 hours or 9.36 years to transfer the DVD's into RAM once they arrived.
If you had 90,157 DVD Drives it would still take almost an hour to transfer the data to RAM. Assuming one person could possibly load approximately 20 drives by themselves you'd need a room large enough to house a staff of approximately 4500 people and the 7.24" x 1.65" x 5.83" = 69.64 cubic inch per drive = 6279000.49 cubic inch = 73,604 cubic feet worth of drives arranged in a fashion that allows for easy access to the front of the drives.
I'm not even going to get into the time it would take to load and unload the DVD's. Another interesting number to chew on is that a Dual Layer DVD written at full capacity on both sides in the fastest available writeable format takes (@ the industry maximum of 2.4x = 1.085 MB/s) 5.25 Hours to write. If the discs are already written, this doesn't matter so much though
:)
I got bored.
See:
<a class="jive-link-external" href="http://www.sdsc.edu/Press/2004/11/111204_SC04.html" target="_newWindow">http://www.sdsc.edu/Press/2004/11/111204_SC04.html</a>
for their press release to this effect.