Researchers tout 'wimpy nodes' for Net computing

Mainstream servers are growing increasingly brawny with multicore processors and tremendous memory capacity, but researchers at Carnegie Mellon University and Intel Labs Pittsburgh think 98-pound weaklings of the computing world might be better suited for many of the jobs on the Internet today.

This first-generation FAWN system has an array of boards, each with its own processor, flash memory card, and network connection.

(Credit: Carnegie Mellon University)

The alternative the researchers advocate is named FAWN, short for Fast Array of Wimpy Nodes. It's described in a paper just presented at the Symposium on Operating Systems Principles.

In short, the researchers believe some work can be managed with lower expense and lower power consumption using a cluster of servers built with lower-end processors and flash memory than with a general-purpose server. And these days, with green technology in vogue and power costs no longer an afterthought, efficient computing is a big deal.

"We were looking at efficiency at sub-maximum load. We realized the same techniques could serve high loads more efficiently as well," said David Andersen, the Carnegie Mellon assistant professor of computer science who helped lead the project.

It's not just academic work. Google, Intel, and NetApp are helping to fund the project, and the researchers are talking to Facebook, too. "We want to understand their challenges," Andersen said.

Flash memory and low-end processors
Today's servers store data on a combination of slow but capacious hard drives and fast but expensive memory. In contrast, FAWN relies on flash memory, storage technology with a price and performance in between the two. Its CompactFlash memory cards are more ordinarily found in higher-end digital cameras, but they are relatively easily repurposed since they communicate with a conventional hard drive interface.

In addition, each computing node in the FAWN system doesn't use powerful Xeon or Athlon processors. In their place are cheap and relatively anemic models--a five-year-old AMD chip in the first prototype and Intel's Atom for handheld PCs and eventually mobile phones in the second-generation design under way, Andersen said.

The system uses a front end to communicate with the outside world, and an internal network adds reliability and flexibility. Each processor runs a stripped-down version of Ubuntu Linux.

Commercialization, though, will come from elsewhere.

"I told my students I'd be willing to do a start-up if they wanted to. Collectively we decided finishing a bunch of Ph.D.s is more important," Andersen said.

There are plenty of other server makers in the world, though, and many of them are interested in flash memory packaged densely into what are called solid-state drives. Sun Microsystems has keen interest in flash memory and built a specialized database system for would-be acquirer Oracle. Dell sees great promise in solid-state drives.

And addressing the brains in FAWN's design, start-up SeaMicro apparently is developing Atom-based servers with numerous processors. Founder Anil Rao is one inventor on a SeaMicro patent application for a computer system with numerous independent processor modules that share access to shared resources including storage, networking, and boot-up technology called the BIOS.

The FAWN approach can be adjusted with hard drives or conventional memory to match various sizes of datasets or rates or the queries retrieving that data.

(Credit: Carnegie Mellon et al.)

The key value of FAWN
So where exactly is FAWN useful? Andersen makes no claims that it's good for everything--but the use cases are often central to companies at the center of the ongoing Internet revolution.

Specifically, it's good for situations where companies must store a lot of smaller tidbits of information that's read from the storage system much more often than it's written. Often this data is stored in a form called "key-value pairs." These consist of an indexing key and some associated data: "The key might be 'Dave Andersen update 10,579.' The update value might be 'Back in Pittsburgh.'"

But conventional memory is expensive, and hard drives are bad at retrieving lots of small bits of data such as image thumbnails or social-network contact names stored all over the disk. "If you look at kinds of workloads that challenge modern computers, those with lots of random access to input-output are incredibly inefficient on high-end CPUs," Andersen said.

Systems such as Amazon's Dynamo, Facebook's Memcached, and LinkedIn's Voldemort store data as key-value pairs. Google's MapReduce technology and Yahoo's Hadoop-based equivalent use key-value pairs in processing search engine data and other tasks.

Programming a FAWN cluster would be difficult, but tailoring it for key-value pairs means it can be packaged as a special-purpose server appliance. Customers need not know the details of its inner workings, Andersen said.

Cut the power
These large-scale systems don't come cheap. Besides the hardware, software, and maintenance costs, there's power, too--and companies often must pay for energy twice, in effect, because servers' waste heat means data centers must be cooled down.

The researchers compared how many datastore queries could be accomplished per unit of energy and found FAWN compelling: a conventional server with a quad-core Intel Q6700 processor, 2GB of memory, and an Mtron Mobi solid-state drive measured 52 queries per joule of energy compared to 346 for a FAWN cluster. And tests of a newer design show even more promise: "Our preliminary experience using Intel Atom-based systems paired with SATA-based Flash drives shows that they can provide over 1,000 queries per Joule," the paper said.

The approach can be tailored for different varieties of work. For larger data elements that needn't be accessed as often, FAWN clusters could be built with conventional hard drives. For those with smaller data elements needed even more frequently, FAWNs could use more conventional memory.

There have been server fads before--the early generation of blade servers, consisting largely of lower-end processors squeezed tightly together--were a commercial flop. But Internet-facing data centers are a big business these days, and with cloud computing on the rise, it's going to get bigger. So perhaps the FAWN approach will catch on at least in that area.

But if somebody wants to commercialize it, they'd better think about changing the name.

"I worry no manufacturer will ever want to produce a device called a wimpy node," Andersen said. "But we like the name."

[mbenedict]

Memcached is used at Facebook but is not their technology. It was developed by the people behind LiveJournal.

[Lerianis3]

Sorry, but these things will not work for POPULAR websites. For a non-popular to semi-popular website, these things might work.. not for Amazon or E-Bay or Newegg even.

[newideaguy]

Perhaps you could explain why you feel this way? And point to some data in support of your claim... <br /> <br />It is worth noting that Amazon already sells compute in slices smaller than an ATOM. <br />Amazon's basic EC2 unit is a 1 Gigahertz Operteron slice. This is less compute than a 1.6 Gigahertz ATOM. <br /> <br />I think the data will show that exactly the most popular websites will be the places that adopt these.

[rapier1]

That's because you are thinking of this in terms of a 'website' as opposed to an internet facing computing resource. Could this set up be used to serve webpages? Sure, but that's not really what its designed to do. A fawn cluster would more likely be used to store the small bits of information used to create database driven sites. For example, the web server would receive a request for a page that has N static elements and Y dynamic elements. The static elements could be served from a more traditional repository but the dynamic elements could be rapidly drawn from a cluster of load balanced wimpy servers each parceling out some fraction of the data elements. Your aggregate speed would be substantially higher allowing you to serve more requests at once. More importantly, the amount of infrastructure support required would be substantially less than a traditional data center thereby cutting costs. Go back and read about the key-value paring. If you understand that then you may understand why this concept has legs.

[Lerianis3]

Yeah, tell the people at the various forums I go to that their databases are 'small'..... not! The one site's database is over 40GB's in size (yes, it is a semi-popular or popular website). Another ones is nearing 50GB's with all the postings on it.<br /><br />This wouldn't even be good for their database-type stuff..... or for anything that I can really think of, to be totally honest.<br /><br />Simply put.... this stuff doesn't have 'legs'.... it has bleeding stumps where people have not thought through the things in question. Wimpy servers simply put are NOT going to be able to do anything. They tried this a few years ago for a non-popular website, and the servers went down almost immediately from the small amount of traffic on them. Of course, they were using old Pentium 3 processors in that experiment, but I think it would still hold true for these Atom chips and other chips that are even wimpier than those Pentium 3's.

[mbenedict]

@Lerianis3:<br /><br />Sorry, but a 50 GB database today is tiny. Look again at the dataset size chart in this article... they're designing for datasets in the 100s and 1000s of _Tera_bytes.<br /><br />What you still don't understand is FAWN is NOT a webserver and are NOT meant to directly "run" websites. Rather FAWN is a back-end component meant to be a specialized type of partitioned database. A typical use case would be for a distributed cache, but the system could be used for anything from storing indexes (think Google's url indices) or to enable large-scale parallel processing (via MapReduce, etc.)<br /><br />Maybe you should read up on things like memcached to understand why FAWN could be a compelling alternative.

[inachu1]

I forgot the url but there is a C-64 webserver out there somewhere.

[ckurowic]

Yeah I've heard of it. There are also sites being run off of an Apple Newton. You don't need much power to run a web page. Come on, an 8-core Xeon based server? *** do you need that for?

[catbutt5]

Oh, I needed a good laugh...<br /><br />"And addressing the brains... Anil Rao is one inventor on a ... patent applied for a computer system with numerous independent processor modules that share access to shared resources including storage, networking, and boot-up technology called the BIOS."<br /><br />Trying to patent something that's existed for more than 20 years are you? Good luck with that.<br /><br />Anil, ever heard of Sun or IBM or companies that sell refrigerator sized (small and large) computers full of little card slots containing memory and processors (even at different frequencies) that share storage, networking and yes, even the BIOS? It's the same concept.<br /><br />What's your act 2? Gonna try to patent the automobile?

[newideaguy]

Its worth taking 5 minutes to review the patent application. Its actually very interesting. The description in the article doesn't do it justice.

[kirkktx]

"52 queries per joule of energy compared to 346 for a FAWN cluster"<br /><br />Somewhere I saw that electricity costs exceed hardware costs amortized over the life of the computer. These numbers should certainly attract investors.

[mbenedict]

It's not just about the raw cost. There's a finite amount of electricity you can bring to a data center, so at some point the number of queries you can do per kWh becomes very important. The article mentions heat as waste but like electricity, heat itself also becomes a limiting factor in a large data center. There's only so much cooling capacity available beyond which you get severe diminishing returns.<br /><br />So a system which promises to be more energy efficient and runs cooler at the same time... that could be a big win.

[symbolset]

I've been a proponent of FAWN for a long time. For ten years the software has provided the redundancy and the scale. FAWN is not the right answer for every problem, but no tool is.<br /><br />Configuring the right solution for massively parallel problems is a fairly complex geometry. If you approach a large-grain problem from a cents-per-compute-per-second perspective then FAWN is a slam dunk. For fine-grain problems you want to use GPGPU instead. When the problem becomes large enough, custom system boards and esoteric processors enter the solution set.<br /><br />It's really only when you don't know the granularity of the problem, or you need a general solution that solves both ends of the granularity scale and the middle too that Industry Standard architectures are ideal. In these cases a mixed cluster of wimpy nodes combined with GPGPU nodes may be more cost effective.<br /><br />Oh, and about cooling: The answer to many problems that start "How do you..." is... don't. As many have shown the correct answer to the cooling problem is not refrigeration, it's location, location, location. Your servers are rated to 35C (95F) at least, and if the ambient temperature where they are rises above that, you located your servers in the wrong geographic area, which is a different problem. There are lots of places you could put your servers that won't get that hot in the next decade. Put your servers some place where the ambient temperature never goes out of range, preferably where they have cheap power (I hear Canada is nice). To find the ideal operation for the fans of your datacenter, heat the inlet temperature to 35C. Fire up the equipment and stress test it at maximum capacity. Measure the outlet temperature. Now you have the ideal outlet temperature. Regulate the fan on the exhaust such that the exhaust is consistently that temperature, less a few degrees for safety, and your server components will remain at a consistent temperature (thus preventing swings in temperature which can cause problems). This is not as complicated as you might think. As an added benefit during a "heat wave" stationary inversion the thermodynamics of a hot exhaust plume exiting high above the building plus the related ground-level cool air inlets creates a cooling breeze which diminishes the air conditioning required to cool the humans in the related office spaces when they're not in the datacenter. Don't insulate the datacenter part either - that's swimming upstream. Maintaining a snow load on the roof should not be a design goal. Also, in really intemperate climes filter the exhaust and pass it through the human workspaces (or if you're really fussy, use a heat exchanger) - the servers are heating air, there's no sense burning extra energy to heat separate air to keep the humans comfy.

[ckurowic]

I disagree with your point of recirculating the hot air from the servers to people's work areas. Some are VERY sensitive to the outgassing that occurs when equipment is new (and even for many months afterward). You have interesting concepts, but I'm afraid you don't have the engineering background to support it.

[Christopher_Mims]

Great article - provides a lot of detail that didn't make it into my own write-up of FAWN for Technology Review. If you're interested in a slightly different take, though:<br /><br />http://www.technologyreview.com/computing/22504/?a=f