Comments on: Deduping: Killer app behind battle for Data Domain

Process known as data deduplication is white-hot. So hot that Data Domain, virtually unknown outside the storage world, is now deemed worth at least $1.9 billion.

[bj1126]

I am watching this one closely. It will be interesting to see what comes of this.

[JohnSWebster]

It was once the case that dedupe was only for backup. Then NetApp allowed users to dedupe primary data stores under ONTAP. Other vendors are following suit. There will be a number of variations on this theme for different applications I believe. I first encountered dedpue when I met a company named Rocksoft back in 2003. Their patents for a process they called Data Coalescence were filed in 1992. They were bought by ADIC, which in turn was bought by Quantum. If you're watching this space closely, that should ring some bells. JW

[Seaspray0]

You need to look at the way they shrink data. Basically, they do what's known as a "single instance store" of data. If you have 10 GB of data, the first full backup takes up to 10 GB (compression usually gets you about 30% savings). The next time you do a full backup, the app sees that you're backing up the same data (sectors are identical) and only puts in pointers to the first data. So... your second full back of the same data takes a very small fraction of space since you only retain one single instance of data and point to it over and over again. If your data is fairly static, you can do hundreds of backups and consume roughly 11 to 12 GB of space (one instance of data and lots of pointers). On tape, you would be consuming roughly 1000 GB of space. "Cool!" you would say, and yes it is. But there is a danger as well. Each tape holds its own instance of the data. For 100 tapes, that would be 100 instances. Redundant? Yes it is. But I'm also concerned of only having 1 instance being referenced 100 times. All is takes is one failure in one instance, and the data is gone for all 100 full backups. For that reason, Data Domain doesn't use raid5. They use the next revision of the raid where you have 2 parity drives per logical disk instead of one (allowing 2 disks to fail and still retain the data). I would prefer to include a mirror of that as well. I want more than one instance. Data Domain gives you the option of duplicating to another appliance and this satisfies my desire, but the cost is double for that solution. Adding the second appliance makes tape very competitive in price. I will still keep an eye on the cost. If Data Domain lowers their prices where doubling up the appliances is cheaper than tape, I'd very much consider it.

[JohnSWebster]

While I?m not going to argue the case for DataDomain or their pricing practices, I will say that with higher density disk on the near-term horizon, the cost per GB will likely come down to the point where your analysis tells you it?s time to jump in. Personally, I think we?re at that point now, which is why there is the huge interest in DataDomain. JW