The Pervasive Data Center

Standards evolve in a lot of different ways. However, broadly speaking, they fall into two main buckets: de jure and de facto (to use the Latin-derived legalese). By law and by fact.

In high tech as elsewhere, it's often a matter of historical accident and political maneuvering that determines which approach wins out in a particular area of technology. And it can be a high-stakes game for the companies involved, with big players often seeking to position their approach as a "standard" even if it's only standard in the sense of being ubiquitous (think Microsoft Windows) while the smaller guys tend to favor approaches blessed by standards bodies or at least industry corsortia.

In cloud computing, we're seeing almost all the forms of standards-making coming into play with the primary goal of promoting interoperability among different cloud service providers and between private and public clouds.

On the de jure side, the most significant standards-making effort is taking place under the auspices of the Distributed Management Task Force (DMTF), an established organization in the management standards space. AMD, Cisco, Citrix, EMC, HP, IBM, Intel, Microsoft, Novell, Red Hat, Savvis, Sun Microsystems, and VMware announced in April 2009 (PDF) that they would comprise the board for an Open Cloud Standards Incubator within the DMTF.

If you were to observe that none of those companies currently have a big play in public cloud infrastructure, you would be correct. Microsoft has its Platform-as-a-Service Windows Azure play, and a number of those companies are very active in working with enterprises to build internal cloud-style IT, but none have an Infrastructure-as-a-Service (IaaS) offering in the vein of the conspicuously absent Amazon.

And it's Amazon Web Services (AWS) that has clearly emerged as the de facto standard for IaaS. The fact that Amazon is one of the first vendors that comes to mind in just about any discussion of public clouds is one indication. Another is the growing ecosystem of companies like RightScale that add additional features to AWS--not uniquely, but first and foremost. We now even have an open-source project and company, Eucalyptus, that lets organizations implement their own clouds that are compatible with many AWS services.

Now one of Amazon's competitors, Rackspace, is taking yet another approach to promoting its implementation as a standard. It's open-sourcing the specifications for its Cloud Servers and Cloud Files API under the Creative Commons 3.0 Attribution License. It also has made available its Cloud Files language bindings for Java, PHP, Python, C#, and Ruby under the MIT license.

The Creative Commons license that Rackspace is using lets users both share and change the work so long as they provide attribution to the creator, in this case Rackspace. The MIT license is a very permissive license in the vein of BSD that allows essentially any use of the code including its use within proprietary software.

Rackspace has emerged as a major player in public cloud infrastructure. Cloud Servers competes with Amazon EC2 and Cloud Files with S3. It also has a product called Cloud Sites, based in part on technology from its Mosso acquisition, that is more of a PaaS offering with dynamic scalability (but that's not part of this announcement).

This announcement doesn't fundamentally change the landscape. However, it does give an already well-established IaaS vendor a point of clear differentiation from its biggest competitor.

Toward the end of last year, I more or less decided that I wanted to get myself a Kindle, but I wanted to hold off for the next generation. So when Amazon announced the Kindle 2 in February, I put my order in right away.

(Credit: David Carnoy/CNET)

I've now had it for a few days and have had a chance to play around with it a fair bit. Here are some early thoughts.

Yes, it's expensive. $359--and typically add to that at least $29 for a case. Although there are many free books available (more on this in a bit), and new releases are generally cheaper on the Kindle than in hardcover, it's a good bet that you're not going to save enough on book purchases to come close to paying for the device--especially if you buy a lot of books used (as I do) or get them from the library. This is a premium-price convenience device.

It's the convenience that the Kindle 2 offers that convinced me to buy one. I travel a lot, and the idea of having a library in the form factor of a single paperback is immensely appealing to me. Frankly, I probably would not have purchased a Kindle, if I didn't spend so much time traveling by air with as little luggage as I can get away with.

Plenty of others, including CNET Reviews' David Carnoy, have reviewed the device itself, and I agree with what seems to be the general consensus: the Kindle 2 is easy on the eyes, and the controls seem to work reasonably well.

For reading books, it is a qualitatively different experience from reading on a laptop or a phone. It's not that you can't read on those other devices--in fact, I do it all the time--but the Kindle's e-paper display and long battery life make it far better suited for reading books.

That said, I do believe that we're still in a relatively early stage of this device's evolution. There may or may not be any truth to these specific rumors from Fast Company. (After all, we heard various inaccurate Kindle 2 "leaks" and predictions throughout much of last year.)

An article by Edward Wyatt in the New York Times discussed how the Amazon Kindle e-book reader was stirring unease at the BookExpo America trade show.

But excitement about the Kindle, which was introduced in November, also worries some publishing executives, who fear Amazon's still-growing power as a bookseller. Those executives note that Amazon currently sells most of its Kindle books to customers for a price well below what it pays publishers, and they anticipate that it will not be long before Amazon begins using the Kindle's popularity as a lever to demand that publishers cut prices.

I'm a bit skeptical about this particular concern. From my perspective as a consumer, one of the problems that I have with e-books today is that I have to buy a $400 device and then still have to pay almost as much for the bits as for the dead tree version even though many of the costs associated with printing, distributing, and inventorying physical books are eliminated.

That's not to say that costs go to zero--nowhere close. And there's a legitimate concern that buyers may naively assume that they do. An earlier post on this topic: Digital distribution isn't free. But costs are lower--and the prices should reflect that.

What would seem a more germane concern is whether pervasive e-books lead to pervasive trading and copying. DRM, my other beef with a lot of today's e-books, inconveniences legitimate users as much as it retards piracy. So I think we can take that off the table a solution that's either desirable or especially effective. Today, the economics of photocopying and the restraints that time and space put on physically giving a read book to the next reader sharply limit the amount of duplication and trading that can take place.

In my view, you shouldn't discount limits imposed by the physical world too much. While movies consume plenty of bandwidth on the Torrents, their quality and the effort it takes to download them--paired with the ready availability of modestly-priced, full quality movie rentals--means that they aren't that attractive for a lot of people. Certainly music lends itself far better to downloads--legal and otherwise. And the apparent impact on the recording labels has been proportionately greater as well.

A lot of the dynamics associated with creating, producing, distributing, and purchasing books are considerably different than those of music than those of movies. Electronic distribution creates possibilities. Impulse purchases from the living room and ready availability of the longest, "long tail" work are just two.

But, at the same time, the cost of putting toner powder on a page of paper and the time associated with putting all those pages on a copier glass will no longer be a defense in a world where "If it can be copied, it will be copied" often seems to rule.

One of the problems with putting things into categories is that as technologies and the environment change over time, those which were once separate and distinct can become much less so. But, because we've grown so accustomed to thinking of them as independent entities, we can miss that shift.

From a practical business perspective, this can mean failing to notice that someone we never thought of as a competitor is now serving the needs of our customers. They may well be doing it in a different way or coming at a problem from a different mindset or design point. But, at the end of the day, they end up solving the same customer problems and taking away business. The datacenter space is replete with examples such as more distributed systems replacing centralized ones (through several cycles) and standard interconnects replacing proprietary ones.

Here's a new one to think about in the Web application and Cloud Computing space.

Is Web Hosting the same as Cloud Computing? It's tempting to say not. After all, isn't Cloud Computing the future while Web Hosting something that's been around since, well, the beginning of the Web (and even earlier if you consider all the various pre-Web Internet services)? But what is Web Hosting exactly? It's providing access over the network to a set of services--such as those associated with the LAMP stack--together with some storage capacity, and a bandwidth contract. For this reason, in Defining Cloud Computing, I wrote "we take a fairly broad view of Cloud Computing. It’s not just about the enterprise, just about the consumer, or just about delivering entire applications. Software as a Service (SaaS), Hardware as a Service (HaaS), Data as a Service (DaaS), and Web 2.0 are all part of the cloud. Even hosting providers are a sort of specialized, narrow case. We take such a broad view because all of these intersecting sub-categories do share at least one common characteristic: the Network is the abstraction layer. "

I bring this up because, as Netcraft writes:

With Amazon's recent offering of low-cost web application hosting, and now Google's free web application hosting, the conventional web hosting industry may be set to see some radical changes. With both services providing high scalability, yet without adding complexity, these could be seen as an attractive alternative to setting up a busy website on dedicated servers. Conversely, they are less likely to appeal to casual website owners, simply because the services require more knowledge and skill to use than simpler services such as Google Pages, Blogger or Apple iWeb.

There's nothing fundamentally different or special about Web Hosting. It's just a certain selection of network services and abstractions that hosting providers expose. As Netcraft suggests, it will probably continue to be a useful choice of services and abstractions for a basic Web site. However, newer approaches, whether Goggle's, Amazon's, or from a lesser-known company like Mosso, may well become the favored approach as environments get more complicated and capacity needs more variable.

When photo site SmugMug initially contacted me, it was in the context of some of the pieces that I had written about competitor Flickr and some of the issues associated with protecting photographers' works online.

In a nutshell, relative to Flickr, SmugMug has opted for less of a open-community orientation than for ways to store and display photos with a rather granular set of access controls. (See some discussion by CEO and "Chief Geek" Don MacAskill.)

These are important topics that I'll be discussing further in due course, but today, I'm going to focus on SmugMug's physical infrastructure.

During my conversation last week with President Chris MacAskill, he made some points about using Amazon.com's Simple Storage Service (S3) that may not be widely appreciated. (S3 is Amazon's "storage as a service" offering that users pay for based on the amount of storage space used and data transferred.

Like Amazon's EC2 compute service, it falls roughly into the "Hardware-as-a-Service" concept.)

SmugMug was one of the earliest S3 users. As Chris tells the story, SmugMug was buying a "mindblowing" number of Xserves from Apple. The Silicon Valley-based company was running out of power and space--the usual story.

However, Chris raised another point that bears mention. The company was having to buy all this gear up-front, in advance of the revenues (i.e. user subscriptions) that it would hopefully generate. This was difficult from a cash flow perspective--especially for a company that wasn't venture capital-backed. But the reality is actually worse.

Not only were the expenses up-front, but they were capital expenses. From an accounting perspective, this means that the depreciation on the systems hit the P&L in a given year. The result? You may look profitable, but cash flow is tight and you could end end up effectively "prepaying" taxes.

Then Amazon called out of the blue, after a conference, and told the site about S3. At Amazon's initial target of 50 cents per gigabyte, it was intriguing. When Amazon ended up pricing its offer at 15 cents, Chris says the company's "jaws dropped."

Initially, SmugMug used Amazon S3 for backup while keeping all of its primary storage in-house. At the beginning, it wasn't thrilled with uptime, but it said that it wasn't disappointed, either. More troubling was that Amazon wasn't so transparent about the time and length of outages, which seems to remain a big issue.

However, over time, SmugMug started seeing better uptime from Amazon than it could deliver in-house. It now has more than 400 terabytes of photo and video storage on S3, and it can add as much as 1TB on busy days.

Now that the company has switched much of its primary storage to S3 as well, there's another economic point worth making. Were SmugMug to host all this storage in-house, it'd actually have to buy more like 1.2 petabytes because it'd need enough to support any growth spurts and enough for backup, as well as primary storage.

With Amazon S3, the company effectively gets backup for "free." (Of course, that assumes that you trust Amazon not to lose data, but as far as I know, there has been no data loss associated with any Amazon outages.)

SmugMug is also a heavy user of Amazon's Elastic Compute Cloud (EC2), even though the service is still in beta test mode. One of the most appealing features of EC2, according to Chris, is that it can handle load spikes without paying for the capacity all the time. For example, loads go way up after a three-day holiday weekend, when people upload all their pictures on Tuesday.

All that said, the company does maintain some of its own servers. It does this, in part, to provide a sort of cache for "hot" photos. (Chris estimates that 10 percent of the photos on the site get 90 percent of the traffic.) Related is the fact that SmugMug runs its MySQL database servers in-house (so it'll be physically close to the hot photos.)

Amazon's recently announced SimpleDB could potentially offer an alternative, but it's missing some features that SmugMug's software, as currently written, requires. (See some technical discussion here.)

I suspect that we'll see these hybrid architectures--even at aggressive Cloud Computing adopters--a lot. You sometimes need that little bit of customization or specialization that you can't get from a service that has to be relatively standardized. That said, SmugMug is an aggressive adopter, and it gives us some good insights into what can be gained by making the infrastructure largely someone else's problem.

When my posting frequency drops a bit, the usual reason is that I'm flying here and yon and otherwise occupied with goings-on at some conference, meeting, or client engagement. The situation in January was a bit different. For the first time in a while, I had some decent blocks of uncommitted time. And I put those to use fleshing out and writing some longer research notes that had been sitting on the to-do list for way too long.

Two of these deal with so-called "cloud computing"--the idea that software will increasingly run in the network. These were originally planned as a single paper, but for structural and length reasons, I decided to break out the definitional piece, "Defining Cloud Computing." To tell the truth, I don't typically find formal taxonomies and categorizations especially interesting, but I thought it useful in this case to be clear about the topic under discussion.

The main research note, "The Cloud vs. Open Source," focuses on the relevancy of open source in a cloud computing world--and, especially, whether other types of protections and rights may not be more important than the right to view, modify, and redistribute source code. Tim O'Reilly has written and spoken on this topic.

At the just-concluded Sun Analyst Summit, I also had the opportunity to broach this topic with Simon Phipps, Sun's Open Source Officer. An interesting perspective that he added is that we're really talking about two different kinds of rights. One is essentially individual--the right for me to decide who can access what "data" that I "own" (whatever those terms mean exactly) and to transfer my data from one place to another. However, there's also the idea of what I'll call community or collective rights--the idea of reciprocal obligations associated with providing application programming interfaces and access.

One follow-up piece that I want to write when I have time will be something along the lines of "Why Not the Cloud?" in which I'll look at some of the inhibitors to moving computing into the network.

Finally, "The Future of the Operating System" looks at how changes in the way that we operate computers and deploy applications is starting to change how we view the operating system, a technology construct that, in important ways, hasn't really changed for decades. Server virtualization is the big driving force behind change here. However, virtualization is hardly unrelated to cloud computing--both through services like Amazon EC2 and, more conceptually, in the fact that virtualization is all about masking lower-level details from users.

These three Illuminata research notes are all available as free samples.

The idea of the "long tail," a concept popularized by Wired's Chris Anderson, permeates much of what is going on with the evolution of IT.

After all, it's the mass participation of almost everyone in creating content of various types that's driving an enormous amount of IT build out--which, in turn, may well change even how and who builds computers in the future. Simply put, the long-tail premise is that bestsellers aren't in the majority when one tallies up the sales at Amazon.com or the page views on blogs. Rather, it's the total of the far more numerous other 80 or 90 percent of content.

Less abstractly, Anderson's argument is about business. Namely, he argues companies can make money selling to the long tail as shown in the data that I discuss in this 2005 post. I thought and think that it's a powerful concept--although I also think it fair to ponder how many companies are truly well-positioned to make money from the long tail.

When Amazon, Netflix, and Google make their appearance as exemplars for the umpteenth time, one starts to wonder. (In all fairness, Anderson has additional examples; Amazon and Netflix just make particularly rich, data-heavy case studies.)

However, as well noted by Alex Iskold over at Read/Write Web this morning, there's a slightly subtle, but very important, distinction to be made when we're discussing making money on the long tail. It's about making money on the long tail, not making money in it.

A couple of weeks back, Amazon.com announced an expansion of its Elastic Compute Cloud (EC2) service. The still-in-beta EC2 is a twist on the much-discussed, if rarely seen in the wild, compute utility whereby customers rent computing by virtual machine (VM)-hour; Amazon's EC2 infrastructure is based on a Xen hypervisor structure rather than running directly on physical hardware.

One implication of Amazon using VMs is that they can easily offer a variety of different VM sizes up to the size of the physical hardware. That was the most recent change announced. In addition to the default "Small Instance," users can now get "Large Instances" or "Extra Large Instances." These might be useful if, for example, you need to pair a heavyweight database instance with some lightweight Web services.

Another implication is that VM images, called Amazon Machine Images (AMI) in this case, can be archived and transported. This is analogous to VMware's virtual appliances. Amazon itself hasn't done much to jump-start an image marketplace at this point as VMware has. However, it does provide a mechanism for customers to post and publicly share AMIs and sees the opportunity for people to offer paid AMIs over time.

I bring this up because Emre Sokullo over at Read/Write Web has a post and table that does a great job of crystallizing why getting into Web services is such a big deal for Amazon. In short, Amazon's revenue is comparable to Google's. The difference is that, while Google is operating at a 29 percent profit margin, Amazon is under 2 percent. Which is probably about the best one can hope for with a big "mail order" retail operation.