Outside the Lines

On this week's EIC Squared podcast, ZDNet's Larry Dignan and I talk about the tanking economy, the challenges facing an Obama administration CTO, and Microsoft's search quests with Verizon Wireless and Yahoo.

The holiday shopping season is looking grim as Circuit City files for bankruptcy and Best Buy lowers its forecast for its fiscal year. When will it ever end?

President-elect Obama has called for a national CTO. Given the complexity of technology infrastructure, the abundance of projects, the squeeze on budgets, and policy controversies, this will be an extremely challenging position.

We also discuss Microsoft's next moves to increase its share of the search market, with Verizon Wireless or Yahoo, or both.

On this week's EIC Squared podcast, ZDNet's Larry Dignan and I discuss the legal tussle between Apple and the Mac cloner, Psystar.

This week, Psystar sued Apple on antitrust grounds. Psystar execs said they just want to make the Mac OS "more accessible" by offering it on cheaper hardware than what Apple provides. It's hard not to imagine Apple fighting this one to the bitter end and Psystar getting crushed in a lengthy litigation.

Another battle is brewing with Cisco Systems adding e-mail and calendaring to its on-demand, collaborative software platform with the acquisition of PostPath. This might speed up Microsoft's delivery of an on-demand software suite. If Cisco wants to push its suite further, Zoho would be an acquisition target.

Larry and I also discuss the coverage of the Democratic National Convention. And I share my thoughts on Dell's cloud computing efforts, which means selling bare-bones servers optimized for cost, operational efficiency, and energy conservation.

SAN FRANCISCO--Standing 52 stories in the air at the upscale Carnelian Room in the Bank of America building here, executives from Dell, Facebook, and Salesforce.com discussed the meaning and use of the latest technology buzzword, cloud computing.

The sky was blue and cloudless, but it didn't adversely impact the atmosphere of what turned out to be a Dell marketing event. It was pitched as an announcement about a partnership that involves "the next generation of cloud computing."

You might recall that Dell is the company that owns the URL Cloudcomputing.com, and made a failed attempt to trademark the phrase. Earlier this month, the United States Patent and Trademark Office rejected the company's application. Dell marketing head Andy Rhodes wasn't willing to comment on whether Dell would appeal the USPTO decision.

Despite the cloudless sky, the speakers offered genuine insights into cloud computing, an umbrella term for "hyperscale" computing that covers everything from delivering compute services like a power utility delivers electricity, to simply hosting applications off-premises (see also software-as-a-service and on-demand computing).

Event host Forrest Norrod, vice president and general manager of data center solutions at Dell, defined cloud computing as an economic enabler for applications, not just for single applications but for platforms-as-a-service, such as Salesforce.com. He emphasized the economies of scale advantage that cloud computing has over client/server and previous generations of infrastructure deployment.

Dell is currently a cloud computing arms supplier to companies such as Facebook and Salesforce.com. "Dell is focused on early adopters and large customers, about 50 worldwide, to provide optimized servers, storage, and data center infrastructure," he said. "Cloud computing is still an emerging market, with standards across the framework and software stack still emerging. We are trying to promote an ecosystem to build the software stack on top of the infrastructure. You will gradually and judiciously see us add capabilities up and down the stack.

Norrod pointed to recent Dell acquisitions--Message One, Silverback Technologies, and Everdream--as examples of Dell's focus on software, not just the hardware piece. Increasingly, both Dell and HP are building out their software stacks to compete with Sun and IBM for providing highly automated data centers running commodity hardware optimized for cloud computing.

Jonathan Heiliger, Facebook's vice president of technical operations, had some praise for Dell. "Dell is doing the most aggressive things possible to optimize for cloud computing," he said. "We think Dell is perhaps the furthest along and we see them as a thought leader." Facebook has more than 10,000 servers, Heiliger said, and it's safe to assume any of them come from Dell.

He noted the price of hardware is not the biggest issue. Vendors can even sell hardware at a loss or at a fixed margin cost to get the initial business. "What we have seen in the landscape is that most server providers are trying to provide Lexus quality products at a Toyota price. We are looking for Scion products at a Scion price," Heiliger explained. "(Vendors) have to be creative around power and airflow optimization. The cost of operating the hardware is key; you have to take down the operating cost, not just the server cost."

For Heiliger that means bare-bones servers. "We don't need fancy graphics chips and PCI cards," he said. We need one USB port and optimized power and airflow. Give me one CPU, a little memory and one power supply. If it fails, I don't care. We are solving the redundancy problem in software." Blade servers are not ideal, he said, because of the higher cost and proprietary lock-in that come with the lack of a standard chassis.

Check out the video interview I conducted with Heiliger about managing infrastructure hypergrowth as Facebook adds 250,000 users per day.

Claus Moldt, vice president of technical operations at Salesforce.com, offered similar comments to the previous speakers. The company is phasing out Sun equipment and standardizing on Dell servers (Dell is a customer of Salesforce.com). Salesforce.com has two data centers in the U.S. and one due to go online in Singapore later this year. Moldt said his biggest challenge is capacity planning, making sure that as customer usage patterns change, the Salesforce infrastructure can adapt instantly.

Dell is betting big on cloud computing to boost its enterprise footprint. At this point, Dell doesn't have plans to build its own cloud to provide hosting for external applications, Norrod said. But, there may come a time when being an arms supplier won't be enough for Dell to be competitive. In addition, selling bare-bones servers can't be much of a high margin business, which is why Dell is moving more into software and services. Norrod said Dell's cloud computing efforts have been a large component of Dell's recent market share growth. Dell's second quarter earnings due tomorrow should give a more precise indication of the impact of the cloud on company's business.

On this week's EIC Squared podcast, ZDNet Editor in Chief Larry Dignan and I debate whether Amazon.com's Kindle e-reader is the next iPhone.

That is a big stretch, especially given the way the iPhone has turned the smartphone business on its head, at least from a product design standpoint. The Kindle is a nice product, and Amazon could bring music, video, and other kinds of content to the device, but it's doesn't have the Steve Jobs touch.

In addition, all the talk about Kindle's skyrocketing sales doesn't ring true. If the Kindle were on such a hot streak, Amazon founder and CEO Jeff Bezos would be talking up the sales numbers. All I know is that I keep getting huge Kindle ads in my face every time I go to Amazon.

After a few hundred times, Amazon should figure out that I am not interested in the Kindle and should show me something that I might actually buy based on my history and the recommendation engine. That would certainly be a more lucrative way to use the front door advertising space.

In the podcast, we also discuss Best Buy becoming an iPhone distributor (good for both companies and not for Wal-Mart) and Gartner's endorsement of the iPhone as enterprise-ready. In addition, we note Dell's latest refresh of its Latitude laptops, including a quick-start feature and a battery that can give up to 19 hours of juice.

A few weeks ago I talked with Jonathan Heiliger, vice president of technical operations at Facebook, about the challenge of innovating quickly and building stable infrastructure while 250,000 new members are added to the social network every day. Check out the video on ZDNet.

Q: You've been at Facebook, I think, for about a year and it's been quite a ride I guess, scaling up from zero in 2004 to over 80 million today. How do you keep up with that hyper growth?
Heiliger: You're absolutely right--we've had a lot of growth. We add over 250,000 users every day, and that means a lot of infrastructure, a lot of servers, and constantly looking at new processes and looking at how we're doing things and ensuring that we're doing things the most efficient way possible, not just for delivering all the content to our users but to stay on top of what it costs to run the site.

How do you stay on top of the cost in terms of the kind of equipment you buy and how you work with the vendors? How do you prioritize those things?
Heiliger: One of the things we recently did was we ran an RFP process for the servers we buy from vendors and essentially did a bake-off with a number of different people looking at building servers on our own. What we concluded from that process was to continue to buy servers from a couple of major OEMs (original equipment manufacturers), but through that process we were able to lock in prices today and carry those prices forward as all the commodity components costs drop.

When you're buying those servers, and I assume you're doing just a huge scale out of commodity servers, what do they look like? How are they configured?
Heiliger: We're pretty lucky in that we run a wide variety of applications, literally tens of applications on our own and hundreds of applications for our platform developers that use Facebook as a distribution mechanism, as a way of interacting with their users. But one of the reasons we're very lucky is our engineering team has selected to use PHP as the primary development language. That allows us to use a fairly generic server type. So we, with a couple of exceptions, have three main server types and run a fairly homogeneous environment, which allows us to then consolidate our buying power.

You're different from Google in the kinds of applications that you run. They are mostly running search queries, and you're running all kinds of queries and bringing back all kinds of data from the social graph. How is it different in terms of the way you build out your data center from the inside?
Heiliger: Google has a tremendous amount of information that they index and archive and present to users, but fundamentally if you go to Google and type in a search for a "tiger" and I go to Google and type in a search for a "tiger" we're going to see generally the same results, so they're presenting that same information to both of us. Facebook is a little different in that the context for our data is all social. When you look at your friends and their status updates and their photos and the notes they may have written, you're going to see one set of data versus if I look at my friends and their photos and their notes and status updates, and those tend to be non-intersecting sets of data.

So it's much more dynamic?
Heiliger: Much more dynamic data set--and what that means is it's caused us to do a bunch of different things relative to caching and relative to federating all of that data up amongst thousands of different databases so that as a user requests all of that information we're not using one particular server every time for different data.

You recently introduced a chat application on Facebook, and it seems like it took a lot of time to test it to make sure it could scale having all those simultaneous conversations going on. Could you give us a little background and color on how that came to be?
Heiliger: Chat is actually one of our most recent launches. It started as a hack-a-thon project, which is one of the things we do about every other month. People get together and work all night and pick a project they don't have time to do necessarily during the day. From the time it really germinated as an idea to the time it launched and was available for our entire user base, it became a more formal development project. One of the things we did as part of that was actually built a new back-end service to be able to deal with all of the millions of simultaneous connections that we persist for users.

One other thing I was reading up on some of the work you've been doing--you say that clouds don't solve single points of failure in your stack. What are those single points of failure?
Heiliger: Interesting question, and the notion you are referring to there was part of the talk I give in regards to cloud computing is just a panacea, and for a start-up or even a more mature start-up like Facebook, isn't the answer to solving failure points in an application. By that I mean the underlining infrastructure that powers an application is typically the result of, or the outcome of, how the application is originally designed and how users interact with that application. If an application is poorly designed or designed to constantly reference a single set of data, the underlining infrastructure is going to be the victim of that. Guys like myself in the infrastructure world have to figure out how to best make that work.

As someone who is in operations how much impact do you have on the application development to make sure that once it gets into the data center that it can work properly and scale and not have the kind of failures we're seeing with some of the new applications?
Heiliger: I think it's a constant challenge in any organization, particularly a fast-moving one like Facebook, where we want to iterate quickly and get product out in our customers' hands so we can get feedback on that product and continue to tweak and enhance it over time. We have one force that's moving in that direction, and we have another force that says we want to keep the site up, we want the site to be reliable, and we want the site to be fast.

So there's a fine balancing act, where everyone in management and everyone in both the engineering and operations department constantly just sort of works, interacts, and goes back and forth, figures out just how to make those trade-offs. Sometimes we err too aggressively on the side of innovation and iteration, and put things out on the site in perhaps a small quantity that may break the site or cause the site to slow temporarily. Other times we air on the side of conservatism, of not releasing new functionality or new features, and that then delays the sort of user gratification of having that feature or fixing that bug.

What are the challenges that you see--let's say you're at 80 million unique users per month, 250,000 being added per day and 50,000 transactions per second. What happens when you get to 500 million or a billion if you ever get there?
Heiliger: Hopefully, tremendous things. I think we can only look forward to those days.

But what are some of the bottlenecks or barriers you have to overcome to get to that kind of scale?
Heiliger: Some of the bottlenecks we're facing are how we scale this extremely distributed set of data. One of the challenges we have is figuring out how to make that replicated such that it can exist in multiple places around the world and we don't also have to bring users back to the U.S. or back to one of our data centers. I think it's a challenge that most Web sites tend to face as they scale, which is you start in one location with a single database and then you have to figure out how to grow from there, primarily driven by the amount of latency or the amount of time it takes to reach the site and interact with the site. Being able to replicate the data across multiple data centers and across multiple geographies allows users to not just read their data from a local version but write that data as well. That is one of our key challenges over the next 12 months.

As you learn more about building up this very large scale infrastructure do you ever see the possibility that a Facebook could be a service provider?
Heiliger: What do you mean by service provider?

In the sense that right now you're just running the Facebook application but what if a developer or user wanted to do something similar to what Amazon is doing, using your infrastructure to run their applications in the cloud?
Heiliger: Gotcha. So one of the values of Facebook is the Facebook platform. We have over 100,000 developers and several hundred applications that have over a million users using them. We've talked about perhaps opening up or further opening up the platform by offering compute power for those application developers. One of the steps we've already taken to improve that development environment and improve the experience for our developers is just to open-source our platform, which we announced just a couple of weeks ago as well.

After attending GigaOM's Structure 08, I came away with a cloud-computing hangover. Just trying to define cloud computing is daunting given all the hype and companies thunderclapping.

Today the research firm Gartner has jumped on the cloud computing bandwagon, proclaiming that it "heralds an evolution of business that is no less influential than e-business," and defining it as massively scalable IT-related capabilities provided as a service using Internet technologies to multiple external customers.

Yahoo just announced a Cloud Computing & Data Infrastructure Group, which will develop computing infrastructure that balances scalability with cost effectiveness. What was Yahoo doing before it created this group?

I prefer the way Sun Chairman Scott McNealy talks about cloud computing. Ten years ago he was calling it the "big freakin' Webtone switch." Following is how he described it in December 2001:

That is the server, the storage, the operating system, the monitoring software, the clustering, the alternate pathing, multiple domaining, dynamic reconfiguration--and then it has a mail tone, a calendar tone, a news tone, an app server tone, and a directory tone. It has all of the different features of a big freaking WebTone switch and allows you to create this big jukebox. You can buy that all complete. Or you have one throat to choke and you can buy it all through a service provider that is SunTone certified. Or you can do what many IT directors do and they go out and buy the telephone switch by buying the chip from Intel, the operating system from Microsoft, the disk drive from EMC, the Compaq power supply, the Oracle database, the Novell directory, the BEA app server, the SAP, ERP, and CRM from here, blah-blah-blah, this, that, and the other thing, a SoundBlaster card from somebody else, the anti-virus uninstaller from Norton, and then go bring in IBM Global Services to try to make the whole thing work. Buy the big freaking WebTone switch.

At that time McNealy was talking about how enterprises provision their data centers and user services. Now we are seeing Amazon, Google and others take their data center expertise and make it available to developers and companies. Enterprises will be slower to move to the cloud, but they will eventually get there. Software-as-a-service providers are flourishing, and increasingly enterprises are considering off-premises, hosted solutions.

In essence, we are at the beginning of the age of planetary computing. Billions of people will be wirelessly interconnected, and the only way to achieve that kind of massive scale usage is by massive scale, brutally efficient cloud-based infrastructure.

More on cloud computing and Structure 08

SAN FRANCISCO--Speaking at the Structure 08 conference here, Sun Microsystems CTO Greg Papadopoulos predicted that by the beginning of 2010 the majority of systems sold would be for Web, high performance computing and software-as-a-service applications. "We are going through this phase change in computing in a big way," he said. He made a similar prediction last year.

Papadopoulos also advocated a free market in which all interfaces and formats are based on open standards; customers own their data, relationships, and metadata; and customers can extract, synchronize or purge their data unilaterally. This echoes recent efforts to promote openness and data portability.

Papadopoulos acknowledged that the nirvana of every customer or user in charge of their own data that lives in the cloud has challenges. Today, users cede control of their data to service providers like Google, Facebook, Microsoft, Yahoo, and others. It's not as easy for users to manage and move their data as it should be, which means users are generally stuck with the user experience and monetization schemes of the host sites. "It's proprietary systems all over again," Papadopoulos said. Over the last several years Sun has differentiated itself proprietary vendors, focusing on free open-source software and open standards.

Further out into the future, Papadopoulos expects that the technology infrastructure industry will be similar to the energy industry. In past presentations, he has called this transition the Red Shift.

Papadopoulos has predicted a "neutron star collapse of data centers," meaning at some juncture it won't make sense for businesses to build their own data centers. Instead they will contract for computing resources from hosting providers who bring "brutal efficiency" for utilization, power, security, service levels, and idea-to-deploy time.

There will be a grid of a half dozen very large cloud infrastructure providers and a hundred or so regional providers, Papadopoulos said. It will also look more like the banking world, he continued, with customers willing to trust the service providers with their private data as they do banks with their money. It's a question of when, not if, this scenario will occur.

Papadopoulos also laid out a map (see below) of the current universe of cloud computing in terms of increasing virtualization and consolidation across various categories: processor, operating system, language, and application services. Over time, the categories will fill out more especially as more languages and applications services or platforms rise up. Papadopoulos pointed to two Sun projects, Dark Star and Project Caroline. Dark Star is about software infrastructure designed to simplify the creation massively scalable online games, virtual worlds and social networking applications. Project Caroline is a hosting platform for developing and delivering Internet-based services. It's not clear why the Sun research projects are positioned at the far right on the chart, and players such as Google, Joyent, and Rackable are missing.

Click here to see more stories from the Structure 08 conference and on cloud computing generally.

Click here for more from GigaOM on Structure 08.

SAN FRANCISCO--During a panel discussion at the Structure conference here Wednesday, various representatives from the cloud-computing world offered their views. Panelists included:

• Christophe Bisciglia, senior software engineer, Google
• Jason Hoffman, founder and chief technology officer, Joyent
• Tony Lucas, CEO, XCalibre Communications
• Lew Moorman, senior vice president of strategy and corporate development, Rackspace
• Geva Perry, chief marketing officer, GigaSpaces
• Joe Weinman, VP of Strategic Solutions at AT&T

The panelists agreed that there will be open and proprietary, as well as specialized, cloud platforms. The discussion got a little heated between Google's Bisciglia and Joyent's Hoffman on the subject of open platforms and Google's BigTable software for distributed data storage.

"The question is, is it about selling your soul? You can't leave," Hoffman said during the panel, referring to Google's App Engine and cloud-computing platform. "There's been a lot published on what an open, loving cloud should do. We should give people real assurances that the cloud is a good place to be."

During the panel, Bisciglia said people can build a better mouse trap and compete with what Google offers. "When we publish something on BigTable, it is not to say that it is a lock-in, but it's our attempt to say that this is something that worked for us," he said.

"If your data is in Google's BigTable, you can't pull it out. You can't install it on your own hardware or leave. You have big brother telling you everything will be OK," Hoffman told me after the panel concluded. "One solution is that Google should provide nice export tools, but that doesn't solve the problem of where you run it. If I were a big enterprise company, I might want to run BigTable on my own hardware. If Oracle had the equivalent of a Google App Engine, a customer could run it on their own or someone else's hardware. What if Facebook started on Google App Engine? They would be stuck on Google."

Joyent is a David facing at least one Goliath, and its livelihood depends on an open-infrastructure approach. It doesn't have the market power to create its own standards. The company is doing 5 billion page views on month, which includes about 25 percent of third-party Facebook application pages, according to CEO David Young.

Joyent is working on a cloud-computing standards initiative called Cloud 9.

"We want to make it easy for people to leave," Hoffman said, adding that application programming interfaces should not hard-code server provider names into APIs.

"We need to interoperate just like the electrical grid," Young said. Google's BigTable and Amazon's SimpleDB are not pushing standards, which are needed to move things forward."

See GigaOM for more coverage.

Click here to see more of CNET's stories from the Structure 08 conference and on cloud computing generally.

In the early morning at Structure 08, AMR Research's Jonathan Yarmis described various tech trends around cloud computing. Mendel Rosenblum, a founder and technical lead behind VMware, outlined the role of virtualization in data centers.

Now Werner Vogels, vice president and CTO at Amazon.com, is talking about why Amazon is in the cloud computing business, how it got there, and why customers should want it. Instead of every company or developer doing the heavy lifting, dealing with the "muck" as Amazon CEO Jeff Bezos likes to say, Amazon opened up its software-as-a-service stack (Amazon Web Services) and infrastructure (Elastic Compute Cloud, S3, and SimpleDB) to external parties.

I've heard the Amazon story many times, but Vogels offered a few new tidbits, such as S3 is storing 18 billion objects and how Amazon thinks about building to its 1,000 services.

"Amazon built these services internally as tools, not as a framework. Each team can use whatever development tools they need. Infrastructure services need to be very generic and people can switch to competing services internally," Vogels said. For example, users could work with Amazon EC2 and a different storage service than S3.

Vogels outlined the core objectives and principles that cloud computing must meet to be successful:

Security

Scalability

Availability

Performance

Cost-effective

Acquire resources on demand

Release resources when no long needed

Pay for what you use

Leverage other's core competencies

Turn fixed cost into variable cost

Vogels noted that cloud computing is in its infancy, but it's not difficult to see the broad outline of how it will evolve. Nick Carr's book The Big Switch tells the story.

Live coverage of Structure 08

Click here to see more of CNET's stories from the Structure 08 conference and on cloud computing generally.