Outside the Lines

In this week's EIC Squared podcast, ZDNet's Larry Dignan and I talk about how the economic crisis will impact the tech sector. Both the House and Senate have passed the bailout package, but the legislation doesn't mean that tech or any other industry sector will reverse the downward spiral. Tech companies and financial analysts are rapidly cutting estimates to prepare for a potential nuclear winter in the global economy.

We also discuss Microsoft's forthcoming moves into cloud computing and the state of citizen journalism following the fake Steve Jobs heart attack story that showed up on CNN.

Microsoft is applying its tried and true formula of creating software platforms that can attract millions of users and developers to the hosted applications world. It will be the next major frontier for Microsoft to conquer, competing with companies such as Amazon.com, EMC, Google, IBM, and others. And it's safe to bet that Microsoft becomes one of the major players in the cloud. More to come at Microsoft's PDC event later this month.

On this week's EIC Squared podcast ZDNet's Larry Dignan and I discuss Google's latest disruption in the Web 2.0 field, the Chrome browser, as well as Apple's product launch event on September 9. In addition, Larry explains his idea that Dell and Salesforce.com could merge. Dell is trying to be more of a software company and is using the Force.com platform, and Salesforce.com is a major Dell customer. But, it's unclear how Salesforce.com, its shareholders and customers, would hugely benefit from a union.

SAN FRANCISCO--Standing 52 stories in the air at the upscale Carnelian Room in the Bank of America building here, executives from Dell, Facebook, and Salesforce.com discussed the meaning and use of the latest technology buzzword, cloud computing.

The sky was blue and cloudless, but it didn't adversely impact the atmosphere of what turned out to be a Dell marketing event. It was pitched as an announcement about a partnership that involves "the next generation of cloud computing."

You might recall that Dell is the company that owns the URL Cloudcomputing.com, and made a failed attempt to trademark the phrase. Earlier this month, the United States Patent and Trademark Office rejected the company's application. Dell marketing head Andy Rhodes wasn't willing to comment on whether Dell would appeal the USPTO decision.

Despite the cloudless sky, the speakers offered genuine insights into cloud computing, an umbrella term for "hyperscale" computing that covers everything from delivering compute services like a power utility delivers electricity, to simply hosting applications off-premises (see also software-as-a-service and on-demand computing).

Event host Forrest Norrod, vice president and general manager of data center solutions at Dell, defined cloud computing as an economic enabler for applications, not just for single applications but for platforms-as-a-service, such as Salesforce.com. He emphasized the economies of scale advantage that cloud computing has over client/server and previous generations of infrastructure deployment.

Dell is currently a cloud computing arms supplier to companies such as Facebook and Salesforce.com. "Dell is focused on early adopters and large customers, about 50 worldwide, to provide optimized servers, storage, and data center infrastructure," he said. "Cloud computing is still an emerging market, with standards across the framework and software stack still emerging. We are trying to promote an ecosystem to build the software stack on top of the infrastructure. You will gradually and judiciously see us add capabilities up and down the stack.

Norrod pointed to recent Dell acquisitions--Message One, Silverback Technologies, and Everdream--as examples of Dell's focus on software, not just the hardware piece. Increasingly, both Dell and HP are building out their software stacks to compete with Sun and IBM for providing highly automated data centers running commodity hardware optimized for cloud computing.

Jonathan Heiliger, Facebook's vice president of technical operations, had some praise for Dell. "Dell is doing the most aggressive things possible to optimize for cloud computing," he said. "We think Dell is perhaps the furthest along and we see them as a thought leader." Facebook has more than 10,000 servers, Heiliger said, and it's safe to assume any of them come from Dell.

He noted the price of hardware is not the biggest issue. Vendors can even sell hardware at a loss or at a fixed margin cost to get the initial business. "What we have seen in the landscape is that most server providers are trying to provide Lexus quality products at a Toyota price. We are looking for Scion products at a Scion price," Heiliger explained. "(Vendors) have to be creative around power and airflow optimization. The cost of operating the hardware is key; you have to take down the operating cost, not just the server cost."

For Heiliger that means bare-bones servers. "We don't need fancy graphics chips and PCI cards," he said. We need one USB port and optimized power and airflow. Give me one CPU, a little memory and one power supply. If it fails, I don't care. We are solving the redundancy problem in software." Blade servers are not ideal, he said, because of the higher cost and proprietary lock-in that come with the lack of a standard chassis.

Check out the video interview I conducted with Heiliger about managing infrastructure hypergrowth as Facebook adds 250,000 users per day.

Claus Moldt, vice president of technical operations at Salesforce.com, offered similar comments to the previous speakers. The company is phasing out Sun equipment and standardizing on Dell servers (Dell is a customer of Salesforce.com). Salesforce.com has two data centers in the U.S. and one due to go online in Singapore later this year. Moldt said his biggest challenge is capacity planning, making sure that as customer usage patterns change, the Salesforce infrastructure can adapt instantly.

Dell is betting big on cloud computing to boost its enterprise footprint. At this point, Dell doesn't have plans to build its own cloud to provide hosting for external applications, Norrod said. But, there may come a time when being an arms supplier won't be enough for Dell to be competitive. In addition, selling bare-bones servers can't be much of a high margin business, which is why Dell is moving more into software and services. Norrod said Dell's cloud computing efforts have been a large component of Dell's recent market share growth. Dell's second quarter earnings due tomorrow should give a more precise indication of the impact of the cloud on company's business.

Sridhar Vembu, CEO of AdventNet, is not afraid of going up against Microsoft Office or Google Apps. He is also the CEO of Zoho, which recently announced that it had achieved 1 million registrations (between 300,000 and 350,000 log on to the service monthly) for its cloud-based set of productivity applications. Vembu is now making a financial case that Zoho is better positioned than Google to take on Microsoft in the upcoming office suite sweepstakes.

Vembu's analysis is based on a comparison of revenue per employee and profit per employee metrics. "The gap in revenue per employee between Google and SAP and Salesforce.com, for instance, indicates that Google would more likely be more interested in what eBay does or in monetizing YouTube than in Zoho or Salesforce.com's barely profitable business. Companies invest in what generates the best return on investment," Vembu explained to me.

In an e-mail explaining his financial analysis, Vembu wrote:

We simply don't believe Google has the rational business incentive to go deep into the business/IT software category. The lower revenue and profit per employee figures would be tolerable if there were huge growth opportunities there; but when very successful companies like Adobe and Intuit pull in revenues well shy of a Yahoo, when even the enterprise software leader SAP is the same size of Google (Google makes more in profit per employee than SAP makes in revenue per employee), it is fairly clear this market is not going to make a material contribution to Google's growth and profitability objectives. So what is Google's plan here? It is fairly obvious they are in it to put Microsoft on the defensive on its home turf, so that Microsoft's offensive capability in the internet is diminished. It is also perfectly clear why Microsoft wants to be an Internet player--as Google has shown, it is a higher margin business even than its monopoly-profit core business.

"Google's margins are a once in a lifetime occurrence, and Google will move in that high-growth direction--that's why Microsoft is so desperate about search. It has a higher growth rate. We are more worried about Microsoft than Google. Microsoft will address the Internet, but pulling down Office margins is a challenge for them. No company peacefully accepts a lowering of margins," Vembu said. "Our intention is to help erode Microsoft's profit margin, coming in from below." Zoho has built a more comprehensive suite of cloud-based apps than Google or Microsoft, and most of them are currently free to users.

Vembu cites the cost of sales and support as a drag on revenue per employee and profit per employee. "If salesforce is a proxy, it would be difficult for Google to justify the investment. More costs are associated with support than in R&D, even with on-demand software. The moment you have paying customers, the expectations are different, and Google is finding that out with recent Gmail problems," Vembu said. In addition, he noted that selling into small- and medium-size businesses is difficult, but the margin is higher than for large enterprise accounts. Adobe Systems and Intuit, for example, have more revenue per employee than Oracle or SAP.

Zoho's revenue per employee is mostly nonexistent given most of the Zoho suite is currently free and not-ad supported. Vembu estimates Zoho's revenue per employee will be in the $200,000 to $250,000 range when the revenue spigot is fully turned on at some undetermined point.

While Zoho behaves like a scrappy start-up, it is well-funded by India-based parent company AdventNet, which develops enterprise IT management software. AdventNet has 900 employees and is profitable, according to Vembu. "One of the privileges we enjoy as a private company is to not disclose revenue/profit numbers, which lets us do the kind of analysis on competitors they can't do on us," he joked.

The problem with Vembu's logic is that Google has an enormous pool of cash to invest in improving the economics of business and consumer productivity software suites. And, part of being a software company is having multiple and adjacent revenue and user data streams. Microsoft is a highly profitable software company with many adjacent divisions. Google Apps won't be as profitable as search, but it will be profitable and ties users into the Google platform and monetization engine.

If Google can attract consumers with its apps, gaining entry into small- and medium-size business won't be a huge profit-sucking sinkhole of sales and marketing. The search giant claims that more than 500,000 businesses and schools have signed up for the free and $50 per-user-per-year Google Apps. According to Dave Girouard, head of Google's enterprise division, the Google suite has about 10 million active users. Google can afford to invest in building the the market for Google Apps, and Microsoft will be forced to alter the economics of its Office business as cheap and capable cloud-based suites, with offline capabilities, gain traction.

What does that mean for Zoho? Run faster and hope that Google and Microsoft move slowly.

On this week's EIC Squared podcast, ZDNet's Larry Dignan and I discuss this week's big stories. It was a busy week on the search front. Adobe is providing Google and Yahoo with Flash Player technology that allows their search engine crawlers to find and index SWF content, including Flash "gadgets" such as buttons or menus and self-contained Flash Web sites. It's good to make more information accessible via search engines. However, Microsoft has been silent on whether Live Search would index Flash content.

In addition, Microsoft bought Powerset for about $100 million to enhance its search platforms. It's not a substitute for acquiring market share via Yahoo Search, but it provides a foundation for making the search experience far more compelling and precise in fewer clicks.

Of course, the Microhoo drama continues this week with the latest rumors. Larry is ready for this opera to be finished.

Finally, we discuss a judge's ruling in Viacom's $1 billion copyright infringement suit against Google and YouTube.

U.S. District Judge Louis L. Stanton ruled that records of every video watched by YouTube users, including login names and IP addresses, should be given to Viacom's lawyers. Larry said it was like combining the worst aspects of a fishing expedition and a witch hunt. Viacom is maintaining that it won't look at personal data and Google is asking for time to anonymize the information. If Judge Stanton's ruling stands, the last shreds of personal privacy on the Web could be thrown out the window.

A few weeks ago I talked with Jonathan Heiliger, vice president of technical operations at Facebook, about the challenge of innovating quickly and building stable infrastructure while 250,000 new members are added to the social network every day. Check out the video on ZDNet.

Q: You've been at Facebook, I think, for about a year and it's been quite a ride I guess, scaling up from zero in 2004 to over 80 million today. How do you keep up with that hyper growth?
Heiliger: You're absolutely right--we've had a lot of growth. We add over 250,000 users every day, and that means a lot of infrastructure, a lot of servers, and constantly looking at new processes and looking at how we're doing things and ensuring that we're doing things the most efficient way possible, not just for delivering all the content to our users but to stay on top of what it costs to run the site.

How do you stay on top of the cost in terms of the kind of equipment you buy and how you work with the vendors? How do you prioritize those things?
Heiliger: One of the things we recently did was we ran an RFP process for the servers we buy from vendors and essentially did a bake-off with a number of different people looking at building servers on our own. What we concluded from that process was to continue to buy servers from a couple of major OEMs (original equipment manufacturers), but through that process we were able to lock in prices today and carry those prices forward as all the commodity components costs drop.

When you're buying those servers, and I assume you're doing just a huge scale out of commodity servers, what do they look like? How are they configured?
Heiliger: We're pretty lucky in that we run a wide variety of applications, literally tens of applications on our own and hundreds of applications for our platform developers that use Facebook as a distribution mechanism, as a way of interacting with their users. But one of the reasons we're very lucky is our engineering team has selected to use PHP as the primary development language. That allows us to use a fairly generic server type. So we, with a couple of exceptions, have three main server types and run a fairly homogeneous environment, which allows us to then consolidate our buying power.

You're different from Google in the kinds of applications that you run. They are mostly running search queries, and you're running all kinds of queries and bringing back all kinds of data from the social graph. How is it different in terms of the way you build out your data center from the inside?
Heiliger: Google has a tremendous amount of information that they index and archive and present to users, but fundamentally if you go to Google and type in a search for a "tiger" and I go to Google and type in a search for a "tiger" we're going to see generally the same results, so they're presenting that same information to both of us. Facebook is a little different in that the context for our data is all social. When you look at your friends and their status updates and their photos and the notes they may have written, you're going to see one set of data versus if I look at my friends and their photos and their notes and status updates, and those tend to be non-intersecting sets of data.

So it's much more dynamic?
Heiliger: Much more dynamic data set--and what that means is it's caused us to do a bunch of different things relative to caching and relative to federating all of that data up amongst thousands of different databases so that as a user requests all of that information we're not using one particular server every time for different data.

You recently introduced a chat application on Facebook, and it seems like it took a lot of time to test it to make sure it could scale having all those simultaneous conversations going on. Could you give us a little background and color on how that came to be?
Heiliger: Chat is actually one of our most recent launches. It started as a hack-a-thon project, which is one of the things we do about every other month. People get together and work all night and pick a project they don't have time to do necessarily during the day. From the time it really germinated as an idea to the time it launched and was available for our entire user base, it became a more formal development project. One of the things we did as part of that was actually built a new back-end service to be able to deal with all of the millions of simultaneous connections that we persist for users.

One other thing I was reading up on some of the work you've been doing--you say that clouds don't solve single points of failure in your stack. What are those single points of failure?
Heiliger: Interesting question, and the notion you are referring to there was part of the talk I give in regards to cloud computing is just a panacea, and for a start-up or even a more mature start-up like Facebook, isn't the answer to solving failure points in an application. By that I mean the underlining infrastructure that powers an application is typically the result of, or the outcome of, how the application is originally designed and how users interact with that application. If an application is poorly designed or designed to constantly reference a single set of data, the underlining infrastructure is going to be the victim of that. Guys like myself in the infrastructure world have to figure out how to best make that work.

As someone who is in operations how much impact do you have on the application development to make sure that once it gets into the data center that it can work properly and scale and not have the kind of failures we're seeing with some of the new applications?
Heiliger: I think it's a constant challenge in any organization, particularly a fast-moving one like Facebook, where we want to iterate quickly and get product out in our customers' hands so we can get feedback on that product and continue to tweak and enhance it over time. We have one force that's moving in that direction, and we have another force that says we want to keep the site up, we want the site to be reliable, and we want the site to be fast.

So there's a fine balancing act, where everyone in management and everyone in both the engineering and operations department constantly just sort of works, interacts, and goes back and forth, figures out just how to make those trade-offs. Sometimes we err too aggressively on the side of innovation and iteration, and put things out on the site in perhaps a small quantity that may break the site or cause the site to slow temporarily. Other times we air on the side of conservatism, of not releasing new functionality or new features, and that then delays the sort of user gratification of having that feature or fixing that bug.

What are the challenges that you see--let's say you're at 80 million unique users per month, 250,000 being added per day and 50,000 transactions per second. What happens when you get to 500 million or a billion if you ever get there?
Heiliger: Hopefully, tremendous things. I think we can only look forward to those days.

But what are some of the bottlenecks or barriers you have to overcome to get to that kind of scale?
Heiliger: Some of the bottlenecks we're facing are how we scale this extremely distributed set of data. One of the challenges we have is figuring out how to make that replicated such that it can exist in multiple places around the world and we don't also have to bring users back to the U.S. or back to one of our data centers. I think it's a challenge that most Web sites tend to face as they scale, which is you start in one location with a single database and then you have to figure out how to grow from there, primarily driven by the amount of latency or the amount of time it takes to reach the site and interact with the site. Being able to replicate the data across multiple data centers and across multiple geographies allows users to not just read their data from a local version but write that data as well. That is one of our key challenges over the next 12 months.

As you learn more about building up this very large scale infrastructure do you ever see the possibility that a Facebook could be a service provider?
Heiliger: What do you mean by service provider?

In the sense that right now you're just running the Facebook application but what if a developer or user wanted to do something similar to what Amazon is doing, using your infrastructure to run their applications in the cloud?
Heiliger: Gotcha. So one of the values of Facebook is the Facebook platform. We have over 100,000 developers and several hundred applications that have over a million users using them. We've talked about perhaps opening up or further opening up the platform by offering compute power for those application developers. One of the steps we've already taken to improve that development environment and improve the experience for our developers is just to open-source our platform, which we announced just a couple of weeks ago as well.

After attending GigaOM's Structure 08, I came away with a cloud-computing hangover. Just trying to define cloud computing is daunting given all the hype and companies thunderclapping.

Today the research firm Gartner has jumped on the cloud computing bandwagon, proclaiming that it "heralds an evolution of business that is no less influential than e-business," and defining it as massively scalable IT-related capabilities provided as a service using Internet technologies to multiple external customers.

Yahoo just announced a Cloud Computing & Data Infrastructure Group, which will develop computing infrastructure that balances scalability with cost effectiveness. What was Yahoo doing before it created this group?

I prefer the way Sun Chairman Scott McNealy talks about cloud computing. Ten years ago he was calling it the "big freakin' Webtone switch." Following is how he described it in December 2001:

That is the server, the storage, the operating system, the monitoring software, the clustering, the alternate pathing, multiple domaining, dynamic reconfiguration--and then it has a mail tone, a calendar tone, a news tone, an app server tone, and a directory tone. It has all of the different features of a big freaking WebTone switch and allows you to create this big jukebox. You can buy that all complete. Or you have one throat to choke and you can buy it all through a service provider that is SunTone certified. Or you can do what many IT directors do and they go out and buy the telephone switch by buying the chip from Intel, the operating system from Microsoft, the disk drive from EMC, the Compaq power supply, the Oracle database, the Novell directory, the BEA app server, the SAP, ERP, and CRM from here, blah-blah-blah, this, that, and the other thing, a SoundBlaster card from somebody else, the anti-virus uninstaller from Norton, and then go bring in IBM Global Services to try to make the whole thing work. Buy the big freaking WebTone switch.

At that time McNealy was talking about how enterprises provision their data centers and user services. Now we are seeing Amazon, Google and others take their data center expertise and make it available to developers and companies. Enterprises will be slower to move to the cloud, but they will eventually get there. Software-as-a-service providers are flourishing, and increasingly enterprises are considering off-premises, hosted solutions.

In essence, we are at the beginning of the age of planetary computing. Billions of people will be wirelessly interconnected, and the only way to achieve that kind of massive scale usage is by massive scale, brutally efficient cloud-based infrastructure.

More on cloud computing and Structure 08

SAN FRANCISCO--Speaking at the Structure 08 conference here, Sun Microsystems CTO Greg Papadopoulos predicted that by the beginning of 2010 the majority of systems sold would be for Web, high performance computing and software-as-a-service applications. "We are going through this phase change in computing in a big way," he said. He made a similar prediction last year.

Papadopoulos also advocated a free market in which all interfaces and formats are based on open standards; customers own their data, relationships, and metadata; and customers can extract, synchronize or purge their data unilaterally. This echoes recent efforts to promote openness and data portability.

Papadopoulos acknowledged that the nirvana of every customer or user in charge of their own data that lives in the cloud has challenges. Today, users cede control of their data to service providers like Google, Facebook, Microsoft, Yahoo, and others. It's not as easy for users to manage and move their data as it should be, which means users are generally stuck with the user experience and monetization schemes of the host sites. "It's proprietary systems all over again," Papadopoulos said. Over the last several years Sun has differentiated itself proprietary vendors, focusing on free open-source software and open standards.

Further out into the future, Papadopoulos expects that the technology infrastructure industry will be similar to the energy industry. In past presentations, he has called this transition the Red Shift.

Papadopoulos has predicted a "neutron star collapse of data centers," meaning at some juncture it won't make sense for businesses to build their own data centers. Instead they will contract for computing resources from hosting providers who bring "brutal efficiency" for utilization, power, security, service levels, and idea-to-deploy time.

There will be a grid of a half dozen very large cloud infrastructure providers and a hundred or so regional providers, Papadopoulos said. It will also look more like the banking world, he continued, with customers willing to trust the service providers with their private data as they do banks with their money. It's a question of when, not if, this scenario will occur.

Papadopoulos also laid out a map (see below) of the current universe of cloud computing in terms of increasing virtualization and consolidation across various categories: processor, operating system, language, and application services. Over time, the categories will fill out more especially as more languages and applications services or platforms rise up. Papadopoulos pointed to two Sun projects, Dark Star and Project Caroline. Dark Star is about software infrastructure designed to simplify the creation massively scalable online games, virtual worlds and social networking applications. Project Caroline is a hosting platform for developing and delivering Internet-based services. It's not clear why the Sun research projects are positioned at the far right on the chart, and players such as Google, Joyent, and Rackable are missing.

Click here to see more stories from the Structure 08 conference and on cloud computing generally.

Click here for more from GigaOM on Structure 08.

Speaking at Structure 08, Debra Chrapaty, corporate vice president of Global Foundation Services at Microsoft, shed some light on the cloud-based infrastructure supporting Microsoft's online services.

Despite characterizations that Microsoft is stuck in the client/server world, the company is spending billions to apply the cloud, or server/client, model, where most of the computing happens in the cloud and some small amount on the client (offline support for applications). But until Microsoft Office and other applications are built for the cloud, the laggard characterization will continue to stick to the company's forehead.

Microsoft has one of the biggest collections of Web sites, with 550 million users, 2 billion search queries, and 10 billion page views per month, as well as 8 billion messages on Microsoft Messenger per day. The company deploys 10,000 new servers per month on average to keep up with demand, Chrapaty said. She broke down Microsoft's model for building infrastructure into a three-letter acronym.

The cloud is all about GET--Growth, Efficiency, and Trust, Chrapaty said. In terms of growth, data centers are a $300 million to $500 million investment. "You have to make every kilowatt count," she said, noting that Microsoft has 35 criteria, such as network egress, power, and available staff, to determine locations for data centers.

Efficiency involves tools for manageability, operability, and sustainability, which translate into cost savings. "It's nice to go to Steve (Ballmer) and say you can save millions of billions of dollars," she said. Trust is having the security, reliability, availability, performance, and familiarity with the local languages and markets, Chrapaty explained.

Trust is also the user community feeling that privacy will be respected as people live their lives on line. That is a challenge that every large site will have to grapple with long after technology issues are resolved.

Click here to see more stories from the Structure 08 conference and on cloud computing generally.

Click here for more from GigaOM on Structure 08.