Outside the Lines

As the tragic events unfolded in Mumbai, India, the Internet backchannel came to the foreground with messages, photos, and videos from the masses using Twitter, Flickr, YouTube and so-called citizen reporting sites such as Global Voices, as well as CNN and NDTV.

The terrorist attacks have left more than 100 dead and several hundred wounded in Mumbai, the country's financial center.

In major disasters, Twitter has become a conduit for real-time information and conversation.

As you would expect, the flow of information has been chaotic and potentially unreliable, which presents some problems, especially for those with family or friends at risk. A few posts on Techmeme question the quality of Twitter messages, which are not easily verified or tracked. Mathew Ingram argues that unverified eyewitness reports may not be accurate, but they represent a "first draft of history."

It's true that messages posted to Twitter aren't verified in any sense of the word, and in many cases could be wrong, or could perpetuate misunderstandings or factual inaccuracies -- although I think it's worth noting that dozens of Twitter messages corrected the Marriott reports not long after they first appeared on Twitter. At the same time, however, I think he's blaming Twitter for something that occurs during every similar news event: in other words, unverified eyewitness reports. Every time there is a bombing or an earthquake or a tsunami, there are reports -- many of which appear on television and other "traditional" media outlets -- that turn out to be completely wrong.

Does that make those reports invalid? No. Obviously, no one wants a loved one to be worried by false reports. But at the same time, chaotic situations result in poor information flow -- even to the "professional" journalists who are working at the scene. First-hand and second-hand reports on Twitter are no worse. Should anyone take them as gospel, or the final version of the events? No. Obviously, at some point someone has to check the facts, confirm reports, analyze the outcome, and so on. News reporting and journalism are much more of a process than they are a discrete thing. But as I have tried to argue before, Twitter reports are a valuable "first draft of history," and that is a pretty good definition of the news.

Of course, as Mathew points out, reports need to be confirmed, and no one is going to put the Twitter genie back in the bottle. It's a very rough first draft. There has to be a better way to triangulate and confirm news reports, where you could verify that eyewitnesses are actually on the ground where they say the are. If you are getting multiple tweets from several people in the same area, the likelihood that the information is accurate would increase. Of course, they could all be spreading the same rumor, which happens in traditional media as well. Using video as the source material would make the information easier to confirm.

In any case, Twitter and other sources of citizen-generated information provide a continuous pulse of data that will eventually be harnessed, and integrated with traditional media, in ways that lead to more accurate and real-time comprehensive accounts of what is going on in this troubled world.

A trio of Twitter executives--Jack Dorsey, Biz Stone, and Alex Payne--showed up at the BearHug Camp, held at CNET headquarters in San Francisco, to answer questions about the future of the service, such as when the "track" feature would return. CEO Dorsey said that the acquisition of Summize and work on redoing the back end have made it easier to bring track back, but didn't provide any specific timetable.

The company has 24 employees and recently got $15 million in funding. Dorsey said that Twitter is taking a measured approach to introducing new features, and won't put out track until it can perform satisfactorily. "We are getting to a great place with scaling so we can get ahead of this tsunami," said co-founder Stone.

Twittermen: Jack Dorsey, Andrew Payne, and Biz Stone.

(Credit: Andrew Mager)

Developers attending BearHug Camp, including RSS pioneer Dave Winer, asked about getting the full feed from Twitter exchanges--the fire hose of data--which allows developers to create applications on top of Twitter's data and APIs. BearHug Camp moderator Steve Gillmor said grandiosely, "The question is whether access to this functionality will be granted in a critical moment in history." The Twitter trio didn't offer a specific answer at first. "We are not closing any doors, we are trying to open them," Dorsey said.

Stone added that Twitter just staffed the API development team with two new developers. Payne, the lead API developer at Twitter, said that the issues around giving developers access to the full Twitter data flow are business- and user-oriented. "We are trying to figure out appropriate terms and conditions around the fire hose. It's not necessarily monetary," he said.

He gave an example of developers using the Twitter feed in ways that make users uncomfortable, such as an update on Twitter that shows up on a third-party site with ads a user doesn't approve of. Payne said Twitter is now working with its lawyers to come up with the terms and conditions. For developers who want the full feed, "We have to finish the agreement so we can give out the feed in a proper way," he said. Fundamentally, it's about Twitter's business model: how the company gives away the fullness of its data and generates revenue to keep its VCs happy.

It's somewhat incomprehensible that Twitter has been unable to keep the service up and running. More than 10 years into the age of the Internet, with a huge amount of R&D publicly available about scaling Web applications, you would think that Twitter's engineers could figure it out.

A recent blog post from Twitter co-founder Biz Stone said help is on the way in the form of about $15 million in funding:

Twitter will become a sustainable business supported by a revenue model. However, our biggest opportunities will be worth pursuing only when we achieve our vision of Twitter as a global communication utility. To reach our goal, Twitter must be reliable and robust. Private funding gives us the runway we need to stay focused on the infrastructure that will help our business take flight. We will continue hiring systems engineers, operators, and architects, as well as consultants, scientists, and other professionals to help us realize our vision.

But the natives who love Twitter are getting restless, and they're losing faith. Twitter is chronically up and down, and key features, such as track and replies, disappear as company engineers try to save their patient from flatlining.

The father of RSS Dave Winer recently said, "Twitter, as it was conceived, was never meant to live."

"It's very possible with better engineering its architecture might have gone on for a few more years, but eventually it would have hit this wall, where there were too many people posting too many twits to too many followers. The scale of the system as conceived rises exponentially."

Check out Winer's Twitter "spewage" report to get an idea of Twitter's scaling challenge.

Allen Stern of CenterNetworks likes Twitter's simplicity and is willing to bet that Twitter can be stabilized.

"I don't believe Twitter is going to die, be killed or go for a suicide. Twitter is easy to understand and use. It's perfect for the mainstream. FriendFeed isn't. FriendFeed will do very well also for the set of users currently using it. I am not sold that there's mainstream appeal coming for FriendFeed."

The weekend conversation is pivoting on whether FriendFeed will replace Twitter as the new conversation hub among the digerati. Winer, who has a high authority rank on this topic, doesn't think that FriendFeed is the answer.

... before we all move to FriendFeed and think we've solved anything, this underscores the problem with putting all our eggs in one basket. We just move the problem into the future. FriendFeed may be able to scale where Twitter can't, but there are other problems with centralization, putting all your trust in a corporation, esp. one so young and unformed. Instead, we should start bootstrapping a decentralized Twitter-like thing immediately, building off the base of clients that connect to Twitter. It can connect to any service we want to connect to, and if one should go away, we do the thing the Internet does so well, route around the outage. I wrote about this, extensively, in early May.

As Winer predicts, Twitter, as a concept, is not going to die. An open platform for microblogging and broadcasting with followers has clearly taken hold. Just as instant messaging spawned numerous silos and a kind of standard in XMPP, Twitter's twist on messaging will go through an evolution that eventually leads to a common standard and stable infrastructure. The Twitter concept has been cloned (Pownce and Plurk), and it won't be long before Facebook, MySpace, or other big players figure out how to make following, followers, tracking, and summizing part of their services.

See also:

Steve Gillmor: Saving the FailWhale

TechCrunch: Twitter Conversations Come To A Screaming Halt; Users Simply Move To Friendfeed

FriendFeed and Twitter: Let it Be

Some perspective on Twitter and its brethren

What Twitter brings to the party

Former Googler and FriendFeed co-founder Bret Taylor in this video offers his views on Twitter and describes the new summarization feature coming to FriendFeed soon. Taylor said he was not interested in cloning Twitter, but in improving FriendFeed's communications tools. The next major FriendFeed improvement is an algorithm that processes signals from inputs, such as comments and "likes," to surface the best-shared items from a user's set of friends.

See also:

Gillmor Gang: Inside FriendFeed

Jeremiah Owyang: What FriendFeed's Micromeme means to you, brands and the Web

The obsession with the ups and downs of Twitter among my friends has generated a great deal of bloviation, including my own. On a slow news weekend, Twitter's performance problems are fodder for a bit of theater and for getting some daily keyboard exercise.

The image below is meant to bring some perspective to the Twittersphere. On one hand, Twitter navel gazing (or any other navel gazing) is a waste of resources in the context of what is going on in the world. On the other hand, Twitter and its brethren are becoming viable communications vehicles for spreading the "word" and images.

For example, I first learned of the recent tragic earthquake in China via Twitter messages from people I follow on the service. To be clear, Twitter is not the Holy Grail of communications services--it's an extension of instant messaging and technologies such as RSS. Nor are the 140 characters in a Twitter message a substitute for a blog post or news article. But a "tweet" can be a network amplifier, providing a brief snapshot, innervated by followers and the followed, that can be broadcast around the world in near real time.

Twitter and related services are currently noisy, spammy, unwieldy, overrated, and often unreliable. But over time, the core concepts will become an integral part of the Internet's communications fabric.

Lately the echo chamber of the blogosphere inhabited by the Gillmor Gang (of which I am a member) has been caught in a loop of Twitter-FriendFeed convulsions.

Steve Gillmor believes that Twitter is the communications medium of the future. Send out a message to your followers and track (when the feature is enabled) the loosely coupled conversation as it wafts deeper into the cloud. FriendFeed, on the other hand, aggregates feeds from Twitter and many other sources, creating an index of the content (gestures in Gillmorspeak) an individual chooses to share with followers.

Twitter's friendly API allows applications to be built on top of it (when the site is up), letting FriendFeed and other services tap into the Twitter stream. In addition, FriendFeed allows users to comment on the contents of the aggregated feeds and has "rooms" for discussions among a group of people.

Steve Gillmor makes the claim that Twitter is being strangled by FriendFeed and that his pal Robert Scoble is hijacking the conversation away from the unreliable Twitter site to FriendFeed. It's much ado about nothing. Users have the freedom to head to their communications medium of choice. The Twitter conversation stream isn't locked into a walled garden--tweets can flow like water into applications such as FriendFeed, Summize, and Facebook.

It's not clear precisely where this latest twist on instant messaging and feed aggregation is heading, but just let it evolve without the prejudice in its own Darwinian way. That doesn't mean to back away from criticism or debate, but to do so in the context of open networks that provide ways for individual users and groups to shape their online experience.

Steve Gillmor prefers the Twitter funnel, while Robert Scoble likes the FriendFeed blender, which can include Twitter streams.

With increasing attention, continuing outages and new funding of $15 million, Twitter is providing more insight into its operations. In a recent blog post, the company shared the technical problems it is facing and how it plans to make Twitter more stable.

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability. When we're in crisis mode, adding more instrumentation to help us navigate the web of interdependencies in our current architecture is often our primary recourse. This is, clearly, not optimal.

Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative - scrapping everything for "the big rewrite" - is untenable, particularly given our small (but growing!) engineering and operations team.

Providing this kind of disclosure will go a long way toward appeasing the frustrated fans, as long as the dialogue continues...and Twitter performance shows some ongoing improvements. For a translation of Twitter's outage explanations, see this blog by my colleague Charles Cooper.

On this week's EIC Squared podcast, ZDNet's Larry Dignan and I discuss the latest developments in the Microhoo saga.

It appears that Microsoft will do anything to keep Google from cutting a deal with Jerry Yang. In addition, we discuss the One Laptop Per Child XO-2 device, as well as the ongoing fascination with Twitter by the digerati crowd, despite the fact that the site is often out of commission.

Twitter and tweeting are rapidly becoming part of the lexicon, at least among the digerati who have discovered the jouissance of followers and following. Twitter hasn't unleashed a unique technology, but an inspired broadcast pivot on existing messaging models. As the generation that has grown up texting rather than e-mailing takes over the planet, Twitter and its ilk will go mainstream.

With Twitter, you have followers (those who subscribe to your 140-character-limited tweets) and following (those whose tweets you follow). As you can see from the graphic below, Twitter usage comes in all shapes and sizes.

At the top end of the scale, TWiT.tv's Leo Laporte has the most followers to date, with 32,728 (see Twitterholic for the top Twitterers list), followed by Barack Obama. You'll note that the digitally savvy Obama (the Obama camp, not the candidate himself) follows practically every follower it has, whereas Leo follows on 427 Twitterers.

The Obama camp is definitely not prolific--only 119 tweets to date to thrill and inform its more than 32,000 followers. Leo is more generous with 895 tweets, but the most energetic and prolific man of Twitter is Robert Scoble.

He follows almost as many people as those who follow him, and has showered his fans with more than 12,000 tweets to date. On Saturday, he scribed 103 tweets to his more than 20,000 followers.

Paying attention to the daily tweets of 20,000 to 30,000 people, even at a maximum of 140-characters per pop, isn't remotely possible unless you spend your entire day and night in the Twittersphere.

As with Facebook, you can collect thousands of friends and followers but only a small portion will matter. Leo's ratio of followers to following--32,728 to 427--is a more realistic and practical approach to using Twitter.

Search and filtering applications for Twitter help to reduce the Twitter overflow. FriendFeed and Alert Thingy (a desktop app for FriendFeed) vacuum up Twitter, Flickr and other feeds into a single, uber stream. Summize tracks Twitter conversations in real time.

Others are taking the Twitter concept into new areas. Seesmic has created a video equivalent of Twitter, and Pownce extends the Twitter model to file transfer and possibly music trading.

As a communications medium, Twitter is a self-promotional mechanism, as in the Twitter feed of CNET News headlines, as well as a vehicle for people to share random and often useless data, such as what you had for breakfast. Twitter can get noisy, but essentially it's an efficient broadcast and sharing conduit. For example, I first learned about the recent earthquake in China and ongoing developments from my Twitter stream.

Twitter is a complement, not a substitute, to blogging (also called "writing"). With its roots in SMS and instant messaging, the 140-character limit forces thought economy, saying more with less.

It's also having a side impact on blogging, implicitly training writers to write more succinctly, conserving bits even though they are in near infinite supply. A person's time and attention don't have infinite supply--getting to the point without rambling is a plus in our data-rich and continuously partial attention world.

One of the potential downsides of Twitter and other short-burst messaging apps is that context can be lost or it is very loosely coupled, which makes connecting the dots more difficult. On the other hand, if you consume a big enough supply, and the most salient, bite-size chunks of data, the bigger picture will come into focus.

The reality is that humans in the early 21st century will be required to process and buffer more discrete, loosely coupled bits of data than in the past of human history. Over the next few decades, more intelligence will seep into the network, filtering the overflowing stream for each of the 7 billion or 8 billion inhabitants of the planet and shaping more meaningful connections.

See also: What Twitter brings to the party