Twitter puts new limits on API calls: Who's affected

Twitter on Tuesday announced a new limit on third-party access to its service via its application programming interface, or API. Later this week, according to Twitter developer Alex Payne, the Twitter platform will limit API calls from a single IP address at approved ("whitelisted") Twitter services to 20,000 per hour. This will affect services that use Twitter APIs in bulk for non-messaging functions, like managing followers. Normal calls to read and write status updates are not part of this change.

There is already an API limit in place for third-party applications that give users general access to their Twitter accounts and message. AIR apps like Twhirl and TweetDeck, and iPhone apps like TwitterFon, can't hit the Twitter service more than 70 times an hour. Twitter has been known to lower this limit when the service gets overloaded. But for most end users, the limit is not noticeable. Twitter client apps are generally configured by default to access the Twitter services far less frequently than Twitter allows.

But for developers who are working on apps that extend Twitter in new ways, the new limit could have a serious impact. Services like SocialToo, which does follower management for Twitter users, rely on the Twitter API to gather data on behalf of its users. (Another product that might be affected: Mr. Tweet.) The problem, according to SocialToo developer Jesse Stay, is that for some of its individual users, getting the basic information they need requires not one API call, as they would prefer, but rather hundreds, due to the way the Twitter platform functions.

The complaints, then, from developers, are two-fold. First, as Stay says, "Why develop for the Twitter platform any more if we know we can only grow to your limit?" Related to that are complaints about the Twitter API being inefficient. It shouldn't take a hundred calls to gather one chunk of data, developers say.

For Twitter's part, Payne says that Twitter needs to put a throttle on API access to keep the service available to all developers, and furthermore that its limit will affect fewer than 10 applications (see Is Twitter Strangling its Famous API? on ReadWriteWeb).

The Twitter back-end has had periods of not keeping up with the popularity of the service, and this new limit is clearly an attempt to get ahead of the problem, even if it annoys a few developers and hobbles some services.

What I find most surprising is that Twitter is not using this issue as an opportunity to make partners, instead of enemies, of its developers. There are several ways the company could do this, from asking developers to help them re-cast the APIs so they work for more types of applications, to passing along the cost of high-volume API use to the developers of the apps (which would also force the app developers to come up with business models of their own).

As long as Twitter keeps its service free for developers, it is essentially asking for trouble. Developers will continue to build apps that add value to users but stress the platform--in the absence of constraints, why should they do any different? Limiting access to the platform by installing a governor on the engine will initially throttle developers' creativity, although as necessity is the mother of invention, it may also lead to even more clever solutions for improving the Twitter experience.

The logical, eventual outcome of this change should be an improvement in the Twitter API that allows the follower-management products to work more efficiently. However, Twitter developers are already working on even higher-profile updates to improve the platform's services. As the recently-updated developers' Wiki says, eagerly anticipated features like OAuth support and the open "fire hose" feed of status updates are scheduled to go into test this month or next.

[citizencontact]

Hmmmm. A limit on the use of the APIs points out a major problem for those who use APIs. Actually, much of what the APIs do can be done with actual Twitter pages. For example, a Twitter tweet has a unique ID in the form of a URL (e.g. http://twitter.com/citizencontact/status/1136538073 ) and it has a simple XHTML schema (xquery for the tweet w/in page is doc(http://twitter.com/citizencontact/status/1136538073)//span[@class = "content-entry"] ). And figuring out equivalents for most of the APIs using the actual XHTML pages is not difficult. And there is no limit.

The real answer, I think, is to trash most APIs and use web pages or XML pages with XSLT, so everything is human readable and machine readable/processable. The main thing is for Twitter to fully document their schema for their XHTML pages (or someone else). XQuery tools and URL discovery techniques (like RSS feeds) or search.twitter.com combined would allow for great apps w/o APIs.

Daniel Bennett
http://advocatehope.org/tech-tidbits/theory-of-the-web-as-one-big-database

[JerellKlaver]

My take: Twitter is beginning to stand its Data ground, and folks dont like it. They want FREE Twitter data, and forget that it is Twitter's Data to begin with.

[EricMarcoullier]

Jerell -- most of the affected developers have come out very strongly in support of a freemium model that would enable them to pay for more access to data.

For instance: http://blog.socialtoo.com/2009/01/21/twitter-limits-potential-app-growth-how-this-hurts-our-users/

[jeffbonnes]

I was under the impression that Twitter was already throttling their API to white-listed parties in a fashion. When we were testing in November 08, they were only allowing concurrent connection per IP Address. I always assumed this was done at the firewall or load balancing level. As each request as taking a while (say .2 seconds) to complete, the most you could get per hour per IP was about 18,000.

[WeCanDoBIZ]

When contemplating revenue models for Twitter I have often thought that charging third party developers to use the API could be one. So I read this with interest.

Has this move revealed their strategy for revenue generation?

Ian Hendry
CEO, WeCanDo.BIZ
http://www.wecando.biz