Twitter search is broken

There's too much noise in Twitter Search.

(Credit: Screenshot by Rafe Needleman/CNET)

In the Webware 100 this year, both Twitter and Twitter Search are finalists, in separate categories. (Vote here and here.) I added the search feature of Twitter as a product because it's a research tool quite separate from the social/publishing network of Twitter itself. In some ways I think it's even more important.

But Twitter Search has been failing me. It's still good when the result set is small. Last winter, when I was looking for tweets on road conditions over the Siskiyou Pass, it was useful. But try using it to see what's happening on a popular topic (like "iPhone"). What a mess!

The problem with Twitter Search is that it's purely time-based. Even though you can create fairly complex and specific search queries with the Advanced Search feature, the results you get are not sorted except by time.

What we need is for Twitter Search to also take relevance into account when displaying results. What is relevance? Timeliness is a part of it--a big part. But let the sort order be influenced by some other data and you might get something incredible.

At the very least, Twitter Search could be more relevant if it (optionally) displayed only results from the people a user was following, or if it ranked more highly those tweets that had been retweeted or linked to from elsewhere on the Web. Or better yet, make the highest-ranked results those from users who are typically retweeted or linked to the most.

You can't discount the importance of timeliness, since Twitter Search is unparalleled at taking the pulse of the Web moment-to-moment. But as a research tool it needs more finesse. And while I realize that adding features like these (as options, please) would undermine the brutal simplicity of Twitter Search, they'd also make it more useful.

Google, of course, figured this out long ago. Search is about relevance. You have to determine what real humans are paying attention to if you want to give them information they can use. There's no universal solution to this, so Google constantly tweaks its PageRank algorithm. That shows how important ranking technology is.

I've been using Twitter Search a lot at work, and it's been by turns fantastic and fantastically frustrating. It's one of the reasons I'd love to see Google get hold of Twitter's data stream, either through a partnership or acquisition. Until then, for a few tips and links for more effective Twitter searching, turn the page...

Twitter Search tips

If you're searching on a hot topic, filter out the noisy retweets by appending your query with -RT.

Refine your Twitter Search.

(Credit: Screenshot by Rafe Needleman/CNET)

If searching for a person who's on Twitter, search for the username, e.g., @rafe. You can also search for tweets directly to or from people with to:@username and from:@username.

Use Advanced Search. There are very good options to refine a query, especially by date, which can be a huge help. You can also type in advanced queries directly. See the Search Operators page for help.

See also How to search Twitter smarter from CIO.

To find Web pages linked to from Twitter users, check out the beta Twitter search feature in OneRiot.

I use TweetDeck to monitor multiple searches at a time--very useful for tracking real-time news. Other multicolumn Twitter clients, like Nambu, Seesmic Desktop, and PeopleBrowsr, also allow you set up and monitor multiple searches. Sideline, a Yahoo experiment, is also worth a look.

And here's one that's not yet out but looks promising: Tweetprobe, which provides search analytics from the Twittersphere. It should be useful for marketers and brand managers. There will be free and pro versions, and I hope to get a demo version to try in a week or so.

[PacGamer]

You gotta love the "cooliphoneapps" spam. Twitter is getting bigger and bigger. Like most things that get bigger, the clutter grows as well. - Ex: You build an extension to your home. What do you do? You fill it up until nothing else fits in it. - Spammers will start entering into the stream, flooding it.

[mjconver]

Finesse? Twitter? C'mon, twitter is just noise. Why do I need to search through noise?

[pjhenry1216]

It'd be difficult to get real relevance however. Google builds relevance with web search because the web has so much more context. Imagine if every web page was limited to 140 characters. It'd be extremely difficult to get relevance out of that. Plus, with any equivalent techniques Google uses on the web, tweets that aren't extremely current will never be relevant. Therefore making it relevant to virtually everything else other than time would be moot because almost every other characteristic is so heavily influenced by time that I think the end results will be very similar. Those items you mention (sort by tweets most retweeted or users most retweeted, search only your friends) would also introduce another problem. Any time it refreshes, the entire order would most likely change as opposed to just adding a few tweets up top. If you wanna refresh but not necessarily lose your place of certain tweets or what not, this would ruin it. Also, can you really count on just retweets for relevance?

Honestly, I don't think twitter search has to change, I think how people use search has to change. If you're searching for "iphone" than what would you really expect? Would you even use "iphone" in Google search? Its way too broad even for a search engine that uses this whole relevance idea. Sometimes the user has to adapt to the tool because maybe the way the tool was used before wasn't always the best way.

[reneecassard]

Thanks for the thoughtful post!

Have you heard about www.tweefind.com? It?s a Twitter search engine which returns results based on rank, hopefully returning more relevant results and users on top.

Rank is calculated through several parameters. Creator of Tweefind, luca Filigheddu, lists them:

# followers
# following
# of tweets
# of RT he/she receives
# of replies
# of distinct users who reply
# of distinct users who retweet
# of RT he/she makes
# of links the user shares

For the full article, click on the link on Mashable: http://mashable.com/2009/04/06/tweefind-applies-google-magic-to-twitter-search/

Enjoy - and thanks for the post!

Renee

On Twitter: reneecassard
On LinkedIn: www.linkedin.com/in/reneecassard

[nwjerseyliz]

Twitter search is great time for real-time results. But for 90% of my searches, I only get search results from the past week even when I know that Tweets exist from weeks & months ago. For example, I participate in #FollowFriday (which started in January) but when I search for my Tweets, I just get the most recent one instead of the past three months worth. This happens all of the time with Search.

It's a mystery to me why Twitter Search gets so much praise when it clearly skims the surface and does search through the entire Twitter database of Tweets. I realize that is a huge task but when you provide the author and keywords, it shouldn't be that complex!

[crystal37]

I am suspecting that the database trouble might be affecting search lately, and that's why we can't search for tweets more than a month old. (At least, last I checked, it seemed to draw the line at one month.)

I'm really, really hoping this gets cleared up, and I have a Help request in the queue that I hope will get an answer someday. I said I'd rather get an answer this time than get swept under the rug, so here's hoping that will get me something. (I had complained about search results about a week before submitting this request, and it got misinterpreted and considered closed, as if nobody even read what I wrote.)

I have to disagree: the usefulness of Twitter, in my opinion at least, is that it is a real-time view of opinions and perspectives. It's not a research tool. The ubiquitous 140-character boundary makes it next to impossible to present an argument, especially in the context of search; most valuable research presented on Twitter is done via links to blog posts and articles (and shortened links, at that).

In fact, the only time I imagine I'd do research using Twitter would be if I were researching Twitter *itself*.

If I search for "iphone" on Twitter, chances are I want to see what people are saying *right now* about the iPhone. That, to me, is the strength of Twitter. If I search for "earthquake," I'll want to see the latest tweets about earthquakes; if I want to do research on the most devastating earthquakes in history, I'll go to Google (or Google Blog Search, or Technorati). The services provide different functions based on the type of content being searched.

Using an old media analogy, Twitter is the newspaper as Google is a card catalog. Trying to add a fairly arbitrary relevance criteria as a filter seems to defeat the purpose for me.

[EricTPeterson]

Rafe, have a look at http://search.twitalyzer.com where we are using Twitalyzer (http://www.twitalyzer.com) data to add influence scores and number of followers to Twitter Search results. A number of people have commented that Twitalyzer influence used this way is very similar to Google's PageRank.

http://search.twitalyzer.com

Eric T. Peterson
@erictpeterson
Twitalyzer
http://twitalyzer.com

[crystal37]

I like using twitter search to review my own tweets so I can quote them or point people to them, or so I can copy formatting from a past tweet (like when I do wine reviews). It infuriated me the moment I realized my past tweets are no longer searchable... and I'm really, really hoping it gets cleared up sooner rather than later.