URL shortening is hot--but look before you leap

Twitter's dramatic rise has helped ignite an industry to shorten Web addresses to fit within 140-character messages. With the technology, though, comes a new handful of challenges.

Among the challenges are reliably connecting people to the Web sites they want to reach, keeping spam and phishing attacks at bay, and maintaining the service into the future.

Joshua Schacter, founder of Yahoo's Delicious site for storing and sharing Web bookmarks and now a Google programmer, summarized the issues in an April rant about short-URL problems. "I feel that shorteners are bad for the ecosystem as a whole," he concluded.

TinyURL's interface for creating short Web addresses.

(Credit: Screenshot by Stephen Shankland/CNET)

Until a remote future arrives when Twitter and the telecommunications industry decide 140-character messages are too short, though, URL-shortening services aren't going to go away. Fortunately, their potential problems can mitigated through careful use, and newer services such as Bit.ly are being designed expressly to avoid the pitfalls.

And even if some service falls by the wayside and stops functioning--well, welcome to the real world, where not all information is preserved.

"In the digital age, everything has a certain amount of bitrot," said Paul V. Mockapetris, who invented the Domain Name System (DNS) that serves as the Internet's address book.

Growing like weeds
URL-shortening services are abundant and becoming more so. They're usually designed with a priority on minimum character length, not easy reading: Is.gd, Bit.ly, Twurl.nl, Tr.im, Sn.im", Cligs, and TinyURL. If you want to see dozens more, Mashable has a long list.

And the traffic they handle is large. On a typical day right now, Bit.ly is used to create 5 million to 7 million shortened URLs each day, and it handles 25 million requests to expand them--and the growth rate is at a breakneck 5 percent to 15 percent week over week, the company said. Snipurl has delivered 53 billion since its inception. And TinyURL has a database of 293 million URLs.

URL-shortening services have been around for years--TinyURL was founded in 2002, and SnipURL, which also operates Sn.im and Snurl, in 2001. The services typically were used to keep long URLs from being split into chunks in e-mail, where line limits of 80 or fewer characters could break up a Web addresses.

So what's new now? First, Twitter, and second, shortening URLs is becoming an actual business--notably at present through the addition of "analytics" features that can let those who use the service see data about how many people clicked on links, when, where they're located, and the Web page where they found the shortened link.

TinyURL's funding today primarily comes from advertising on its Web page, but that's changing, said founder Kevin Gilbertson. "I'm working on something else that should increase that (revenue) quite a bit," Gilbertson said. He declined to share details at this stage beyond saying, "It will not change any functionality."

He's employed contractors, but with the new funding, he expects to hire full-time programmers and improve his computing infrastructure. Also coming is analytics. "We will be offering that sometime here soon," Gilbertson said.

Snipurl has been run as "a personal endeavor," said leader Shashank Tripathi, with just three employees, but the new climate has got him thinking about venture financing.

Getting analytical
Analytics are important for URL shorteners, in part because it helps the services break out of low-value freebie use to corporate accounts that need to track what's going on.

Bit.ly can tell you how many people clicked your short URL, where they clicked it, and which country they live in.

(Credit: Screenshot by Stephen Shankland/CNET)

"We have a bunch of commerce providers who have embedded Bit.ly into their systems. They're using it for tracking and understanding ROI (return on investment) on various promotions," said John Borthwick, Betaworks CEO and a Bit.ly investor.

Bit.ly is just shy of a year old. So why get started now when there already are dozens of alternatives? In short, to do it better, Borthwick said.

"At Betaworks, we had a series of companies who said there's a need for an URL shortener that was more scalable and reliable than the stuff out there today, that had real-time metrics associated with it, and that had an open API (application programming interface) so people could encode and read decodes through the API," he said. "So we said that's a problem we think we can solve."

The API lets third-party software such as TweetDeck or Twhirl call upon Bit.ly's servers to create a short URL. It also can be used to more easily show the destination URL hidden away behind the short URL.

URL shorteners also can give insight into hot trends by spotting sites people are sharing moment by moment. Bit.ly Now spotlights Bit.ly's top Web addresses hourly.

Grappling with reliability
Then there's the problem of reliability. URL shorteners add a new step to the process of retrieving a Web page, and when the service goes down, Web pages can become inaccessible.

"We've had some growing pains and some issues," such as a database replication issue in March and a faulty switch in April, said TinyURL's Gilbertson. And Snipurl's Tripathi said traffic spikes such as those from Michael Jackson's death can be tough to accommodate.

Compare that to DNS, an arrangement of impressive scale that converts a typed address, such as "slashdot.org," into the numerical Internet Protocol address that's actually houses the address, 216.34.181.48. There are primary "root" servers for DNS, but countless servers on the Internet mirror the primary address book or portions of it, providing broad protection against failure.

DNS is used not only every time a person visits a Web page--several times per page in many cases--but also each time an e-mail is delivered. And while it's had issues, it's generally been highly reliable.

The shortener companies are working on greater reliability for their services. And in fairness, their scale of operation, while growing, is vastly smaller than the DNS.

Still, there's a longevity issue. Shurl.net and URLtea.com no longer function, for example. It's easier to start a shortening service than it is to maintain it for perpetuity.

"Many knockoffs have come and vamoosed," Tripathi said, but he expects Snipurl to do better. "Unlike recent post-Twitter entrants into the space, our links have been around since 2001. While we cannot guarantee anything, we can tangibly claim to have been the best in terms of longevity."

Even without shortening services, URLs aren't guaranteed to last forever. Some "linkrot" is inevitable as companies come and go, services such as Geocities are closed, Net addresses are sold, and people decide not to pay the annual fee to maintain their domains. Here, Bit.ly can actually help by maintaining its own copy of the page.

Can you trust that link?
One of the problems with short URLs is knowing what you're getting into when you click them. Is that link really the fun video of the guy tripping into the lake, or is it site that will spam you or attempt to install malware? Is it really a warning from your bank about a bad transaction, or is it a phishing attack to try to fool you into parting with your password?

Clicking many regular URLs involves a leap of faith, to be sure, but not being able to see a "youtube.com" or "bankofamerica.com" name because it's masked by a short URL makes that leap even longer.

TweetDeck has a preferences option to show details of shortened URLs before you open them.

(Credit: Screenshot by Stephen Shankland/CNET)

"My thinking is that the No. 1 concern is the masking of dodgy domain names from user inspection, for example, those registered in .cn for seemingly English-language sites," said Vern Paxson, a University of California-Berkeley associate professor of computer sciences who focuses on Internet security.

Some services don't do much to help the situation. "Add adverts to your URL to get FREE Traffic," promises the LongURL.net site. "When you send this link to newsgroups, forums, Twitter, Facebook and other social networks, people will load source Web page with your own ad."

And SNVC asks, "Are you an affiliate marketer looking to cloak your affiliate links?...If you choose to check the 'Hide Real URL?' box then it will allow you to keep your shortened URL in the address bar by using a hidden-frame trick."

But avoiding the issues of opaque links is top of mind for many services. For example, adding the word "preview" in front of a TinyURL link will show a Web page with the destination URL expanded. Greasemonkey scripts and extensions let some browsers automatically show the expanded link. And Bit.ly's API permits the Tweetdeck application to automatically show the ultimate destination.

Many services also try to screen out nefarious Web pages through use of blacklists such as SpamCop or Google's Google Safe Browsing. There's brand value in being seen as a shortening service that can be trusted.

"With our brand, we're trying to create a relationship with our users where it's understood we're looking out for them," said Andrew Cohen, a member of Bit.ly's still-small team.

Clicking links from your friends involves trust--they're your friends, after all. But as Twitter becomes evolves into a global conversation, with tweets revealed through search and hashtag identifiers, short URLs from strangers become more common. The more companies use URL shorteners for analytics purposes, the more people will encounter those links outside of Twitter, too.

Fundamentally, URL shorteners are just the latest iteration an old problem.

"The trust issue...is not fundamentally different from other Web, email, and link techniques that are out there," said UC Berkeley professor Randy Katz. "It all comes down to context and being sufficiently aware not to blindly open anything suggested to you."