Web Crawler

TwitVid, a free video host for Twitter, is launching two new, useful features on Monday. The first is a real-time search engine that will let users sort through videos both on TwitVid and YouTube. The other is an analytics engine that lets video owners know more about who is watching their uploaded clips, and where they're from.

Between the two, TwitVid and Twitter users are likely to get more utility out of the new search engine. Similar to what OneRiot has done with its own real-time search engine, TwitVid's approach takes into account how fresh the video is, along with whether it's been popular on social sites. Recently popular videos then get better real estate on the results pages. It's a big step up for TwitVid users, who up until Monday had no way to search through videos other users had uploaded on the site.

As for the new analytics tools, TwitVid breaks down some very basic information, including when people were watching your video. This is charted out down to the hour, day, week, month, and the total lifetime of that video. The service also shows where your viewers are coming from--or at least the top 10 locations. Included as part of that list are links to the top referrers, which can show you if your video got picked up somewhere.

TwitVid made its debut in March and competes with TwitVid.io (not .com), TwitLens, Twiddeo, Twitc, Posterous, and Tweetube--just to name a few.

Previously: TwitVid app lets you send iPhone videos to Twitter

The new analytics features let video owners see where their videos are being played as well as information on the people watching them.

(Credit: TwitVid / CNET)

New start-up Factery Labs is launching its first service on Tuesday, a technology called FactRank that can tear through Web pages and collect what it calls "facts." These are bits of information from each source page that Factery Labs' algorithm then organizes into an order of importance.

What this means for you is that developers will soon make use of the technology in third-party search engines or on Web pages to very quickly deliver reading summaries. This cuts out most (or all) of the parts you don't care about, while organizing the bits you might. It also manages to do all this in real time.

The FactRank technology was created by Paul Pedersen, who has a good background in search, including gigs at Inktomi, Google, and Powerset. CNET News met with him and co-founder Sean Gaddis (former Skype and eBay'er) on Monday to get a demo of how the technology works.

In a nutshell it goes like this: FactRank goes through each Web page or source (in whatever index it's searching from) finding semantic tip-offs like declarative sentences. It then cross references each of those against one another, surfacing some of the most relevant ones to the top, as well as factoring in the order of how they appeared. What the user then gets is a tidy list of statements, each of which is sourced and given a level of relevancy based on their appearances in all of the indexed source pages combined.

Whew. Got that? Great, here's an example of what it looks like in motion, as seen on a basic search for Sarah Palin on Twitter:

One of the Factery Labs example applications is a search engine that finds facts from Twitter source results.

(Credit: CNET)

Of course, one of the problems with Factery Labs' approach across multiple sources--be it Twitter, or multiple URLs is accuracy; like how can it realize something like The Onion is not the same as the Associated Press?

The short answer is that it can't. Factery Labs can't determine the truth value of what it finds, nor will it ever. "It goes beyond any existing technology. And nobody knows how to do that. I mean, I don't even know how to do that--people don't even know how to do that," Pedersen said. "We are absolutely neutral. We have nothing in the system that has any bias in terms of anything. The only mechanism we maintain is egregious spam, the bad guys."

Along with maintaining a blacklist of these bad sites, FacteryLabs also keeps a list of good sources, or ones that continuously deliver. The more often an author successfully recommends a usable page, the faster they'll accumulate rank among the results.

What you can play with today
As for applying that technology to some consumer products, Factery Labs is launching with a handful of development partners, each of which has already built a tool that makes use of FactRank. The most notable one comes from Sobees which is using the service to add relevancy to Twitter and FriendFeed search results--something that's no small feat.

Users can do a search on Sobees' Silverlight-based Twitter client as usual, but there will now be a FactRank button that can sort through those tweets. It does a quick once-over of all of the results, and will filter the most relevant information to the very top. Included in each of its results is also a shortlist of the facts it finds on every page.

One of the first third-party apps to make use of Factery Labs is Sobees, which is adding its fact finding filters and relevancy tools to Twitter and FriendFeed search.

(Credit: Factery Labs)

Advanced users might find more utility in an updated version of Ultimate Info, an extension for Firefox that does a number of things with on-page data. Starting Tuesday, it will let users select links on a page, each of which gets the fact-finding treatment using FactRank.

In our demo, Gaddis used Ultimate Info on the front page of popular site Drudge Report, highlighting about six or seven URLs that were on the page, then running a FactRank query, which brought in its fact results in just a few seconds. As Pedersen explained, users could run something similar on a long article (or several long articles about the same subject), and FactRank's algorithm would be able to provide a fact summary in short order.

Not launching on Tuesday but where the company expects to see the most development is on mobile devices. "Our analysis shows that mobile devices are a prime target for this technology because the latency produces a lot of resistance in the browse experience," said Pedersen. Instead of a user just getting back a link dump of all the URLs it finds, the FactRank engine will go out, process those results, then deliver users with a summary of the best selection of facts--a move that will save the end user from having to wait for any extra pages to load.

If you want to give some of the third party Factery Labs tools a run, you can find them on the company's implementations section. There you'll also find a test search engine that's running off of Twitter's index.

Social news site Digg is experimenting with a new way to give upcoming stories a chance at the limelight with an experiment the company is calling Digg Trends.

Stories that begin to experience a heightened amount of user interest in the form of off-site sharing, user discussion, and of course Diggs, will be presented up at the very top of Digg's home page, as well as being spouted in a special RSS and Twitter news feed. Once at the top of the page, those stories have 10 minutes to get voted onto the front page as a normally dugg story, otherwise they're buried into Digg's dead pool. All the while a giant counter ticks down how much time the story has left.

Along with the countdown timer, Digg is also putting forward some of its outgoing traffic numbers. Users can see how many clicks a trending story has gotten from the front page. Normally, the only other traffic numbers you see on a Digg story is when you're on the source site itself, though users must have the DiggBar enabled.

Trending stories get just 10 minutes to prove their worth like any other front-page story. The company is also making available how many users have clicked to view the source content.

(Credit: Digg)

This new system is a stark difference from the somewhat nebulous promotion algorithm that exists for regular stories. Under the current system, stories have to earn their way onto the front page which involves standing out among an ever-growing pool of other upcoming stories. The company made this process a little more custom-tailored with the introduction of its recommendation engine, but it still requires that users actively visit that part of the site to see what's new. The new trending idea puts some of those stories up for everyone--and right on the front page.

In a company blog post about the new process, Digg's senior software engineer Kurt Wilms called it an "experiment," and said that it could change based on user feedback. Some Digg users have already voiced their opinions in the posting's comment section, citing that "bury brigades" (groups of highly opinionated users) could keep some stories from ever making it past their 10-minute window. The end result being that a story that could have legitimately made the front page on its own gets shut down before ever having a chance under the normal algorithm.

As with other new features, Digg seems to be rolling out Trends slowly, and to a small group of users. I'd expect it to show up for everyone in the next day or two.

Moodspin, a very eccentric social-networking add-on service has just had its app (link opens in iTunes) approved on the App Store. In case you're unfamiliar with the site, it currently works only on Twitter, and lets you very quickly broadcast how you're feeling to the people who follow you.

But here's the catch: it's not just text, the service goes in and makes small changes to your profile icon, adding that same mood-related emoticon to your photo, which in turn lets others see what you're feeling no matter what you just tweeted.

Is this useful? In the grand scheme of things, not at all. What's interesting though, is how it tracks that data back on Moodspin.com. Each time you update your mood status it's charted on a timeline, which if you're actively using the site, will show others what you were up to at various parts of the day. Each mood selection is also represented on your daily mood graph which you can compare to just your friends, or everyone else on the Moodspin service.

As an iPhone app Moodspin's initial offering is relatively simple, letting you select your mood in three taps, which can be fired off as a personalized update to Twitter. If there's enough room, it also (annoyingly) tacks on a URL to your Moodspin profile so that others can see what you've been brooding about posted as moods during the past 24 hours. There is however, no way to track your past moods from the app itself, something I'm told is coming in the next big update.

In the meantime, a version of the service which is due later this month will be implementing Facebook status messages and mood changes to user's Facebook profile photos which tend to be quite a bit bigger than Twitter's. However, there it will compete with a myriad of other mood apps that have existed since the launch of the Facebook apps platform.

Moodspin's iPhone app lets you set your mood in a couple of taps, the result of which is a new tweet and Twitter profile picture.

(Credit: CNET)

Imageshack's Yfrog, the image- and video-hosting service for Twitter, can now record videos from a user's Webcam. The recording tool also doubles as a way to take photo stills, either manually or with a five-second delay. These images are then attached to an outgoing tweet which can be penned right from the site.

The move differs from some recently released services like Twitcam and Camtweet which can record video as well as stream it out live for your followers to watch and interact with. On the plus side, Yfrog's implementation has very generous time limits, as my test video went well past the 40-minute mark.

For now, Webcam recording will remain a site-specific feature, and not a part of Yfrog's API, meaning third-party posting tools cannot take advantage of it. However a representative for the company told CNET News that that could change if developers are interested in integrating it into Flash-based video widgets. In the meantime, many developers have already integrated video into their apps using Yfrog's existing video uploading API.

Want to record a video of yourself to put on Twitter? You can now do that through Yfrog.

(Credit: CNET)

(Credit: Microsoft)

Microsoft was expected to release Twitter, Facebook, and Last.fm apps for the Xbox 360 as part of a system software update that went out back in August. It was decided that the apps should be delayed for a fall release, but CNET got a sneak peek of the apps on Wednesday from Ron Pessner, Microsoft's general manager of games for Windows Live.

The company is still mum on an exact release, despite marketing materials on the U.K. Xbox.com site briefly pointing to a November 17 debut. Pessner confirmed that the new apps were indeed set for November release, but he could not provide a hard date. In the meantime the company is releasing a public preview ahead of time for a select group of users who sign up to be a part of the beta test (you can sign up here).

What we saw of the three apps was impressive but in many ways extremely limited compared with their desktop counterparts. Microsoft has had to shoehorn the three Web apps into a system with a control scheme that does not implement a mouse, keyboard, or Web browser. Instead, all three make use of the Xbox 360 controller and the on-screen keyboard, or an attached USB keyboard (or Microsoft Chat Pad accessory which you can buy for $30).

This lack of the usual top-to-bottom controls found on each service's Web sites changes each experience considerably from what users are familiar with on their computers and mobile devices. This is made even more noticeable by the fact that the Xbox 360 still does not have a built-in Web browser. Never before has this been an issue, but stop and think for a second: are Twitter and Facebook as useful, or as interesting if you can't click on any of the links? We don't think so.

That said, there are some definite things to look forward to when the software update drops next month. Read our early impressions after the jump.

Stateless Systems, the creators of BugMeNot and PDFMeNot, have a new tool called Trendsmap that hasn't been designed to solve any productivity problems. Instead, it does just the opposite and serves as entertainment. It tracks trending Twitter topics by geographical location by combining data from Twitter's API and What The Trend. It then sticks it onto a Google Map where users can sort by city or general region and see trending topics in real time.

All of this information is organized into something resembling a tag cloud, which floats around without any specific, or pinpointed location within each city. Clicking on any of them pops up a small info box that aggregates the latest tweets, local and global seven-day histories of that trend's popularity, as well as some top-related news links that change depending on what's trending.

Trendsmap gives you a birds-eye view of trending topics on Twitter, per city, region, or worldwide.

(Credit: CNET)

Where the site shines though, is in letting you dig even deeper by giving each city its own trends page. Here you can cruise through info boxes without first having to find each tag, as well as see all of the trending charts stacked up against one another--something I think makes for a better experience. It also collects all of the related media like photos and videos in one single section (try giving it a spin for Las Vegas).

One thing the service doesn't do very well though, is serve smaller towns. This wasn't a big deal killer for me since I'm based in San Francisco, but if you want to use it for somewhere that's outside a major city, you're out of luck. This may simply be a limitation of how deep the data set is, but it keeps you from seeing trends starting up in smaller towns, which can be more interesting than in major cities.

See also Palm's Trendtracker, which lets you see trending topics not only geography but by time of day as well. We checked it out last week.

SAN FRANCISCO--Two new companies are launching products designed to get the news to users faster--and from a wider variety of sources. Both are in private beta and not yet available to the general public but were demoed live at the TechCrunch50 conference.

Thoora is a new tool that clusters and aggregates news. It offers people a way to track the latest headlines with a simple ranking tool, ordering incoming stories by "Web reaction." It uses a mix of sources, including Twitter messages, blog posts, and breaking stories from more traditional news outlets. These stories are then filtered and pushed to a front page as well as Thoora's category pages.

One of the things that factors into what ends up on Thoora's front page is real-time chatter. The company tracks how many news-related tweets there have been about that topic in the last hour, as well as "Twitter impact," which is a percentage of density about that topic per 500 messages across all of Twitter over the past hour. It also tracks things like blog comments and linkbacks.

Thoora tracks hot news topics across a variety of chatter networks including blog comments, tweets, and news stories.

(Credit: Josh Lowensohn/CNET)

Insttant, on the other hand, cuts out traditional news sources entirely and uses Twitter's public stream instead. It takes these tweets and turns them into an interactive news page that covers people, places, and companies, including a way to track trending topics and user sentiment. All of this goes on a front page, which can be reordered and personalized with topics the user wants to see.

One of the service's more interesting tricks is that it automatically creates profile pages for people containing links and interests based on what they've shared in their tweets. This also happens for trending news topics, which makes for a more in-depth news-reading experience, since you can drill down on any topic and see things like recent mentions, related news and links, and a history of how popular it's been in the past few weeks.

Instant's front page is made up entirely of real-time chatter.

(Credit: Insttant)

Related:
Yahoo's Delicious adds a little Twitter
Full TechCrunch50 coverage

Finding the hot conversation keeps getting easier, but predicting what the next big trend will be continues to be a crapshoot. Palm and Federated Media have teamed up to create a new tool called Trend Tracker that does its best to figure out, what in fact the next top trend will be by analyzing items that are gathering buzz.

The system is a mix of tools that can help spot popular URLs and trending topics before they hit it big. But it's more about organizing that data in a simple-to-parse format.

Included are the current top 30 trending topics on Twitter, which can be stacked up against each other to see what's pulling in the highest percentage of tweets. Each trend is represented over a 24-hour time line, where you can see how each particular trend has gone up or down in popularity.

Trend Tracker can give you a visual analysis of when each trend became popular, as well as its decline.

(Credit: CNET)

But 24 hours doesn't tell the full story, which is why the tool will soon expand to keep an archive that covers the last 10 or 30 days.

Along with the top 30 trends, Trend Tracker includes a "Pre Trend Watch" (emphasis mine) which tracks five up-and-coming trends that are about to break into the top 10 based on their velocity--the speed in which tweets on that particular topic are gaining in popularity. These are also marked in the trend archive with a little blue flag.

When I was looking at the tool last week, one of the most interesting things this picked up on was... Read more