Beyond Binary

Bill Gates may not be hanging around Microsoft's research labs 24/7, but his vision for going beyond the mouse and keyboard seems to be doing pretty well without his day-to-day oversight.

At a user interface conference this week, the software maker plans to present several research papers, including a number designed to take the multitouch interface used in Microsoft's Surface and expand it into new arenas.

Although Microsoft's tabletop computer is still in the midst of its earliest commercial deployments, the company is already hard at work trying to figure out where the technology can go next.

Andy Wilson, one of the Microsoft researchers who helped create the Surface, is among those presenting at the User Interface Software and Technology conference, which is being held in Monterey, Calif. He is set to talk about how the same kind of physics engines used in 3D games could help make surface computing much more realistic.

Although multitouch computing is a huge leap forward in making computer objects feel more tangible, the illusion is challenged because all touch is treated the same, unlike in the real world where we can touch lightly, or push, or grab an object.

While a child using Surface for the first time will tend to use his or her whole hand to interact with objects, adults learn to use just a fingertip because they quickly realize that essentially the computer is only recognizing a single point for each "touch."

"The problem with that is you are flushing away a lot of the subtlety," Wilson said.

But, if the physics engines were better, Wilson says, objects can be folded and twisted and even torn like a piece of paper.

"How can we enhance the interaction model so we don't fall into this trap of thinking of every contact as a discrete point?" Wilson said. In his paper, he suggests a few different interactions, showing how a user can grasp a solid object and interact with it (say rolling a ball), or fold or tear an on-screen piece of cloth.

Another team of researchers from Microsoft's Cambridge, England, lab is showing a technique called SecondLight that allows a surface computer to project two images, one on the computer's surface and the other at some other point in the air.

This one's a little harder to explain. Essentially, the surface of the computer is one that quickly alternates between a transparent display and one that catches an image. The projector is in sync with this alternating pattern and sends one image when the display is transparent and a second when it is not. The first image is projected above the device, while the second appears on its surface. Because the images can alternate faster than the eye can detect, both images appear to be constant.

Real-world applications
Among the potential applications for this would be gaming. Clear plastic pieces could sit on top of the game and become chess pieces or checkers or other game tokens as needed. Medical imaging could be another interesting use, where doctors could look at an entire X-ray on the main display and hold up a piece of paper to see a second image, perhaps a close-up or an earlier X-ray.

"We're actually bringing the display into the real world," said Steve Hodges, one of the researchers behind SecondLight.

Such a move also helps break one of the inherent limitations of current surface computing. "It's still bound to the surface," Hodges said. "You are interacting on the surface."

One of the nice things about the SecondLight approach is that although the technology is complex, the objects that interact with the computer can themselves be simple. "All the peripherals are very cheap, either bits of plastic or pieces of paper," said Sharam Izadi, another researcher on the project.

Microsoft is also presenting a round surface computer prototype known as Sphere, which CNET readers got a look at back in July.

Another touch research project is aimed at trying to record gestures without using the screen as the surface. Microsoft already explored one notion, dubbed LucidTouch, in which users could control a screen by moving their hand below the device. Microsoft tries a different approach in its latest project, dubbed SideSight. In this example, the device sits flat on a table, while infrared sensors on the side of the device can record gestures made on either side of the display.

Such alternatives are important for two reasons. One, on very small devices, there is often not enough screen real estate for a touch screen. Secondly, by their nature, the very thing being pointed at is blocked while someone is touching it, hampering the ability to be precise. Both LucidTouch and SideSight are aimed at, quite literally, getting around these issues.

"Across Microsoft Research, in different parts of the world, there's a strong theme of finding new ways of interacting," Hodges said. "These projects all relate and overlap at the edges."

REDMOND, Wash.--When it comes down to it, Microsoft's Sphere really is kind of like taking the Surface computer notion and squishing it into a giant ball.

"The basic design is really quite simple," Microsoft researcher Andy Wilson told CNET News in an interview last week. Like the tabletop Surface computer introduced last year, Sphere uses a combination of infrared cameras for input and a projector for output to create a multitouch computer. "The camera and the projector share the same optical axis by virtue of mirrors."

Unlike the tabletop, though, Sphere is a ball-shaped display (either 18 inches or 2 feet in diameter) mounted on a pedestal. Its 360-degree display makes it ill-suited to some tasks, but perhaps even better for some types of gaming and mapping uses. At this point, though, it's just a research effort.

As noted Monday, an outside crowd will get its first look at Sphere later Tuesday as Microsoft shows it to academics attending the software maker's annual Faculty Summit here.

Microsoft had pursued the notion of a spherical computer on its own, but concluded that the hardware work was too difficult to do by itself. Instead, it chose to go with technology from another company--Global Imagination--which already had a spherical computer display on the market for things like museum exhibits and marketing displays. Its product, known as Magic Planet, comes in sizes ranging from 16 inches in diameter to one that is 6 feet tall. It's made of an acrylic, specially coated to allow projected images to display clearly.

Although it didn't have to reinvent the, well, sphere, Microsoft did have a lot of work to change the way the software both senses and renders content.

"For one thing there are no straight lines," Wilson said. You don't move an object in a straight line so much as you rotate it around a sphere."

Games, maps, and secret stuff
As for uses, Wilson said that while a sphere is impractical for many things, it does have some neat characteristics that make it well-suited to certain tasks. For one thing, many people can view and interact with a sphere-shaped computer, each having a different, but equally valid view.

"There is no privileged view of the Sphere," Wilson said. "If you think about it in terms of multiple simultaneous users, that is an interesting property."

It's also interesting from the perspective that any one person can only see just under half the screen. "You can imagine scenarios that involve gaming would be fun." (Just imagine the board game Battleship, for starters).

Beyond gaming, Wilson said that fact could allow a sphere display to be used for someone to have a public section that others can see, but also a second side, with a "personal stash of stuff."

Indeed, that property might be useful for Wilson himself. His studio is filled with different surface computer designs, only some of which he was really ready to talk about. Still, he was gracious enough to let me in, though he pointed to some interesting prototypes that he said were not quite ready for public consumption.

Other potential uses are videoconferencing and mapping. In one of the Sphere's canned demos, it shows the Sphere rendering a 3D street scene and a user touching the device to start driving through the scene. It's kind of the opposite of being inside a 3D world, since you are viewing it from the outside, but still a very interesting application--interesting enough that Wilson has started talking with the Virtual Earth team about some broader cooperation.

For videoconferencing, Microsoft already has a 360-degree camera, known as RingCam, so a Sphere could provide an interesting display for that as well.

The shape of things to come?
But while he can rattle off several potential uses for such a product, Wilson says it's not that he sees sphere-shaped computers as the next big thing.

"We are interested in this because it has some very unique properties and no one has really explored it at this level before," Wilson said. "There are not product plans for this, of course. This is just one of many efforts to explore different form factors that are sort of a play off the Surface."

And some of the learning Microsoft has done applies to more than just spheres, Wilson said.

"This has been a really fun project and it has got us thinking about other form factors as well," he said. "It's given us confidence in thinking about non-flat surfaces, and redoing the rendering pipeline was one good nugget of technical work that we did."

Sphere got its start, in earnest, about a year ago. Wilson had a demo unit from Global Imagination sitting in his lab when one of his direct reports, researcher Hrvoje Benko, saw it.

"When Benko started, he saw this sitting in the lab...He sort of picked it up and ran with it," Wilson said.

Update:Some folks asked for video, so below is a video from the Seattle Post-Intelligencer. Benko has also put some video up on his Sphere project site. In the latter video, check out the demo of what the infrared camera is seeing, as well as Sphere Pong.)

Microsoft's Andy Wilson shows off the technology behind Sphere, a globe-like surface computer developed by Microsoft Research. A group of academics gathered in Redmond this week will be among the first outside Microsoft to see the technology.

(Credit: Ina Fried/CNET News)

REDMOND, Wash.-- A group of academics will be among the first people outside Microsoft to see Sphere, a spherical surface computer developed by Microsoft Research.

The sphere-shaped, multitouch computer is similar to the tabletop Surface computer that Microsoft announced last year after years in development. This incarnation, however, remains a project within Microsoft Research and the company has no current plans to bring it to market.

The university researchers are at Microsoft as part of its yearly Faculty Summit. Also at the event, Microsoft announced a series of tools for researchers, including a plug-in for Office that lets people embed a Creative Commons license directly into their Word, Excel, and PowerPoint documents.

The software maker also announced a hosted e-Journal service to allow online academic publications and conference proceedings to more easily be published.

As for Sphere, attendees will get to see that starting Tuesday as part of an exhibit hall at the event. However, I had a chance on Friday to sit down with surface computing pioneer Andy Wilson, who showed me some of the technology powering Sphere, which was developed in large part by Wilson's colleague, Hrvoje Benko.

Sphere shares much in common with the tabletop Surface that is now being used in places like AT&T retail stores. At its core is a projector that beams the "screen" onto a globe-like display. As with Surface, infrared cameras are used to sense input, although the ones in Sphere are less precise than those used in the commercial Surface.

Sphere can run the same kinds of programs as Surface, such as a photo-sharing application in which multiple users can rotate, stretch, and move pictures. Its spherical shape though makes it more practical for some uses, such as gaming and mapping, and less useful for others.

Bill Gates has talked about a vision for surface computing that stretches far beyond the high-end commercial applications of Surface and in several years' time would have many, many of today's surfaces becoming computerized, both at home and at work.

I'll have quite a bit more on Sphere and my visit with Wilson in a follow-up post.