Beyond Binary

After taking Natal for a test drive, CNET News' Ina Fried wants to know when the rubber will meet the road. One game maker suggests the answer is late next year.

(Credit: CNET News)

Now that I have gotten a chance to try out Project Natal, Microsoft's gesture recognition technology, I have the same question as everyone else. When is it going to be on the market?

While Microsoft isn't saying, one game maker has spilled the beans. In its recent earnings conference call, game maker THQ said to expect it late next year.

"We have for example, Natal from Microsoft, a platform addition coming late next year," THQ chief Brian Farrell said on last week's conference call.

In an interview with me, also from last week, entertainment unit President Robbie Bach declined to offer any more details on Natal timing.

"I'm not planning on being any more specific today," he said.

In addition to the comments from THQ, Electronic Arts also indicated on its earnings call that it plans to support Natal.

Microsoft has also said it has plans for Natal that stretch well beyond just gaming or the Xbox. Chairman Bill Gates told CNET that it is a technology he sees moving onto Windows, while Chief Research and Strategy Officer Craig Mundie has shown some of the same gesture, voice, and facial recognition technologies as being key parts of the office of the future.

Project Natal is both great gaming and a great workout, as CNET News' Ina Fried experienced firsthand when she got to try out the technology last week.

(Credit: CNET News)

REDMOND, Wash.--One of the reasons that Microsoft got such buzz for Project Natal is because it is so easy to see how the technology could change the face of gaming.

But it's even easier to appreciate once you get a chance to try the gesture recognition technology yourself. When I was in Redmond, Wash., last week, I got a chance to do just that.

Playing Ricochet, a 3D breakout-like game, I found myself wanting to do whatever I could to stop the balls from passing me. It felt less like a traditional video game and more like I was a soccer goalie and an entire team was firing shots at me. (For a firsthand look, check out the embedded video below.)

It was both a lot of fun and a bit of a workout. Apparently, I'm not the only one who has noticed that.

"Since I started working on this project, I've lost almost like 10 pounds," said Kudo Tsunoda, general manager of Microsoft Game Studios and the creative director for Project Natal. "We're going to have the most in-shape development team you've ever seen."

The effort is important to more than just the waistlines in Redmond. Microsoft is counting on Natal to give an important bump to the Xbox 360, which Microsoft has said is only mid-way through its lifecycle, even though it has been on the market since 2005.

After Ricochet, I tried my hand at an existing driving game that had been connected to the Natal interface. And while my steering hasn't gotten any better than when I checked out a set-up from GestureTek earlier this year, Microsoft's technology is quite impressive. The steering and other controls were both intuitive and responsive.

I moved my foot forward to accelerate and backward to slow down, brake, and eventually reverse the car. To steer, I simply used my hands like a steering wheel.

Although Microsoft demonstrated Natal at this year's E3 trade show, the software maker hasn't said when the technology will be available. The company has said that Natal, which incorporates face, voice, and gesture recognition technologies, will be sold as an add-on to the current Xbox 360 console.

The effort to turn Natal from concept to shipping product has been something of a mini Manhattan Project inside Microsoft, according to former Carnegie Mellon researcher Johnny Chung Lee, who is among those working on the effort.

And while smashing bricks and cars are some of the first ideas on how to use Natal, the vision clearly goes a lot further.

Inside Xbox, Tsunoda noted that Natal can be useful for more than gaming. He noted that for many first-time console users, the controller itself can be intimidating, even when trying to do things like navigate through menus. Oftentimes people get their first experience with the Xbox when they are at the house of a friend or family member who has an Xbox and they are handed a controller with lots of buttons.

"For a lot of people that can be intimidating," Tsunoda said. "You don't really know what to do and you're starting to feel stupid and everyone is looking at you and you are not being successful. That's really not a good first way to interact with our console."

Tsunoda and Entertainment Unit President Robbie Bach both said they are confident that Natal will also have great appeal for the core gamers already spending hours a week playing on the Xbox.

"Even the folks who are hard-core Halo or Splinter Cell players, they are also going to want to play Natal games," Bach said in an interview.

In an interview with CNET News last month, Bill Gates talked about how the technology has applications well beyond just gaming.

"I think the value is as great for if you're in the home, as you want to manage your movies, music, home system type stuff, it's very cool there," he said. "And I think there's incredible value as we use that in the office connected to a Windows PC. So Microsoft research and the product groups have a lot going on there, because you can use the cost reduction that will take place over the years to say, 'Why shouldn't that be in most office environments?'"

At last week's analyst meeting, Bach and Craig Mundie, chief research and strategy officer, also outlined the broad appeal of being able to interact more directly with computer interfaces. After Bach tried his hand at some Natal gaming, Mundie offered a demonstration of how gesture recognition might function in a work setting, saying that the desktop PC of the future could in fact encompass the entire office.

In his role heading policy and enforcement for Xbox live, Stephen Toulouse is widely known for wielding the "banhammer"--that is being the guy who comes down on cheaters and those who harass people over the online service.

Toulouse said he is starting to shy away from the banhammer moniker, given that he and his team employ a range of punishments, from the temporary suspension of a feature all the way up to permanent ban of all users of a particular console. But, he said, as an avid gamer, he is enjoying his role trying to keep Xbox Live as a fun and safe space.

Toulouse

(Credit: Stephen Toulouse)

"It's nice to get to protect people in a new way," Toulouse said in an interview. Before taking over as top Xbox cop, Toulouse worked in Microsoft's Security Response Center and trustworthy computing unit, handling the flaws in Microsoft's products and the resulting security outbreaks they caused.

In some ways, life has changed little for Toulouse since he switched to the Xbox role in August 2007. He's just fighting different kinds of bad guys.

Whereas Microsoft has a large team of people scouring the Internet for reports of security holes, it also has a team of five or six dozen people that are playing Xbox Live at any given time, looking for any type of problems.

"There's always a segment of the population that is going to be miscreant," he said. Still, he said, at any given time just one-twentieth of one percent of all those using the online service have a complaint registered against them. "It's a tiny fraction of the overall interactions."

Toulouse said he relies on the lessons he learned while trying to protect Microsoft customers from bugs that exploited its flaws.

"I carry with me from the MSRC (Microsoft Security Response Center) days that concept of how can this feature be misused or how can this capability be misused," he said.

Cheating is one of the issues that he deals with, though Toulouse said that is somewhat limited given the closed nature of the Xbox as compared with, say, the PC. Most of the issues come around exploiting a flaw in game's map, say a place that one can go where they can shoot other characters but not be hit themselves.

Probably the area he spends the most time policing isn't in any game at all. It's overseeing the regulation of what people put in their gamer tags and profiles.

"They have 255 characters," Toulouse said. "They can say a lot of things."

The company not only responds to complaints but is also constantly working on expanding its lexicon to include new slang for the terms and subjects that it bans. Urban Dictionary, Wikipedia, and other places help the company keep up to date.

"We spend a lot of time researching those terms," Toulouse said. "It's a huge and fast-moving world in terms of how slang develops."

One of the specific issues that has cropped up under Toulouse's watch is the issue of whether and how users can identify their gender identity and sexual orientation. The issue gained some measure of attention starting last summer after several users were prohibited from referencing a gay identity in their gamer tags.

Microsoft's current practice is to ban any discussion of sexuality in either tags or profiles--a move that makes it impossible for those gamers who want to identify as gay, lesbian, bisexual, or transgender to do so.

In part, Toulouse said, that's because 98 percent of those who have tried to use the term gay have done so not as a means of self-identification but by using the word as a put down.

But for someone who wants to be more than a "banhammer," Toulouse acknowledges just prohibiting all reference to sexuality isn't much of a solution.

"I think what we have today is inelegant," he said, adding that he is working on an improvement, but he still doesn't have a timetable for when a better option will be in place. That's basically the same position he took when the company addressed the issue in February.

"I haven't made a change to date but I am committed to making a change," Toulouse said. "We hear very clearly that customers wish to express this."

Toulouse said that part of the reason it has taken so long is that the company is looking at changing not just the policy but also the profile technology, perhaps adding check boxes where people could include their gender identity or sexual orientation and perhaps other characteristics as well.

"That's the thinking we are leaning toward," he said, adding that no final decision has been made.

This past weekend, Toulouse was in San Francisco for a panel discussion on the role of homophobia in virtual worlds--an issue that more than just Microsoft is trying to grapple with. More than 100 people turned out for the discussion, which was sponsored by the Gay and Lesbian Alliance Against Defamation and also included representatives of Electronic Arts, Linden Labs, the Entertainment Software Association, and Flynn DeMarco, founder of gaygamer.net.

GLAAD's Justin Cole said that it is not surprising that it is taking Microsoft time to come up with a workable solution to what is clearly a big problem.

"For a system as big as Xbox Live to be able to change something isn't as simple as just a flip of the switch," Cole said.

Another issue for Toulouse and team is educating parents about the need to set controls for their children's use of the Xbox. With other game consoles, the biggest issues are often deciding which games a child can play and for how long.

Many parents aren't aware of a potentially bigger decision that comes with the Xbox. Because it runs online and has chatting capabilities, parents also need to decide with whom their child can communicate online. With Xbox Live, users can get text and audio messages, as well as pictures.

"Those capabilities, like any capabilities, can be misused," Toulouse said. By default, accounts set up for under-18 users turn off the chat capabilities, but many teens set up their own consoles and decide to make create adult accounts, which allow all such messages by default.

Parents often think about these issues when it comes to their children's computer use, but don't always think about having similar rules for things like the Xbox. To try to make parents aware, Microsoft has launched a "Get Game Smart" Web site as well as recruiting a number of online parents and teens to serve as "ambassadors" to their less savvy counterparts.

It's a lot more complicated than when he was young and his parents could just take away the power cord if he wasn't allowed to use his Atari home computer. However, he got an early taste of how to cheat the system--saving up his money and buying an extra power cord from a local electronics store.

Microsoft doesn't just want to bring gesture recognition to the Xbox with Project Natal. It also wants the technology in Windows, according to a very good source--Microsoft Chairman Bill Gates.

In an interview with CNET News this week, Gates talked about a world in which depth-sensing cameras such as the one allow people to control their PCs, game devices, and televisions. (See a video from the E3 conference below.)

Speaking about all of the technology Microsoft has cooking in its labs, Gates said: "I'd say a cool example of that, that you'll see... in a little over a year, is this (depth) camera thing." Gates said it was not just for games, "but for media consumption as a whole, and even if they connect it up to Windows PCs for interacting in terms of meetings, and collaboration, and communication."

Gates said it is an example where the project started in Microsoft research but is now being commercialized by both the Xbox and Windows units. "Both the Xbox guys and the Windows guys latched onto that and now even since they latched onto it the idea of how it can be used in the office is getting much more concrete, and is pretty exciting."

Using your body to control devices makes a lot of sense, Gates said. "I think the value is as great for if you're in the home, as you want to manage your movies, music, home system type stuff, it's very cool there," he said. "And I think there's incredible value as we use that in the office connected to a Windows PC. So Microsoft research and the product groups have a lot going on there, because you can use the cost reduction that will take place over the years to say, why shouldn't that be in most office environments."

Gates actually dropped the first hint of Natal during his joint appearance with Steve Jobs at the D: All Things Digital conference in 2007

"Imagine a game machine where you're just going to pick up the bat and swing it, or the tennis racket and swing it," Gates said.

Moderators Walt Mossberg and Kara Swisher mocked Gates, saying such a technology already exists and it's called the Wii. But Gates disagreed. "No, that's not it. You can't pick up your tennis racket."

He later added, "You can't sit there with your friends and do those natural things," he said. "That's a 3D positional device. This is video recognition. This is a camera seeing what's going on."

No matter what people thought they heard Steve Ballmer say in Chicago, Microsoft is not planning to introduce a successor to the Xbox 360 next year.

In a blog posting on Thursday night, Microsoft said that the Xbox 360 is "not even halfway" through its current lifecycle and will be here long into the next decade. The company is planning to introduce a new gesture-recognizing controller, code-named Project Natal, and that is expected next year, although Microsoft says it isn't talking timing with regards to Natal.

"As the Xbox team stated at E3 two weeks ago, we are not even halfway through the current console generation lifecycle and believe Xbox 360 will be the entertainment center in the home for long into the next decade," Microsoft's Major Nelson said in a blog post. "Project Natal will be an important part of this platform, but we have not confirmed a launch date at this time."

Of course, that is what has opened the door to all the speculation. In his posting, CNET colleague David Carnoy notes that Microsoft says it isn't planning an all-new Xbox 360 either.

That doesn't completely shut the door on new hardware for 2010. Although Microsoft has said it plans to offer Natal as an add-on to the Xbox 360, it seems reasonable that the company might well come out with some new bundle or set-up to accompany Natal's arrival as it looks to convert all of those Wii owners.

Johnny Chung Lee, the former Carnegie Mellon researcher known for finding creative ways to adapt the Wiimote, has revealed himself as one of the minions behind Project Natal, Microsoft's effort to add motion-sensing capabilities into the Xbox 360.

Lee, who is now a researcher at Microsoft, said in a blog posting that he has been working on the motion-sensing project.

"Now, I should preface by saying I don't deserve credit for anything that you saw at E3," Lee said in the blog, which he posted on Monday night. "A large team of very smart, very hard-working people were involved in building the demos you saw on stage. The part I am working on has much more to do with making sure this can transition from the E3 stage to your living room - for which there is an even larger team of very smart, very hard-working people involved."

Microsoft demoed Project Natal on Monday as part of its press conference at the E3 trade show. The technology allows a person to act as their own remote, with a depth-sensing camera capturing their motion, and software then translating it into actions.

Lee notes that he can't reveal anything beyond what Microsoft shared, but does talk a little about the technology that underlies Natal.

"The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost," Lee wrote. "Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera."

The hard work, he said, is then converting that cloud of points into human actions, something which requires some pretty sophisticated algorithms. That said, the work could lead in some even cooler directions.

"At times, working on this project has felt like a miniature 'Manhattan project' with developers and researchers from around the world to coming together to make this happen," Lee wrote. "We would all love to one day have our own personal holodeck. This is a pretty measurable step in that direction."

Before joining Microsoft, Lee gained attention for his projects using the sensor bar and remote of the Nintendo Wii to work as head-tracking devices, a multitouch user interface and more.

Lee has continued showing off his Wiimote projects since joining Microsoft, presenting at this year's Mix09 event in Las Vegas. (There's a video below, but it requires Silverlight.)

Update 2:25 p.m. I had a chance to chat with Lee briefly by phone.

For his part, Lee said he thinks Microsoft doesn't get the credit it deserves in the consumer arena.

"It's sort of a bummer that Microsoft gets kinds of a bad rap," Lee said. "It's a lot of very ambitious groups trying to do big things. Not everything makes it out the door."

Lee, who works in an applied science group that sits between the research and product arms at Microsoft, says that the company is working on some very cool stuff, though he could not go into a lot of details.

"I played a little bit with the depth cameras before Microsoft," he said. "The technology I have been able to play with since coming to Microsoft is a lot better."

The big news from Microsoft' E3 press conference was clearly its "Project Natal" motion sensing technology.

We're pretty sure that it is based on technology from 3DV systems, an Israeli start-up that we wrote about some time ago. I'm hearing that Microsoft has in fact, acquired the company, as has been reported. I have yet to get that 100 percent confirmed, however.

In any case, here's a video from Microsoft showing the technology in action.

Updated 2:10 p.m., with additional details

Microsoft said on Thursday it intends to acquire BigPark, a Vancouver, B.C.-based game studio.

BigPark, which is made up of former Electronic Arts Canada and Distinctive Software executives, has been working on an Xbox-exclusive game over the past year.

Microsoft plans to buy BigPark, a company chaired by Don Mattrick (pictured above), who already serves as a senior VP in Microsoft's game unit.

(Credit: Microsoft)

"We believe BigPark has tremendous potential to create new properties and innovative gaming experiences for our platforms, one of which we're looking forward to showcasing at the E3 Expo in June," Microsoft game studios boss Phil Spencer said in a statement. The company did not disclose financial terms in its press release announcing the deal.

Among BigPark's founders is Don Mattrick, who joined Microsoft in July 2007 as senior VP of its interactive entertainment business, while continuing to serve as BigPark's chairman. Microsoft noted that Mattrick's investment and role at BigPark was known to the company when it hired him.

However, neither Mattrick's Microsoft biography nor the press release announcing his hiring mention BigPark.

Update: I asked Microsoft for more clarification on Mattrick's role in the deal.

"Don was not a participant in the negotiations with BigPark either in his capacity as SVP of (Microsoft's game business) or Chairman of BigPark," a Microsoft representative said in a statement. Microsoft added that Mattrick's ongoing role as chairman of BigPark was approved by the company, pursuant to its code of business conduct. "As Chairman, Don was not involved in the day to day management of the BigPark business," Microsoft said.

Microsoft maintains that it did not note Mattrick's role at BigPark when when he was hired at Microsoft because, "BigPark had not publicly launched as an organization." Mattrick is noted (though not his Microsoft affiliation) on BigPark's Web site. As for why his role at BigPark is not part of his Microsoft biography, Microsoft said "As Chairman, Don was not involved in the day to day management of the BigPark business, nor was he an employee of the company; Therefore, mention of Don's limited involvement in BigPark was not significant enough to be included in his biography."

As for its decision not to disclose the purchase price, Microsoft said: "BigPark is a small company and the acquisition is not material from a financial reporting perspective to Microsoft."

Microsoft said the company has 50 employees.

What was that old saying? Disclosure is the better part of valor. Something like that anyway.

Among the other topics Microsoft isn't discussing: just what that game is that BigPark is working on for the Xbox. A representative declined to discuss the genre or nature of the Xbox game under development, saying only that it would be revealed at E3.

Microsoft may be poised to boost the Xbox unit with a simple wave of the hand.

The software giant is reportedly close to buying Israel's 3DV Systems, an Israeli start-up whose technology allows a gamer to control a system through nothing more than a hand gesture. (See video below for a CNET video demonstration of the technology.)

According to a report in the Israeli newspaper Haaretz, Microsoft may be willing to spend around $35 million to acquire the company and its technology, which uses a depth-sensing camera to record a gamer's motions. The newspaper also noted that the price tag would be less than the amount of money the company has raised to date from backers, including Kleiner Perkins.

3DV showed its plans for the technology in December 2007, but has yet to announce a partner that will bring the technology to the masses, although Microsoft has used the systems on a small scale in its research labs. More recently, 3DV has been in talks with a variety of hardware and software companies in an effort to try and get the technology onto the market.

A source familiar with the situation indicated that talks with at least one company have progressed beyond just talks, with details getting down on paper.

Neither 3DV nor Microsoft would comment on the Haaretz report or discuss how far discussions might have progressed.

"We have had many conversations/meetings/technical evaluations with many partners from software developers to hardware manufacturers but I am unable to discuss any specific company, due to non-disclosure commitments," 3DV executive vice president Charles Bellfield said in an e-mail interview.

Although Sony and Microsoft thought they would be in a two-horse race in the console market, it has been Nintendo that has been winning this generation in large part because of the Wii's intuitive interface.

Microsoft has talked about ways of adding more natural interfaces to the Xbox, even within this generation of consoles.

Speaking at the 2007 D: All Things Digital conference, Microsoft Chairman Bill Gates suggested that Microsoft was thinking along the same lines as 3DV in terms of where gaming was headed.

"Imagine a game machine where you're just going to pick up the bat and swing it, or the tennis racket and swing it," Gates said.

Moderators Walt Mossberg and Kara Swisher mocked Gates, saying such a technology already exists and it's called the Wii. But Gates disagreed. "No, that's not it. You can't pick up your tennis racket."

He later added, "You can't sit there with your friends and do those natural things," he said. "That's a 3D positional device. This is video recognition. This is a camera seeing what's going on."

Acquiring 3DV could move that vision closer to reality.