Underexposed

IM2GPS compares a sample photo (top left) to geotagged Flickr photos to find other similar shots (top right) to guess where the sample was taken.

(Credit: Carnegie Mellon University)

Thousands of others have taken the trouble to geotag their photos, so why should you have to jump through a lot of technical hoops to add location data to your pictures?

That's the upshot of a technique devised by Carnegie Mellon researchers and announced Wednesday. The technique, called IM2GPS, compares a single photo to the millions already on Flickr that already have latitude and longitude coordinates.

The algorithm looks at a photo's properties, such as textures, color distribution, and line patterns, then looks for matches at Flickr.

"We're not asking the computer to tell us what is depicted in the photo but to find other photos that look like it," said Alexei A. Efros, assistant professor of computer science and robotics, in a statement.

Efros also has been involved in photo research such as the scene completion technology that can patch over unsightly elements in a photo by drawing from similar ones stored at Flickr.

The researchers found they could locate sample photos within 200 kilometers for 16 percent of their test photos, which may not sound terribly useful, but it is 30 percent better than chance would predict, the university said. And that could still be useful for tasks such as forensic crime research or for guiding other image-processing tasks--for example identifying a taxi in Japan.

It worked more specifically at times, for example matching Paris' Notre Dame cathedral well, but the algorithm found Sydney's Opera House similar to a hotel in Mississippi and to a bridge in London.

Geotagging today is a complex task that typically requires a user to run specialized software that pulls location data from a GPS device's track log, then adds it to photos depending on the time each was taken. Geotagging isn't for the faint of heart today, though higher-end cameras from Canon and Nikon make it easier with the ability to plug a GPS directly into the camera, and camera makers have begun building GPS into some models.

Geotagging may seem abstruse, but it has potential advantages. You could find out just where that photo of the nice church in Ireland was taken even long after your vacation itinerary has faded from memory, for example.

Or with technology that converts geographic coordinates into actual place names, you could find your own photos or others' shots with ordinary search terms. For that latter challenge, Flickr is working to try to make it easier for users to identify in works the locations of their geotagged photos.

It's one of the oldest, most common problems in photography: that picture you thought would be the prize shot is out of focus.

Refocus Imaging, a Silicon Valley start-up, thinks its technology can be used to make cameras that can fix that problem--after you take the photo.

By fitting a camera's image sensor with a special lens and then processing the resulting data with new methods, Refocus Imaging's technology will let photographers fix their photos and exercise new creative control after the shutter is released, founder and Chief Executive Ren Ng said.

"There's a lot of physical stuff in the camera that is limiting its performance," Ng said. "What we're doing is to capture much more than a two-dimensional photograph inside the camera...By collecting the light, we can process it in software to do what the hardware usually has to do."

And the technology boosts some aspects of camera performance in the process, he said. Ng said he hopes to license it to camera companies, and boasts that Refocus Imaging's patent portfolio is "very, very good."

The technology, which stems from Ng's research at Stanford University, is an example of computational photography, which augments traditional image capture with computers--either in the camera or on a PC--to achieve new possibilities.

Included here are examples from Refocus Imaging that show how the technology works. The slider on the right of each graphic can be used to change the point of focus from foreground elements to those in the background, or clicking on a different area will bring it into focus.

Ng also showed the technology off at the 6sight digital-imaging conference in November.

The way Ng sees it, the Refocus Imaging technique has several possible advantages. For one thing, being able to focus images after the fact means that cameras could take a picture sooner without waiting for an autofocus mechanism to lock in. For another, because the depth of field also is adjustable along with focus, a pro photographer could fine-tune a picture to properly blur a background or get just the right amount of a subject in focus.

Refocus Imaging CEO Ren Ng

(Credit: Refocus Imaging)

"One way to think of it is just a raw image, except to the nth degree," Ng said, referring to the raw images that higher-end cameras can record directly from the image sensor, leaving processing choices to the photographer. "It contains a ton more information than a raw picture today. There are all kinds of creative controls you couldn't even conceive of now."

Another advantage is that the technology works better in low light, he said. And by transforming the light's optical properties using a computer instead of relying just on the camera's lenses, a computing system can correct aberrations to improve lens sharpness, as well as heighten lens contrast and lower its manufacturing costs.

Sounds swell, right? Well, there's still no thing as a free lunch.

A lot more image processing is required, for one thing, though Ng legitimately points out that camera processors are steadily improving. Another big drawback is that the full resolution of the image sensor isn't available in the ultimate image the camera produces.

Ng isn't willing to discuss exactly how much resolution is lost in the process at this stage in the company's research. "You can get gorgeous 4x6 prints or (larger), and take those much more dependably," he said.

Refocus Imaging's ideas are related to an image sensor that can see in 3D, in a sense, that another Stanford researcher, Keith Fife, demonstrated earlier this month. That sensor also uses tiny lenses, but his are built directly into the sensor, with one lens dedicated to a particular subarray of sensor pixels.

Ng's company is one of several organizations researching the idea of the "light field," which describes all the light entering a camera, not just the subset that gets photographed with a particular camera setting. Ng offers an analogy: where a photograph is like an X-ray image, the light field is like a three-dimensional CT scan that lets a doctor effectively look at the interior of a person from any direction.

One light field research project at Stanford in the 1990s used an array of 100 cameras all taking a photo of the same subject, then compressing the resulting image data into a representation of the light field. With Refocus Imaging's technology, Ng said, "we can make that compression in a single device."

MONTEREY, Calif.--In August, researchers unveiled a new way of shrinking or expanding photos called seam carving. Now it turns out the technique applies to video, too.

Ariel Shamir, a senior lecturer at the Efi Arazi School of Computer Science in Israel and a visiting scientist with Mitsubishi Electric Research Laboratories, showed off the technique at the 6sight digital-imaging conference here last week. (Adobe Systems has hired another seam carving researcher, Shai Avidan.) The technology analyzes a picture for vertical or horizontal "seams"--the term the researchers use to describe a path traversing the photo where pixels are most like their neighbors and therefore least likely to be missed.

The effect is that important areas such as human faces remain intact, while relatively uniform backgrounds such as lawns or skies are squeezed. Seams zig-zag to an extent, for example detouring around clouds through interconnected patches of blue sky. Seam carving works best for photos with multiple subjects separated by an uninteresting background; a subject that occupies the entire frame is likely to be distorted as the image is scaled down.

A related technique called seam insertion reverses the process to add data, giving photos a more spacious look.

During Shamir's demonstration, he widened and narrowed a variety of photos. In addition, he showed how to select specific pixels for priority preservation or removal, in one case excising a girlfriend from a photo the way Soviet censors vanished Leon Trotsky.

And showing off a new twist on the technology, Shamir did the seam carving on a running video of a golfer taking a swing. The golfer remained intact even as the fairway changed from a narrow sliver to a broad tract of green grass. To see the demonstration, check the video above.

The content-aware resizing tool stretched the two narrower images by Japanese artist Utagawa Hiroshige into the adjacent wider versions.

(Credit: Shai Avidan, Ariel Shamir)

One area where seam carving could be useful is in resizing images along with the Web pages they're shown on, for example preserving important parts of a photo even when it's displayed on the tiny screen of some Internet-connected gadget. That could apply to video as well as to still images, though obviously it would require more computational horsepower.

Coming from a journalism background, my instinct is to keep photos generally true to the original, so these automated distortions at first ruffled my feathers.

But on further reflection, it occurred to me that at least when it's working well, seam carving does to an image precisely what my brain does as well: focus on the important bits.

What was that portrait I saw in my art class of youth, an elderly woman seated in black clothing? Her hands and face were painted with detail and care, but the rest of the picture was painted with rough, almost slapdash strokes. But the painting was fine: my mind naturally cared chiefly about the face and hands, the instruments of human expression, and all else was largely optional. It was like a good lossy compression algorithm that saves space by throwing away the data we're not going to miss.

Children, too, instinctively do the same thing. When they draw pictures of people, the features on the face often creep around to occupy the entire head. In reality, you can cover most of your facial features with the palm of one hand, so the significant parts don't actually take up much of the actual surface area. But in children's pictures, anything above the eyes or eyebrows evidently doesn't deserve much attention.

So perhaps seam carving is just an overt manifestation of what we already do ourselves.

Just don't use it to mess with any news sites!

Different sound frequencies indicate where on the screen a user is blowing.

(Credit: Georgia Tech)

Perhaps huffing at your computer might get you somewhere if research at the Georgia Institute of Technology comes to fruition.

Shwetak Patel and Gregory Abowd from Georgia Tech have published a paper that describes how to use a computer microphone to determine where on a screen a person is blowing. The technique, which they call BLUI for Blowable and Localized User Interaction, can distinguish between the different sounds air makes depending on where the breath is directed.

"BLUI supports blowing at a laptop or computer screen to directly control specific parts of an interactive application, such as blowing at a button to activate it," the researchers said in their paper (click for PDF). The technique requires a period of "training" to calibrate the system--blowing on each region of the screen for 3 to 5 seconds.

The hands-free user interface approach could be useful for situations where a person's hands are busy, or for people who can't control computers with their hands or arms in situations where speech control is impractical. Although speech "is reasonable for complicated or command-based tasks, it is not well-suited for direct, low-level controls such as scrolling, button pressing, or selection," the researchers said.

Of course, the resolution isn't as fine as a mouse pointer.

The accuracy was 100 percent when dividing a laptop screen in to a nine-rectangle region. It dropped to 96 percent for 16 regions, 80 percent for 25 regions, and 62 percent for 36 regions.

The technique also could be used for games such as a basic one in which users blow out virtual candles, shown in the YouTube video above.

(Via John Nack.)

MONTEREY, Calif.--Get ready for a new era in which your camera knows not just when you took a picture but who's in it, too.

Many cameras today can detect the faces of those being photographed, which is handy for guiding the camera to set its exposure, focus, and color balance properly. But the more difficult challenge of face recognition is more useful after the photo has been taken.

University of California-San Diego researchers have turned expression-recognition technology into an art exhibit showing the increasingly strained efforts by models to maintain a chipper smile for more than an hour. A buzzer goes off when a waning smile sends a monitor into the red zone.

(Credit: Stephen Shankland/CNET Networks)

That's because of a concept called autotagging, one of a number of technologies that make digital photography qualitatively different from the film photography of the past.

Tags of descriptive data can be attached to digital photos, and they help people find and organize pictures. The only problem is that tagging your photos, today a laborious manual task, is like eating your vegetables. It's good for you but a lot of people don't like it.

With autotagging, the camera attaches tags as the pictures are taken. Today, cameras embed timestamps in photos, which makes it possible to sift through pictures by date. But be honest here--how reliably can you remember exactly when you took that picture of your darling daughter a year or two ago that you'd like to e-mail to her grandparents? Being able to screen for photos only of a particular person could dramatically speed up the search process.

Face recognition requires computational horsepower that is hard to fit into the confines of a digital camera, but one company likely to help make it a reality is Fotonation, which already supplies face-detection software for dozens of camera models from Samsung, Pentax, and others.

The computational challenge is reduced by the fact that most folks tend to photograph the same set of 25 or 30 people, Eric Zarakov, Fotonation's vice president of marketing, said in an interview here at the 6sight digital imaging conference. A camera could be "trained" to recognize just those particular people.

He wouldn't comment on whether Fotonation plans to sell such software to camera makers, but it sure looks likely. "We're looking at a lot of stuff. That would be a natural extension" of today's product lines, Zarakov said.

One camera maker willing to mention its interest in autotagging is Panasonic. "A lot of thought is going into how to tag photos so you can retrieve them at a moment's notice," said Alex Fried, national marketing manager for imaging at Panasonic's Consumer Electronics Co. But he wouldn't go into specifics: "There are things we have in the works that will help benefit consumers going forward."

And faces aren't the only aspect of autotagging that's likely to show up in cameras. Location, too, is another useful attribute that can be attached to photos through a process called geotagging. Geotagging can be used both to look for photos whose location you know and to figure out what exactly is in a photo you already have at hand.

Today, geotagging is generally a laborious manual task that requires geographic data to be merged with photos after the fact using a computer. But more power-efficient approaches will lead to in-camera GPS systems that will enable automatic geotagging, predicted Kanwar Chadha, founder of GPS chip designer SiRF Technology.

"A location stamp is much more important than a time stamp in most cases. A year down the road, you have no idea where those pictures were taken and no way to search for location," Chadha said.

Face recognition is an area of active research and some commercialization. Start-up Riya is working on technology to search through online photo albums to try to identify individuals. Polar Rose is trying to improve recognition by generating 3D models of faces. And 3VR wants to apply face recognition to what's become a highly lucrative market, security.

Software from University of California-San Diego researchers, shown here at the 6sight digital imaging conference, can identify facial expressions. This shot shows the nose-wrinkling detector in action, which Marian Stewart Bartlett believes could be useful for market researchers.

(Credit: Stephen Shankland/CNET Networks)

At the 6sight conference, Marian Stewart Bartlett showed results of her research into not just face detection, but expression detection. Her work at the Machine Perception Lab at the University of California-San Diego lets a computer monitor 30 of the 46 codified components of facial expressions. That includes movements such as raised eyebrows and wrinkled noses.

In the demonstration, software tracked Stewart's face from a video camera and recorded expression parameters. Analyzing the data, the computer can draw conclusions about people. For example, when comparing a video of a man's face as he experienced actual pain from immersing his hand in cold water to another in which he faked the pain, people had about an even chance guessing which showed the authentic pain. The computer, though, had 72 percent accuracy, she said.

That level of sophistication is beyond a camera's abilities today, requiring a full-fledged computer run by people with Ph.D. degrees. But particularly given that Sony already has introduced a camera with smile detection, it's not hard to imagine a day when your photos could also some day be tagged "delighted" or "disgusted," too.

MONTEREY, Calif.--A start-up called Artificial Muscle hopes its actuator technology will provide a cheaper, quieter, and lower-power alternative to the host of motors and other devices that control mechanical movements inside cameras.

An elastomer-based actuator in a 9.5mm housing.

(Credit: Artificial Muscle)

The company's technology employs a particular variety of resilient substances called elastomers. This variety changes properties when a voltage is applied across them, growing softer or firmer. Artificial Muscle mounts a ring of the material to a central disk that's pushed by a spring; when the material relaxes, the spring pushes the central disk outward.

The distance the disk travels, or "throw," is as much as 300 microns, or 0.3 millimeters, for an actuator package 9.5mm across, or 250 microns for the company's new 8.5mm package. (These sizes are standardized by SMIA, the Standard Mobile Imaging Architecture, so suppliers and manufacturers have more sales options.)

A 300-micron throw might not sound like much distance, but it's enough to run a variety of camera mechanics, said Charlie Duncheon, executive vice president of sales and marketing, in a presentation here at the 6sight digital imaging conference Thursday.

"We are going to be starting in the camera actuator market," he said. Among the tasks that could use the technology: autofocus, image stabilization, aperture control, mechanical shutters, and optical zoom.

The actuator can respond quickly enough for image stabilization, since hand shake happens with vibrations at between about 5Hz and 20Hz. However, throw distance decreases with faster actuator movement: the travel drops to 90 microns at 20Hz and 20 microns at 100Hz, he said in an interview.

Artificial Muscle was founded in 2003, spun out of research center SRI International in Menlo Park, Calif.

Today, if you want to trim all the distracting background out of a picture--say, the crowd behind your daughter playing soccer--you have to do a lot of artful selection with high-powered software such as Photoshop. But what if your computer understood the depth of the image, just as you did when you took the picture, and could be told to just erase everything that's a certain distance behind your kid?

Adobe's Dave Story shows a multi-view camera lens for taking 3D pictures.

(Credit: Audioblog.fr)

That's one possible way to use technology that Adobe Systems has begun showing off--and that can be seen in video of a news conference posted by the Audioblog.fr site last week.

Dave Story, vice president of digital imaging product development at Adobe, showed off aspects of how the technology worked. First comes a lens which, like an insect's compound eye, transmits several smaller images to the camera. The result is a photograph with multiple sub-views, each taken from a slightly different vantage point at exactly the same time.

From this information, the computer reconstructs a model of the scene in three dimensions.

Story then showed a video with significant transformations of an image based on this 3D understanding. The image had three major elements--a statue in the foreground, a statue in the middle distance, and a wall in the background. The video showed a simulation of a person shifting vantage point left and right--natural enough given that the multiple views captured that information.

Then, however, the video showed a more unusual transformation: an artificial shift of focus from the original picture, which was aimed at the middle-distance statue, to both the foreground and the background. It took the engineer who developed the technology a week to write the software, and another week to run the simulation, Story said.

Story suggested that the perspective-shifting idea would be useful for dealing with a news photograph of a subject who, you find later, is standing directly in front of a pole. But the 3D comprehension could lead to more useful transformation: "Why don't we have a 3D healing brush and, say, get rid of everything behind his head?"

Story adds focus to some elements of a photo.

(Credit: Audioblog.fr)

He didn't demonstrate that idea, but he showed another application of the 3D technology. "If we know the 3D nature of every pixel, what if we could make a focus brush? What if I had a three-dimensional brush where I could reach into the scene and adjust the focus?"

He then showed what he said this focus brush--along with a corresponding defocus brush--might look like. (To my jaundiced eye he could have just been copying from one focus layer to another, but creating the multiple focal planes from a single image is impressive.)

"This is something you cannot do with a physical camera. With the combination of that lens and your digital darkroom, you have what we call computational photography. Computational photography is the future of photography," Story said. "The more things we can do that are impossible to do in a camera, the more powerful people's ability to express themselves becomes."

(Via The Online Photographer)

Adobe Systems has hired Shai Avidan, co-developer of a technology to dynamically resize photos in a way that preserves the more important areas of the image, and a couple of other researchers as well.

The content-aware resizing tool stretched the two narrower images by Japanese artist Utagawa Hiroshige into the adjacent wider versions.

(Credit: Shai Avidan, Ariel Shamir)

Avidan's presentation this month at the Siggraph computer graphics show and the accompanying video has ignited a frenzy of chatter from Slashdot, TechCrunch and elsewhere. I first heard about it last week from the blog of Adobe Photoshop Senior Product Manager John Nack, who also brought word of the new hire Wednesday.

Avidan began work at Adobe Monday. Another new hire is Wojciech Matusik, who's worked on a camera lens system that can photograph an image simultaneously at four different apertures and on a real-time image selection technique that employs dual-image sensors. And starting in a couple of weeks is Sylvain Paris, who's worked on technology such as the two-scale tone management that can Ansel-Adamsize a photo by transferring the style of one to another.

Avidan's "content-aware image resizing" technology works by searching for the vertical or horizontal pathways that skirt around busy areas of the photo--for example, between clouds in front of an even blue sky. It then removes or adds pixels on either side of that pathway, depending on whether the image is being shrunk or enlarged.

The technology also can be used to crop out specific parts of an image that a user highlights as disposable. Neighboring pixels are stretched to fill in the gap. Though, Avidan and technology co-creator Ariel Shamir caution in a paper about the technology that the resizing doesn't work in some situations--for example a picture of car that occupies the full frame of the image.

Nack labeled the technology "holy-crap-worthy," but not all are so excited about it.

Griped Mike Johnston at the Online Photographer:

"To me this is a form of auto-fakery that will further erode whatever integrity photographs still possess," he said. "I'm not saying it's wrong, just that it's wrong for me given my philosophy of what photography's all about, and I won't say it's bad, just that it's bad in terms of the principles I try to abide by when I picture the world as a photographer."

Personally, I'm in Johnston's camp, but the resizing technology is intriguing. In any event, before getting too antsy about a new spate of wickedly Photoshopped news photos, take Nack's disclaimer to heart: "Just because a particular researcher has worked on a particular technology in his or her past life, it's not possible to conclude that a specific feature will show up in a particular Adobe product