Will the Ultimate Quest 3 and Vision Pro Accessory Be a Book?
Jim Henson's Storyteller in AR, on Magic Leap 2, makes an ordinary book come alive. The idea could come back as an accessory for future headsets.
- Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
An AR adaption of Jim Henson's Storyteller: It's captivating, but can it also be the beginning of other book accessories for AR?
Opening an ordinary-looking book and seeing magical holographic creatures jumping out of it seems like the sort of thing AR glasses were supposed to make happen years ago. In some cases, they sort of did: Way back, Sony had a Harry Potter-licensed AR book that worked with PlayStation's camera, and Nintendo dabbled in its own marker-based cards and books using the 3DS's cameras and 3D displays to make AR experiences, too. A new experience, based on Jim Henson's Storyteller, presents itself as an idea for how physical accessories like books could re-emerge for future mixed reality headsets.
Felix and Paul Studios created the AR experience, but it's a new venture for the otherwise cinematic VR-focused company. The studio's ongoing series of 360-degree films shot onboard the International Space Station does an amazing job capturing reality to view inside a VR headset. But in the case of the AR-based Storyteller experience, it's an opposite challenge. Instead of bringing me into another place, Jim Henson's The Storyteller: The Seven Ravens puts a little story world inside a physical prop like a book. The visuals are charming, like a living miniature video game theater, and the voice narration by Neil Gaiman certainly doesn't hurt.
The experience felt like I was looking down into the book at a world inside, where I could tilt and turn to see more beyond the book's portal. Sometimes, characters also come up and peer back through the window at me, too. Tilting the book to the side can cause water to spill out from an ocean scene. New AR scenes start every time one of the book's thick boardlike pages is turned. Also, when I reached what I thought was the end of the book, I was instructed to flip it around and start turning pages in the other direction. The story continued as I flipped pages back again. A book like this could potentially be infinite.
Tools beyond controllers
The future of VR and AR headsets looks like it'll be controller-free. Hand tracking should be more of a standard, thanks to devices like Meta's upcoming Quest 3, which promises better depth-tracking sensors, and Apple's Vision Pro arriving in 2024. If a controller-free mixed-reality future is coming, relying on our hands to reach out and touch things that aren't there, it's likely that a variety of mixed reality-ready accessories and objects could be how AR feels like it extends into a wider range of tactile tools.
Keyboards and trackpads are the obvious next step. Apple's Vision Pro and Meta's Quest headsets support these, and some companies are already thinking about AR-ready keyboards. Maybe other accessories will come after that. Books, or objects that can be little spaces designed to be ready for AR to project onto them, could be next.
It's reminiscent of plenty of previous marker-coded toys and cards that have worked with AR-ready phones and tablets previously, or even headsets. But the technology's evolution may be coming at the right time now.
The Seven Ravens experience and its book illusion work pretty well and cleverly use its book design to mask one big limit of current AR glasses. Headsets like the Magic Leap 2 have a limited field of view, which, in this case, aligns with the borders of the custom-made prop book for the experience. Stiff pages with marker codes on them are turned one by one, each page making a new chapter come alive and seem to dip into the frame of the book (which also helps with 3D object transparency in AR, since the book's pages are dark black).
The Magic Leap-based Storyteller experience was created five years ago and was intended to debut back in 2020. It was delayed because of the pandemic and because Magic Leap changed its path, shifting from creative-focused experiments to a business-targeted relaunch. The Storyteller experience emerged from the ashes to launch at the Venice Film Festival, but Felix & Paul Studios' co-founder, Paul Raphael, now sees the experience as being a stepping stone to making an actual book product for headsets sometime soon, possibly even for mixed-reality devices like Apple's Vision Pro or the Meta Quest 3.
Raphael sees recent passthrough-based mixed-reality VR headsets that can do AR as a perfect match for ideas like this, partly because people actually own these devices, as opposed to the AR-glasses space, which is currently way off on the fringes.
"I think passthrough would be great for this. In some ways better; in some ways not quite as good. You're aware of the environment, what's in your hands, but the focus is on what's in the book," Raphael said. "Even if what's being rendered is not quite photorealistic, you're getting quite close with the Vision Pro. And even the Quest 3 is going to do a pretty great job, presumably. I totally see those as the prime sort of devices for this experience."
The Storyteller isn't an actual product yet, but the idea for the book could be. "Once AR becomes commonplace, and the pair of glasses like the one you're wearing can do AR, everything should be augmentable," Raphael told me after I tried the experience. "When we started designing this, we were actually thinking of object-based storytelling at large. This book is the first one, the one we could do the most mileage with, but we have concepts for all sorts of augmented objects that we would love to make a reality as well."