Some of the most basic questions surrounding AR/VR tech aren’t entirely solved yet, like making text input a comfortable and familiar experience. Facebook’s Reality Labs (FRL) today revealed new research into hand tracking which aims to bring touch typing to AR/VR users, all without the need of a physical keyboard.
There’s already basic hand tracking on Quest which lets you navigate system UI, browse the web, and play supported games like Waltz of the Wizard: Extended Edition (2019) without the need of Touch controllers, instead letting you reach out with your own to hands to cast spells and manipulate objects.
As interesting and useful those use cases may be, we’re still very much in the infancy of hand tracking and its potential uses for virtual reality. Using your fingers as glorified laser pointers on a virtual keyboard reveals just how much of a gap there is left in natural VR input methods. On that note, Facebook researchers have been trying to build out hand tracking to even more useful applications, and their most recent is aimed at solving some of the most frustrating things in VR/AR headset users to this day: text input.
Facebook today revealed that its FRL researchers used a motion model to predict what people intended to type despite the erratic motion of typing on a flat surface. The company says their tech can isolate individual fingers and their trajectories as they reach for keys—information that simply doesn’t exist on touch screen devices like smartphones and tablets.
“This new approach uses hand motion from a marker-based hand tracking system as input and decodes the motion directly into the text they intended to type,” FRL says. “While still early in the research phase, this exploration illustrates the potential of hand tracking for productivity scenarios, like faster typing on any surface.”
One of the biggest barriers to overcome was “erratic” typing patterns. And without the benefit of haptic feedback, researchers looked to other predictive fields in AI to tackle the issue of guessing where fingers would logically go next. FRL says it researchers borrowed statistical decoding techniques from automatic speech recognition, essentially replacing phenomes for hand motion in order to predict keystrokes—that’s the short of it anyway.
“This, along with a language model, predicts what people intended to type despite ambiguous hand motion. Using this new method, typists averaged 73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else, achieving similar speed and accuracy to the same typist on a physical keyboard,” the researchers say.
With its insights into hand tracking, Facebook is undoubtedly preparing for the next generation of AR headsets—the ‘always on’ sort of standalone AR headsets that you might wear in the car, at work, at home and only take off when it’s time to recharge. Using Quest 2 as a test bed for AR interactions sounds like a logical step, and although the company hasn’t said as much, we’re hoping to see even more cool hand tracking tech pushed out for experimental use on the new, more powerful standalone VR headset.
The post Facebook Develops Hand Tracking Method to Let You Touch Type Without Needing a Keyboard appeared first on Road to VR.
Ream more: https://www.roadtovr.com/facebooks-hand-tracking-research-keyboard/
No comments:
Post a Comment