Is Meta Bringing Photoreal 'Codec Avatars' To Quest Soon?

Republished By Plato

Followers: 0

A firmware finding suggests Meta is well on the way to bringing its photorealisic 'Codec Avatars' to its Quest headsets.

Codec Avatars is Meta's long-running research & development project aiming to revolutionize remote communication by achieving truly convincing photorealistic avatars driven in real-time by headsets with face tracking sensors, and eventually glasses too. Meta first revealed the project's existence in 2019, and has showed off multiple fidelity updates in the years since as well as a full-body version.

[embedded content]

In this interview, it's unclear whether the avatars were being decoded and rendered onboard, or whether a PC was doing this work. A USB-C cable is connected to both headsets.

While Codec Avatars has always been described as in the realm of research, a new finding suggests it could now be well on the path to arriving as a feature in Quest headsets.

VR enthusiast Luna found references to Codec Avatars by decompiling version 62 of the Meta Quest firmware, which is currently available to testers in the Public Test Channel.

Upcoming Quest features have been discovered this way multiple times in the past. It tells us that Meta is working to integrate a feature on Quest headsets. However, it doesn't tell us how far away the feature is, or whether it will ever actually ship at all. For example, Space Sense arrived a few months after being spotted in the firmware, while Quest Pro took over a year. And references to a Discord integration and a Project Avalanche cloud PC VR streaming feature have been spotted for years now, but neither has shipped.

In March last year Meta CTO Andrew Bosworth said the company was "making progress" on shipping Codec Avatars but cautioned "we are still years away".

Strings found in the Quest firmware by Luna.

It's also not clear whether this feature is being developed for existing Quest headsets or a future Quest Pro 2.

In March last year, a Meta Quest hardware roadmap meeting was leaked to The Verge. As well as revealing details of Quest 3 and the upcoming cheaper headset rumored to be called Quest 3 Lite, the roadmap mentioned a new Quest Pro headset “way out in the future” that will include Codec Avatars.

But it's possible Meta intends to bring Codec Avatars to the existing Quest Pro, or even Quest 2 and Quest 3.

Codec Avatars being decoded on Quest 2 standalone.

In 2021, Meta researchers demonstrated a significantly more efficient method of decoding Codec Avatars. On Quest 2, the new method could render one Codec Avatar in a blank scene at full 72FPS, three Codec Avatars at around 63FPS, or five at around 43FPS.

But how could these avatars be driven on Quest 2 and Quest 3, which lack both face and eye tracking? Meta researchers have also demonstrated a version of Codec Avatars that doesn't require face tracking, and recently even a version that doesn't require any tracking at all. Instead, audio from the microphone is fed into a neural network that estimates the likely facial pose for the current speech pattern.

Still, it's entirely possible Meta will want to reserve Codec Avatars for a future Quest Pro 2 powerful enough to support Codec Avatars in Horizon Workrooms meetings, and drive them with real face tracking.

A specialized capture rig used for the highest quality Codec Avatars.

Regardless of which specific Quest headsets the shipping Codec Avatars feature is intended for, the quality almost certainly won't be the same as Meta's most impressive demos, such as the Lex Fridman interview with Mark Zuckerberg. Those demos, and that interview, used Codec Avatars generated using a specialized capture rig with over 100 cameras.

[embedded content]

Meta researchers have been working to replace this process with a smartphone scan, necessary to actually ship Codec Avatars at scale, but this gives a lower quality result. Quest Pro 2 might include a front display and depth sensors for avatar setup like Apple Vision Pro uses for its Personas, but this still wouldn't match the specialized capture rig.

Still, any version of Codec Avatars should be a dramatic improvement over Meta's current avatars, which have a basic cartoony art style that couldn't be further from photorealism. We'll keep a close eye on Meta in the coming months for any further hints of Codec Avatars being on the path to productization.