Newsletter
Streaming VR and AR experiences over 5G data connections could be moving a step closer to reality thanks to a Google invention designed to ensure video file sizes remain sensible even as resolutions shoot up, with the solution being detailed in a newly WIPO-published patent. The technology described doesn’t introduce a new head-mounted display and there aren’t any flashy images shown in the documentation. Google instead developed a method for bringing down file sizes that VR and AR hardware are forced to work with via a new compression method tied directly to foveated rendering.
Although relatively straightforward, pertaining to a compression method prior to streaming and a decoder used in the headset itself after the content is delivered, the process behind the technology is substantially more complex. The entire premise hinges on exactly how foveated rendering works, decompressing pixels at the center of a wearer’s field of view first and to vary decompression based on how far away from that center the pixels are.
By blending the level of compression for pixels that fall between the low resolution and high-resolution extremes to a varying degree, the method can help an AR or VR system replicate how the human eye works. Foveated rendering already accomplishes that at some level by using eye tracking to ensure that the display resolution is higher wherever a headset’s wearer happens to be gazing. All other pixels are rendered at a lower resolution, allowing for a more immersive experience without overworking processors or the display. Naturally, today’s implementations of that tech are largely reliant on locally stored textures.
Similar but not congruent
At first glance, Google’s patent appears to create another method by which an effect similar to foveated rendering could be implemented. On deeper inspection, the process used for the latter solution is completely different and Google’s patent would be used explicitly for making streaming VR or AR content easier from a hardware perspective. It could also drastically reduce the amount of bandwidth required to transmit those experiences.
Foveated rendering largely relies on a reduction in resolution to recreate how the human eye works, enabling higher resolutions to be used at a wearer’s focal point without increasing the workload for the VR or AR hardware. Google’s patent compresses or decompresses those images based on where a user is looking. So Google’s proposed solution would essentially be implemented only in situations where VR and AR are presented using those techniques but are going to be streamed wirelessly.
One for the 5G era
One major obstacle faced by companies looking to bring VR and AR more into the mainstream has been motion sickness and similar maladies caused by how content is rendered. Foveated rendering has also already been explored as a possible solution by various VR-related companies for years, including by tech giants such as Google, NVIDIA, and HTC. The tech not only allows a higher resolution but its mimicry of the human eye makes experiences more natural and less jarring.
Another obstacle and one that seems to be addressed by Google’s latest patent lies in reducing the complexity for end users and getting devices into users’ hands. That’s primarily been approached by combining VR or AR and smartphones or smartphone compatible hardware for wireless headsets such as the Oculus Quest or HTC’s VIVE Cosmos.
Google’s invention could help break through some of those barriers by enabling portable wireless VR systems to present a more powerful and realistic experience but it would also offer another layer of appeal for users. If the search giant releases a commercially viable version of its new technology, it could feasibly allow streamed experiences that are more easily accessed by anybody from anywhere over 5G networks.