Posted on by

Microsoft’s new gaze-based video streaming technology will reduce the bandwidth required for meetings

Microsoft’s new gaze-based video streaming technology will reduce the bandwidth required for meetings

Microsoft recently published a paper and patent for a new form of video streaming called Gaze Based Video Stream Processing. It is a simple yet impressive AI-based software tool that would bring the concept of foveated rendering to video conferencing and chat applications such as Microsoft Teams, Zoom and Discord.

VIEW GALLERY – 3 PICTURES

It is a technology based on how we see as humans, viewing objects within 5 to 10 degrees of our primary focus in the most detail, with this value rapidly decreasing by 20% outside of these 10 degrees.

Foveated rendering for VR games uses eye tracking to ensure that the areas or objects you are looking directly at are rendered at the highest quality and all effects and detail settings are enabled. At the same time, the quality of areas or objects not being viewed is reduced to increase performance. With its new patent, Microsoft transfers this concept to video streaming.

Microsoft's new gaze-based video streaming technology will reduce the bandwidth required for meetings 3

Gaze Based Video Stream Processing uses a system to estimate or predict where the user is looking, allowing the video stream processor to reduce the quality of the video streams that are not being viewed. The immediate benefit would be less bandwidth and a potentially more stable connection.

The patent provides images showing what it would look like. Without the system, you get three video streams of different quality: two 1080p and one 720p. As a standard video stream, all three are presented in full quality based on available bandwidth. With gaze-based video processing, the viewed stream maintains full quality at 1080p 60 FPS, while the quality of the two streams outside of the user’s focus drops to 360p 24 FPS.

It reduces the bandwidth requirements for video conferencing and ensures that bandwidth is delivered where it is needed. With AI and a neural network enabling gaze detection, this feature could be available on Copilot+ PCs with integrated NPUs – with the first example being deployed in Microsoft Teams.