Google Lens now lets you search with video

If you can’t capture what you’re looking for with just one picture, Google Lens now lets you record a video – and even use your voice to ask about what you see. The feature displays an AI overview and search results based on the content of the video and your question. It launches today in Search Labs for Android and iOS.

Google first previewed the use of video for search at I/O in May. For example, Google says that someone who is curious about the fish they see in an aquarium can hold their phone up to the exhibit, open the Google Lens app, and then press and hold the shutter button. Once Lens starts recording, they can ask their question: “Why do they swim together?” Google Lens then uses the Gemini AI model to provide an answer, similar to the one you see in the GIF below.

Speaking about the technology behind the feature, Rajan Patel, vice president of engineering at Google, said The edge that Google captures the video “as a series of frames and then applies the same computer vision techniques” previously used in Lens. But Google goes one step further, passing the information to a “custom” Gemini model designed to “understand multiple frames in sequence… and then provide an answer that is rooted in the web.”

There’s no support yet for identifying the sounds in a video – like when you’re trying to identify a bird you hear – but Patel says Google has been “experimenting” with it.

Google Lens is also updating its photo search feature with the ability to ask a question using your voice. To try it out, point your camera at your subject, hold down the shutter button, and then ask your question. Before this change, you could only enter your question in Lens after taking a picture. Voice Questions is rolling out globally on Android and iOS, but is currently only available in English.