Microsoft gives Copilot a voice and a vision in its biggest redesign yet

Microsoft today unveils a major overhaul of its Copilot experience, adding voice and image capabilities to transform it into a more personalized AI assistant. As I revealed exclusively in my notepad newsletter last week, Copilot’s new features include a virtual news presenter mode to read you the headlines, the ability for Copilot to see what you’re watching, and a voice feature that lets you speak to Copilot naturally , similar to OpenAI Advanced Language Mode.

Copilot is being redesigned for mobile, web and the dedicated Windows app into a more map-based user experience that looks very similar to Inflection AI’s work with its personalized AI assistant Pi. Microsoft hired a number of people from Inflection AI earlier this year, including Google DeepMind co-founder Mustafa Suleyman, who is now CEO of Microsoft AI. This is Suleyman’s first major change to Copilot since taking over the consumer side of the AI ​​assistant.

“At Microsoft AI, we are developing an AI companion for everyone,” Suleyman says in an open letter today. “I firmly believe that we can create a calmer, more helpful and supportive age of technology, unlike anything we have seen before.”

Copilot now looks different than anything I’ve seen from Microsoft before, with a user interface that’s very different from what currently exists. It’s much warmer, with a personalized Copilot Discover page that’s more useful and inviting than a chatbot text prompt. Microsoft will customize the entire Copilot homepage based on your conversation history and will include useful search features, tips, and relevant information over time.

Microsoft gave its consumer version of Copilot to Suleyman’s team earlier this year, which has clearly given the company the opportunity to experiment more with personality and customization. “What we learned from the Pi team and the [Inflection AI] “The people who came by showed that they always had an eye for detail about customer needs,” said Yusuf Mehdi, executive vice president and consumer chief marketing officer at Microsoft, in an interview with The edge. “The way they listen and what they learned from these long conversations as part of this research certainly influenced what we did here.”

Beyond the appearance of this new Copilot, Microsoft is also intensifying its work on its vision of an AI companion for everyone by adding voice functions very similar to those that OpenAI introduced in ChatGPT. You can now chat with the AI ​​assistant, ask it questions and interrupt it, just as you would a conversation with a friend or colleague. Copilot now has four language options to choose from, and you will be prompted to select one the first time you use this updated Copilot experience.

“We rely heavily on voice,” says Mehdi. “When you use it the way we designed it, you really start to let yourself go and have conversations. Then you see the first signs of where we are going in the long term, with a vision where AI can actually help you and see what you see when you want it.”

Copilot Vision is Microsoft’s second big bet with this redesign, allowing the AI ​​assistant to see what you see on a webpage you’re currently viewing. You can ask him questions about the text, images and content you’re watching, and combined with the new Copilot Voice features, he’ll respond naturally. You can use this feature while shopping online to find product recommendations, so Copilot can help you find different options.

Copilot Vision sessions are optional and short-lived, and Microsoft states that none of the content that Copilot Vision interacts with is saved or used for training. This new experience won’t work on all websites yet, as Microsoft has set limits on the types of websites Copilot Vision works with. “We’re starting with a limited list of popular sites to ensure it’s a safe experience for everyone,” says the Copilot team. During preview, Copilot Vision also does not work for paid and confidential content.

Despite the disclaimers, Microsoft clearly has a long-term vision for these new voice and vision features in Copilot. A demo shows how to use Copilot Vision to view photos of old handwritten recipes, explain what the food is, and provide tips on how long the recipe takes to prepare. Microsoft demonstrated a similar assistance experience for Xbox games earlier this year, showing how Copilot can help you navigate Minecraft.

This next phase of Copilot also includes Copilot Daily, an audio summary of news and weather that Copilot reads as if it were a CNN anchor. It’s designed as a short clip that you can listen to in the morning and only uses content from news and weather providers that have authorized Copilot to use their content. Microsoft is cooperating ReutersAxel Springer, Hearst and them Financial Times initially with plans to add more sources over time.

Thanks to the latest models from OpenAI, Copilot can also handle more complex questions. Think Deeper is a new feature in Copilot that allows the assistant to take more time to respond, providing step-by-step answers to complex questions. It’s designed to work best when you’re trying to compare two options side by side, such as “Should I move to New York or San Francisco?”

Think Deeper is still in early development and Microsoft is putting it into Copilot Labs, a new way to test experimental features the company is still developing. Copilot Vision will also initially be part of the Labs feature and participants will be able to provide feedback on the experience. Microsoft is clearly taking a cautious approach to Copilot Vision after the backlash over security and privacy issues surrounding the initial recall. Last week, Microsoft announced that Recall has been redesigned with improved security and privacy options, and you can even uninstall the feature entirely or not enable it at all.

This updated Copilot experience will be available starting today on the iOS and Android mobile apps, on the web at copilot.microsoft.com, and through the Copilot Windows app. Copilot Voice will initially be available in English in Australia, Canada, New Zealand, the UK and the US before expanding to additional regions and languages ​​in the future. Copilot Daily will be limited to the US and UK before expanding elsewhere, and Copilot Vision will be limited to a number of Copilot Pro subscribers in the US.

If you’re like me and wondering where Copilot is going next, Microsoft’s new AI CEO has some great ideas. “Over time, it will adapt to your behavior and develop skills based on your preferences and needs. We are not so much creating a static tool as establishing a dynamic, emergent and evolving interaction,” says Suleyman. “It accompanies you to your doctor’s appointment, takes notes and follows up at the right time. It will take the burden of planning and preparing for your child’s birthday party off your shoulders. And at the end of the day, it will be there to help you think through a difficult life decision.”