Microsoft today unveils a big overhaul of its Copilot experience, adding voice and visual features to transform it into a more personalized AI assistant. As I revealed exclusively in my Notepad newsletter last week, Copilot’s new features include a virtual news anchor mode to read headlines to you, the ability for Copilot to see what you’re watching, and a voice feature that lets you talk to Copilot naturally, just like OpenAI Advanced voice mode.
Copilot has been redesigned across mobile, web, and the dedicated Windows app into a more card-based user experience and very similar to the work Inflection AI has done with its Pi custom AI assistant. Microsoft hired a bunch of people from Inflection AI earlier this year, including Google DeepMind cofounder Mustafa Suleyman, who is now CEO of Microsoft AI. This is Suleyman’s first big change at Copilot since taking control of the consumer side of the AI assistant.
“At Microsoft AI, we are creating an AI companion for everyone,” Suleyman says in an open letter today. “I truly believe we can create a calmer, more helpful and supportive technological era quite unlike anything we’ve seen before.”
Copilot now looks unlike anything I’ve ever seen from Microsoft before, with an interface that’s a big departure from what exists right now. It’s much warmer, with a personalized Copilot Discover page that’s more helpful and inviting than a text entry prompt for a chatbot. Microsoft is personalizing the entire Copilot home page based on your conversation history and, over time, will include useful searches, suggestions and relevant information.
Microsoft handed off its consumer version of Copilot to Suleyman’s team earlier this year, and it’s clearly allowed the company to experiment more with personality and customization. “What we learned from the Pi team and the [Inflection AI] people who have come to us is that they have always had a particular focus on customer needs,” says Yusuf Mehdi, executive vice president and head of consumer marketing at Microsoft, in an interview with The limit. “How they listen and what they’ve learned from these long conversations in research has definitely influenced what we’ve done here.”
Beyond the look and feel of this new Copilot, Microsoft is also ramping up work on its vision of an AI companion for everyone by adding voice capabilities very similar to what OpenAI introduced in ChatGPT. Now you can chat with the AI assistant, ask it questions and interrupt it as you would during a conversation with a friend or colleague. Copilot now has four voice options to choose from, and you’re invited to choose one when using this updated Copilot experience for the first time.
“We’re making a big bet on voice,” Mehdi says. “When you use it the way we designed it, you really start to let go and have conversations. Then you see glimmers of where we’re going to go long term, with a vision where AI can actually help you and see what you see if you want it to.
Copilot Vision is Microsoft’s second big bet with this redesign, allowing the AI assistant to see what you see on a web page you’re viewing. You can ask it questions about the text, images and content you’re viewing, and combined with the new features of Copilot Voice, it will respond naturally. You can use this feature while shopping on the web to find product recommendations, allowing Copilot to help you find different options.
Copilot Vision sessions are voluntary and ephemeral, and Microsoft says that none of the content Copilot Vision interacts with is stored or used for training. This new experience won’t work on all websites yet because Microsoft has placed restrictions on the types of websites Copilot Vision works with. “We’re starting with a limited list of popular websites to ensure a safe experience for everyone,” says the Copilot team. During preview, Copilot Vision will not even work on sensitive, paywalled content.
Despite the disclaimers, Microsoft clearly has a long-term vision for these new voice and visual features in Copilot. A demo shows Copilot Vision looking at photos of old handwritten recipes, helping to explain what the food is and offering suggestions on how long it takes to prepare the recipe. Microsoft demonstrated a similar assistive experience for Xbox games earlier this year, showing how Copilot could help you navigate Minecraft.
The next phase of Copilot also includes Copilot Daily, an audio summary of news and weather that Copilot reads as if it were a CNN host. It’s designed as a short clip you can listen to in the morning and only uses content from news and weather providers who have authorized Copilot to use their content. Microsoft is working with ReutersAxel Springer, Hearst and the Financial times initially, with plans to add more sources over time.
Copilot can also handle more complex queries thanks to the latest models from OpenAI. Think Deeper is a new feature in Copilot that allows the assistant to take longer to respond, allowing it to provide step-by-step answers to complex questions. It’s designed to work best when you’re trying to compare two options side-by-side, such as, “Should I move to New York or San Francisco?”
Think Deeper is still in early development, and Microsoft is putting it into Copilot Labs, a new way to test experimental features that the company is still developing. Copilot Vision will also initially be part of the Labs functionality, and participants will be able to provide feedback on experiences. Microsoft is clearly proceeding cautiously with Copilot Vision after the backlash surrounding Recall’s initial security and privacy issues. Microsoft revealed last week that Recall has been overhauled with improved security and privacy options, and you’ll even be able to uninstall the feature entirely or not turn it on in the first place.
This updated Copilot experience will be available today in the iOS and Android mobile apps, on the web at copilot.microsoft.com, and through the Copilot Windows app. Copilot Voice will initially be available in English in Australia, Canada, New Zealand, the United Kingdom and the United States, and will expand to other regions and languages in the future. Copilot Daily is limited to the US and UK before expanding elsewhere, while Copilot Vision will be limited to a number of Copilot Pro subscribers in the US.
If, like me, you’re wondering where Copilot is headed, Microsoft’s new AI CEO has some big ideas. “Over time it will adapt to your attitudes and develop skills built around your preferences and needs. We are not so much creating a static tool as establishing a dynamic, emergent and evolving interaction,” says Suleyman. “He will accompany you to your doctor’s appointment, taking notes and responding at the right time. He will share the load of planning and preparing your child’s birthday party. And it will be there at the end of the day to help you think through a difficult life decision.
#Microsoft #Copilot #voice #vision #biggest #redesign