ChatGPT gains the power to see, hear, and speak
Giving ChatGPT more multi-modal capabilities makes it even more powerful, and again changes the AI playing field.
Key takeaways
- The update will allow users to have voice conversations with the AI chatbot and interact with it using images as well, the firm said in a blog post on Monday.
- The AI firm believes the new voice technology is capable of crafting realistic-sounding synthetic voices from just a few seconds of real speech, and could opens doors to many creative applications.
- Another major update to the AI chatbot allows users to upload an image and ask ChatGPT about it.
Cite or link to this article
Griffin, M. (2023) 'ChatGPT gains the power to see, hear, and speak', 311 Institute, 14 September. Available at: https://www.311institute.com/chatgpt-gains-the-power-to-see-hear-and-speak/ (Accessed: 1 October 2026).
ChatGPT has a new upgrade that lets the viral Artificial Intelligence (AI) tool “see, hear, and speak”, according to OpenAI. The update will allow users to have voice conversations with the AI chatbot and interact with it using images as well, the firm said in a blog post on Monday.
“ChatGPT can now see, hear, and speak,” the firm also said in a post on X.
The features will be rolled out “over the next two weeks” and enable users to “use voice to engage in a back-and-forth conversation” with the AI assistant.
With the new features, ChatGPT can be used to “request a bedtime story for your family, or settle a dinner table debate,” according to the company, bringing it closer to the services offered by Amazon’s Alexa or Apple’s Siri AI assistants.
Post on X
ChatGPT can now see, hear, and speak. Rolling out over next two weeks, Plus users will be able to have voice conversations with ChatGPT (iOS & Android) and to include images in conversations (all platforms). https://t.co/uNZjgbR5Bm pic.twitter.com/paG0hMshXb
OpenAI (@OpenAI) September 25, 2023
Providing an example of how the feature works, OpenAI shared a demo in which a user asks ChatGPT to come up with a story about “the super-duper sunflower hedgehog named Larry.”
The chatbot replies to the query with a human-like voice and also responds to questions such as “What was his house like?” and “Who is his best friend?”
OpenAI said the voice capability is powered by a new text-to-speech model that generates synthetic human-like audio from just text and a few seconds of sample speech, the company said.
“We collaborated with professional voice actors to create each of the voices. We also use Whisper, our open-source speech recognition system, to transcribe your spoken words into text,” the company said.
The AI firm believes the new voice technology is capable of crafting realistic-sounding synthetic voices from just a few seconds of real speech, and could opens doors to many creative applications.
However, the company also cautioned that the new capabilities may also present new risks “such as the potential for malicious actors to impersonate public figures or commit fraud” – which is a standard byline now for almost everything they release into the public domain.
Another major update to the AI chatbot allows users to upload an image and ask ChatGPT about it.
“Troubleshoot why your grill won’t start, explore the contents of your fridge to plan a meal, or analyze a complex graph for work-related data,” OpenAI explained.
This new feature, according to the company, also lets users focus on a specific part of the image using a drawing tool in the ChatGPT mobile app.
This kind of multimodal recognition by the chatbot has been forecast for a while, and its new understanding of images is powered by multimodal GPT-3.5 and GPT-4. These models can apply their language reasoning skills to a range of images, including photographs, screenshots and documents.
OpenAI said the new features will roll out within the next two weeks in the app for paying subscribers of ChatGPT’s Plus and Enterprise services.
“We’re excited to roll out these capabilities to other groups of users, including developers, soon after,” the AI firm said.
FAQ
Why does this matter?
Giving ChatGPT more multi-modal capabilities makes it even more powerful, and again changes the AI playing field.
About the author
Matthew Griffin Founder, 311 Institute
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath."
Read full bio
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath." 15-time best selling author of the "Codex of the Future" series, Matthew is the Founder and Futurist in Chief of the 311 Institute, a global Futures and Deep Futures advisory firm working with royal households, world leaders, G7, G20, and G77 governments, NGOs, and multi-national mid and mega cap firms to help them explore, shape, and lead the next 50 years of business and society.
An award-winning YouTube creator with over a million followers, with an unrivalled global reach and impact, Matthew is a highly sought-after international keynote speaker, lecturer, and mentor who collaborates with global leaders through the United Nations Alliance of Civilizations (UNAOC) and United Nations General Assembly (UNGA) to shape pivotal initiatives such as the UN’s AI for Humanity program, the United Nations Conference of the Parties (UN COP), and the World Economic Forum in Davos.
As the former Global Head of Cloud, National Security, and Enterprise Sales for companies including Atos, Dell-EMC, and IBM, Matthew has a proven track record of building multi-billion dollar business units and turning failing divisions into market leaders. His ability to identify, analyse, and communicate the implications of hundreds of emerging technologies and trends is unparalleled, and his insights are trusted by many of the world’s most respected organisations, including ABB, Accenture, Adidas, AON, ARM, BCG, Centrica, Citi, Coca-Cola, Dentons, Deloitte, Dow Jones, EY, Google, KPMG, Lego, Legal & General, LinkedIn, Microsoft, PepsiCo, Qualcomm, RWE, Samsung, Siemens AG and Siemens Energy, T-Mobile, UBS, VISA, Walmart, Workday, Worldpay and many others.
Regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, Matthews mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.
What future do you need to see?
Choose one to get started on AI and intelligence and the future of your organisation.
Sources and further reading
- View on X → twitter.com
Source: first published by the 311 Institute on 14 September 2023. Cite as: Griffin, M. (2023). ChatGPT gains the power to see, hear, and speak. 311 Institute. https://www.311institute.com/chatgpt-gains-the-power-to-see-hear-and-speak/
You are welcome to quote this article with credit and a link to the original.