ChatGPT sees and speaks
On 25 September 2023 OpenAI began opening image input and voice conversation in ChatGPT to Plus and Enterprise users over two weeks. The GPT-4V system card came out with it, for the model announced with GPT-4 in March but opened only to a limited circle.
Why it matters
A model that reasons over photos, screenshots and documents became available in a mass product, not only to researchers. Voice became a second way of talking to the same system.
Voice has five options made with professional actors by a new model synthesising from text and a few seconds of sample speech; Whisper recognises the speech. Because of the risk of impersonation, voice was limited to voice chat, and direct statements about people in photos were restricted by technical measures. Be My Eyes tested the model from March to early August 2023 with nearly 200 blind testers, and by September with 16,000 users and 25,000 descriptions a day. OpenAI warns that it reads non-Latin scripts poorly.