Atlas record: https://www.aitimeline.dev/en/events/evt-0312/
Availability · October 22, 2024
Computer use
Anthropic opened a mode in which the model looks at a screenshot, moves the cursor, clicks and types, working an interface the way a person does.
Why it matters
Models stopped needing a purpose-built programmatic interface: whatever a person can reach became reachable for them too.
Until then an agent could act in the world only through tools described in advance: an API call, a function, a plugin. Working the screen removes that limit: any old program with no machine interface at all suddenly becomes operable. Anthropic released the mode as a public beta with a direct warning that it was still experimental, at times cumbersome and error-prone, and encouraged developers to begin with low-risk tasks; the announcement itself does not recommend a sandbox. The risk is plain: a model that acts on what it sees on screen is exposed to whatever reaches that screen.
Event record
- Event date
- October 22, 2024
- Timeline date
- Event date
- Verification
- Sources gathered automatically · September 18, 2026
- Lines
- ID
- evt-0312
Announcement of 22 October 2024.
Related events
Earlier
- Extends Claude 3.5 Sonnet 2024
The same model, given sight of the screen and control of the cursor.
- Related Codex and Copilot 2021
From completing code in an editor to operating the computer itself.
Later
- Related Model Context Protocol 2024links to this record
The second way in a month of taking a model beyond its own window: there a screen, here data.
- Related o3 and tools inside the reasoning 2025links to this record
The model uses tools inside the reasoning itself rather than after it.
- Related ChatGPT as an agent 2025links to this record
Nine months after another laboratory gave a model a screen and a cursor.