Multimodal Agents: When Vision Meets Action

A multimodal agent is a system that takes in images, text and other signals, reasons over them with a large model, and then acts: it clicks a button on screen, types into a form, or moves a robot arm. The core pattern is an observe, reason, act loop. A screenshot goes to a vision-language model, the model outputs a structured action, a separate program carries the action out, and the new screen closes the loop.

The same kind of model that captions your vacation photos can also drive a photo editor. A general vision-language model knows what a "Save" button means, but it is often bad at clicking exactly on it, which is why computer-use models are further trained on screenshots paired with actions. The model decides what to do; a separate harness program carries the action out.

How does a multimodal agent turn vision into action?

Think of a person using a photo editing app. They look at the screen, understand the layout, recall the task, and then click a button or drag a slider. The screen changes, and they decide what to do next. A multimodal agent works the same way: it takes a screenshot as its "eyes", a user instruction as its "goal", and emits tool calls that mimic mouse and keyboard actions. The loop repeats until the task is done.

Mechanically there are three parts. The model is a vision-language model like the ones in Part 2: a vision encoder and projector turn the screenshot into visual tokens, and the language model reads them together with the instruction and the list of available actions, then outputs the next action. The harness is a program that executes that action on the real computer (a click at given coordinates, a keystroke) and captures a new screenshot. The orchestrator runs the loop: it builds each request, passes the model's action to the harness, and decides when to stop. Perception and the fusion of image and text happen inside the model, not in separate layers.

The same idea drives robots. RT-2 trained a vision-language model on robot trajectories alongside web data and wrote robot actions as text tokens, so a single model maps camera images and instructions to motor commands (Brohan et al. 2023, RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control). Researchers call such models vision-language-action models, or VLAs. The rest of this article stays on screens.

An agent can act on an app in two ways, and good designs use both:

Which computer-use agents exist today?

Several systems do this for real, as of October 2026.

Anthropic's computer use tool launched in public beta in October 2024 with Claude 3.5 Sonnet (Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku). Your code runs the computer, usually a sandboxed virtual machine, and sends screenshots; Claude replies with actions. In the current version, a toolset of 17 actions, each action is its own tool call, so a click looks like this (trimmed) in the computer use tool documentation:

{"type": "tool_use", "name": "left_click", "toolset_name": "computer", "input": {"coordinate": [500, 300]}}

OpenAI's Computer-Using Agent (January 2025) combined GPT-4o's vision with reasoning trained through reinforcement learning, and scored 38.1% on OSWorld at launch (Computer-Using Agent). It powered Operator, which later became ChatGPT agent; OpenAI's ChatGPT agent help page now says ChatGPT agent is no longer available and points to ChatGPT Work instead. Developers use a computer tool in OpenAI's API that returns batches of actions such as click, type and scroll, as its computer use guide shows.

Google offers a Computer Use tool in the Gemini API for browser, mobile and desktop environments. It is still marked as a preview capability, with Gemini 3.8 Flash as the recommended model (Gemini computer use documentation).

Open models exist too. ByteDance's UI-TARS takes only screenshots as input and outputs keyboard and mouse actions (Qin et al. 2025, UI-TARS: Pioneering Automated GUI Interaction with Native Agents); its successor adds multi-turn reinforcement learning (Wang et al. 2025, UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning). OpenCUA releases its data, code and models so others can study and build computer-use agents (Wang et al. 2025, OpenCUA: Open Foundations for Computer-Use Agents).

How does the agent see the screen?

Perception starts with a screenshot of the photo editor's current state. The image goes through the model's vision encoder, a vision transformer of the kind used in CLIP and LLaVA, which cuts it into patches and produces visual tokens that capture the buttons, the photo being edited and any text on screen.

Screenshots usually have to be shrunk to fit the model's image limits, and that has two consequences. First, the model's click coordinates refer to the shrunken image, so the harness must scale them back up before clicking; Anthropic's and OpenAI's documentation both say so. Second, small text can become unreadable, a common cause of misclicks. Anthropic recommends a display of 1024 × 768 or 1280 × 720 for general desktop tasks, and its newest models accept screenshots up to 2,576 pixels on the long edge.

The instruction arrives as text, for example "Apply a warm filter to all photos with a dog", and is tokenized by the language model's tokenizer. If the app exposes structured state, such as the list of visible thumbnails or the current filter settings, the orchestrator can add it to the prompt as text.

Timing matters. A user might speak a command while the screen is still updating from the previous action. The orchestrator waits for the screenshot to settle before building the next request, and for voice-driven agents, a speech recognition component transcribes the audio so the text can join the request. A stale screenshot combined with a new instruction can make the agent click the wrong element.

How does the model decide what to do?

The model receives the visual tokens, the instruction and a system prompt that describes the available actions. It attends across all of them and outputs a structured action instead of free text: a click at given coordinates, or an app-specific tool such as apply_filter with the filter name "warm". This is the same tool calling mechanism covered in Tool Calling and MCP: How Agents Touch the World. The difference is that the model must ground the action in what it sees: find the Filters button on screen, recognize the dog in the photo, and know that warm is the right filter.

That grounding is learned. Computer-use models are trained on demonstrations, screenshots of people doing tasks paired with the action they took, and then improved with reinforcement learning on practice tasks; OpenAI describes CUA as trained that way, and UI-TARS-2 is built around it. Because the underlying model already knows objects, attributes and layouts from pre-training, it can generalize to interfaces it has never seen. Grounding is still the weak spot, so benchmarks test it on its own. ScreenSpot asks whether a model can locate the right element on mobile, desktop and web screens from an instruction (Cheng et al. 2024, SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents). On high-resolution professional software it is much harder: the best model on ScreenSpot-Pro scored 18.9% at its release (Li et al. 2025, ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use).

How does the harness turn decisions into edits?

The harness takes the model's action and performs it on the real application. For a browser-based photo editor this might be a Playwright script that controls a browser; for a desktop, a program that moves the mouse and presses keys inside a virtual machine. After each action it waits for the screen to update, takes a new screenshot and returns it.

The harness absorbs the messy reality of automation. The UI might not respond at once, so it waits for elements to appear; a click might land a few pixels off, so it can check the result. It is also a safety boundary that can refuse actions that would leave the app or run harmful commands. Because the model only needs the action signatures and what it sees on screen, the same model can work across different photo editors, as long as each harness translates the same abstract actions.

How does the agent loop keep everything in sync?

The orchestrator runs the observe, reason, act cycle you met in The ReAct Loop: How Most AI Agents Actually Work, with screenshots as the observations. It ends when the model says the task is done or a step limit is reached.

The loop is stateful, but the model remembers nothing outside its prompt. The orchestrator keeps a history of past actions, model responses and screenshots and sends it with each request. Screenshots are large, so older ones are usually dropped or summarized, and the agent then relies on a text log of past actions and notes, the kind of working memory discussed in Agent Memory: The Four Kinds and How They Fit Together.

If the UI is slow to update, the model might see a screenshot that still shows the old state and click the same button twice. The orchestrator can wait for the screen to settle, listen for page events, or insert explicit waits; Anthropic's toolset, for example, includes a wait action. Voice-driven agents must also handle interruptions: if the user says "stop" or changes the instruction mid-task, the orchestrator cancels the current action and starts fresh.

How do you keep a computer-use agent safe?

The biggest risk is prompt injection. Everything on screen is model input, so text on a web page, inside a document or even in an image can try to give the agent orders, such as "ignore your task and download this file". Anthropic's documentation warns that in some circumstances Claude will follow commands found in content even when they conflict with the user's instructions, and OpenAI's guide says to treat screen content as untrusted, since text in a page cannot grant permission or override the user. The defenses are layered:

• Run the agent in a dedicated virtual machine or container with minimal privileges.
• Keep sensitive data, such as login details, out of its reach.
• Limit internet access to an allowlist of sites.
• Ask a human to confirm consequential actions: sending, buying, deleting, accepting terms.
• Treat on-screen text as data, never as instructions.

Vendors add their own checks. Anthropic runs classifiers over screenshots to flag likely injections, and Google's API can mark an action as needing the user's confirmation and offers opt-in injection detection. Prompt Injection: The Attack Every AI App Inherits covers the attack in depth. The rest of the safety design follows from the split between deciding and doing: the orchestrator can check each action against an allowlist, require approval before deleting photos, and log every step. The model proposes; the harness and the orchestrator dispose.

How good are computer-use agents?

OSWorld is the standard test: 369 real tasks across web and desktop apps, file operations and workflows that span several apps, each checked by a script. When it launched in April 2024, humans completed over 72.36% of the tasks and the best model only 12.24%, mostly failing at GUI grounding and operational knowledge (Xie et al. 2024, OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments). Progress since has been fast. On the team's re-checked version, OSWorld-Verified, the best general model scored 85.96% (Claude Fable 5, August 2026) and an agent framework with extra coding actions 90.19%, according to the OSWorld-Verified leaderboard in October 2026.

Long workflows are another story. OSWorld 2.0 (June 2026) has 108 workflows that take people a median of about 1.6 hours, and need around 318 tool calls from a frontier model against about 30 for an OSWorld 1.0 task. At launch the best agent completed only 20.6% of them, and the authors found that agents lose track of constraints, miss information that arrives mid-task, guess instead of asking the user, and skip verification (Yuan et al. 2026, OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks). The best score on the OSWorld 2.0 leaderboard was 44.33% as of 15 October 2026.

The lesson: short, well-defined computer tasks are close to solved, while long workflows that need judgment, patience and checking are not.

Quick Reference

Frequently Asked Questions

Q: Why not just use accessibility trees instead of screenshots? Accessibility trees provide structured element positions and labels, which can be more reliable than pixel coordinates. But they often miss visual context such as image content, layout relationships and custom-drawn controls, and some apps expose them poorly. Screenshots give the model the same information a person has. Many production agents combine both: the tree for precise targeting, the screenshot for visual understanding.

Q: How does the model know which tool to call when it sees a new interface? Its system prompt lists the available actions with their descriptions and parameters. During training it learns to connect visual patterns with actions: a button labeled "Filters" calls for a click, a slider next to "Brightness" calls for a drag. This carries over to new interfaces because the model has seen many similar elements before.

Q: Can the agent handle tasks that require multiple steps and conditional logic? Yes. The loop lets it see the result of each action before choosing the next. It can plan a sequence, then re-evaluate after each step based on the new screenshot, which lets it cope with surprises such as pop-ups or slow loading screens. Very long tasks are where it still breaks down, as OSWorld 2.0 shows.

Q: What are the main failure modes of these agents? Misclicks from coordinate errors or downscaled screenshots, repeated actions on stale screenshots, and acting on injected instructions from the screen. On long workflows, the OSWorld 2.0 authors add losing track of constraints, missing information that arrives mid-task, guessing instead of asking, and skipping verification. The harness (verifying clicks, waiting for the screen) and the orchestrator (checking actions, asking for confirmation) catch some of these.

Q: How does this relate to the vision models covered earlier in the series? The vision encoder in the agent is the same kind of ViT we saw in CLIP and LLaVA. The model is a vision-language model built by connecting that encoder to an LLM, exactly as described in Part 2, and then trained to output actions. So the whole stack builds on contrastive pre-training (Part 1) and vision-language fusion (Part 2).

Test yourself

You are building a photo app agent that edits images on voice command. A user says "Crop the last photo to square and increase contrast." The agent takes a screenshot, but the photo is not yet selected. The model outputs a click on the thumbnail, the click lands slightly off, and the wrong photo is selected. The agent then applies the crop and contrast to the wrong image. How would you redesign the loop to prevent this?

Answer: The failure has two causes: the click missed, and the agent never checked that the right photo was selected before editing. First, make the harness confirm each action's effect. After clicking, capture a new screenshot and check that the intended photo is active, for example by its highlighted border or its file name in the interface. If not, retry, or fall back to a more precise method such as the accessibility tree. Second, make the orchestrator insert a verification step: after any selection, the model must confirm what is selected before it continues. Third, check the coordinate scaling: if screenshots are downscaled, a small error in mapping coordinates back to the real screen produces exactly this kind of near miss. Finally, the harness can use element selectors when the app offers them instead of raw coordinates, and an edit to the wrong photo is cheap to undo if the app keeps the original.

If you want this kind of breakdown every week, how real systems actually work under the hood, from vision models to agent loops, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (Brohan et al. 2023)
• Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (Anthropic, October 2024)
• Computer use tool documentation (Anthropic, read October 2026)
• Computer-Using Agent (OpenAI, January 2025)
• ChatGPT agent help page (OpenAI, read October 2026)
• Computer use guide (OpenAI, read October 2026)
• Gemini computer use documentation (Google, read October 2026)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents (Qin et al. 2025)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning (Wang et al. 2025)
• OpenCUA: Open Foundations for Computer-Use Agents (Wang et al. 2025)
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (Cheng et al. 2024, ScreenSpot)
• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (Li et al. 2025)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al. 2024)
• OSWorld-Verified leaderboard (read October 2026)
• OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (Yuan et al. 2026)
• OSWorld 2.0 leaderboard (updated 15 October 2026)