Snapshot

Using computer control because no API exists is the start of review, not the end of automation design

Critical business work still happens in systems with no API, an incomplete API, or a modernization cost that keeps people operating screens. Primary-source material available by June 30, 2026 from Amazon WorkSpaces for AI agents, Windows 365 for Agents, Microsoft Copilot Studio and Foundry, OpenAI, and Anthropic connects computer control with isolated workspaces, enterprise identity, live intervention, and auditability. The ability to click a screen and the ability to own a production workflow remain materially different capabilities.

24

Primary sources

Official product material is combined with computer-use evaluation papers.

2 modes

Execution model

Choose attended or unattended execution for each workflow.

7 controls

Production minimum

Isolation, identity, restriction, approval, interruption, reset, and verification must work together.

20.6%

Long-horizon caution

Even the best model in the primary OSWorld 2.0 setting completed a limited share of long workflows.

What Changed

By the end of June, computer use was expanding from a model feature into a managed place to work

Amazon WorkSpaces for AI agents became generally available as a managed environment for operating ERP systems, CRMs, mainframes, and proprietary applications inside existing identity, network-isolation, and compliance boundaries. Microsoft describes Windows 365 for Agents in terms of Cloud PC checkout and return, Entra ID and Intune management, attended or unattended execution, and real-time intervention. OpenAI's Codex Remote shows how a person can inspect outputs, diffs, tests, and approvals from a phone without directly exposing the execution host, while Anthropic and OpenAI computer-use guidance warns about prompt injection and unintended actions.

This week's signal

Reaching software that has no API is only a new starting point. Deployment depends on whose authority the agent uses, where the session runs, and whether a person can stop it while it is operating.

Seven Conditions

Define at least seven conditions before treating a workflow as a production candidate

1. Isolated workspace

Use a Cloud PC, virtual machine, or container that can be reset or reproduced per job instead of a person's everyday computer. Limit the impact of a wrong action or hostile screen content.

2. Dedicated identity

Preserve whose work is being delegated while operating with an agent identity or short-lived credential. A shared account cannot establish the operator and approver of each action.

3. Application and network restrictions

Limit applications, URLs, file areas, clipboard behavior, and external connections to what the workflow needs. Visual control does not require unrestricted navigation.

4. Attended or unattended mode

Use unattended execution for predictable, low-risk processing and attended or hybrid execution for financial, medical, legal, or employment decisions. Do not set one autonomy level for an entire product.

5. Approval and live interruption

Place a gate before send, commit, delete, payment, or record update, and make credentials or sessions revocable while the run is still active.

6. Reproducible initial state

Fix or record window layout, application version, login state, and input artifacts. If every retry starts from a different state, failures and evaluation results cannot be compared.

7. Post-execution verification

Re-read the target system and deterministically inspect changed values, created records, or delivery status instead of accepting the agent's statement that the task is complete.

Execution Modes

Choose attended or unattended execution by failure impact, not convenience

Good unattended candidates

  • Low-risk reconciliation with fixed inputs and success criteria
  • Scheduled work that can be rolled back automatically
  • Read-heavy collection and report generation
  • High-volume work where waiting for a person is the bottleneck

Good attended candidates

  • Changes affecting money, health, legal status, or employment
  • Decisions requiring context beyond what is visible on screen
  • Work with frequent ambiguity or multiple target candidates
  • Actions that are difficult to reverse after commitment

A practical hybrid pattern

The agent gathers data and prepares fields without supervision, a person reviews the target, evidence, and diff, and the agent completes the write. Only consequential boundaries return to a person.

Concrete Workflows

Separate use cases by the point of commitment, not by the buttons they require

Claims processing

The agent can open forms and attachments, match a policy number, and prepare fields. Payment category and amount should be presented with source evidence before commitment.

Patient-record updates

Separate appointment or contact changes from diagnosis and medication changes. Reconfirm patient identity and request source, and exclude clinical judgment or require professional approval.

Trade reconciliation

Collect quantity, price, and date from multiple screens and list mismatches. Route correction or settlement through approval levels based on transaction count and value.

Invoice entry

Extract supplier, amount, tax, and due date and prepare the entry. Stop before saving when duplicate invoices, supplier changes, or threshold exceptions appear.

Research Evidence

In long computer workflows, state loss and skipped verification matter more than isolated click accuracy

WebArena, VisualWebArena, WorkArena, OSWorld, and AndroidWorld report large gaps between agents and people in realistic web, desktop, and mobile environments, especially as interfaces change and action sequences grow. OSWorld 2.0, released on June 28, includes long workflows requiring an average of 318 tool calls. The best model in its primary setting fully completed 20.6% of tasks. Failures included missed information arriving mid-task, implicit state, insufficient verification, and continuing with a guess rather than asking the user.

Short demonstrations therefore cannot be translated directly into production automation rates. Dividing work into short, verifiable segments with explicit state checks is a more defensible design.

Evaluation

An evaluation case needs initial and final system state, not only screenshots

Completion state

Inspect target-system values, created records, and delivery history directly. Do not treat an on-screen success message as sufficient proof.

Forbidden actions

Confirm that the agent never saved before approval, moved to an out-of-scope customer, transmitted data externally, or removed a protected file.

Recovery behavior

Test session disconnects, unexpected dialogs, layout changes, and expired credentials, checking that the agent stops or resumes safely.

Human handoff

Measure whether the agent can present the decision needed, current state, completed actions, and remaining options. A generic failure message is not enough.

Rollout

The first production scope should contain one workflow, one environment, and one commit action

Start with one application, fixed input material, and one type of consequential action. Automate reading and field preparation first, keeping a person before save. Allow low-risk commits only after execution logs and final-state checks are stable. When interfaces change, improve safe stopping and reproducibility before widening permissions or adding more workflows.

Takeaway

The condition for letting an agent operate software without an API is not visual competence alone. The session must run in isolation, with limited identity and authority, live human interruption, and verifiable final state.