Computer-use agents: a production-readiness evaluation

Computer-use systems interpret screenshots or accessibility state and issue UI actions. Product names and surfaces change quickly: OpenAI’s standalone Operator was integrated into ChatGPT agent in 2025, and OpenAI’s current help documentation now directs longer tasks to ChatGPT Work. Verify the current surface before publishing setup instructions.

These systems can complete some UI tasks and fail others. This article is a documentation- and threat-model review, not proof that a particular vendor, website, or workflow meets your success target. OpenAI’s original Operator announcement explicitly described limitations, and its current agent help page shows why product instructions must be re-verified.

We covered the basics in Browser agents and computer use. This one goes deeper: production-readiness criteria, failure modes, and a unit-economics worksheet.

The production reality

Use these as design hypotheses to test in a shadow deployment:

Pattern 1: Narrow tasks dominate. Production deployments succeed for specific, well-defined tasks. Not “do anything.” Not “operate any website.” Specific workflows on specific sites.

Pattern 2: Heavy scoping. Tasks are scoped tightly. The agent is allowed only certain actions on certain sites. Anything outside the scope triggers stops, not improvisation.

Pattern 3: Compare recorded workflows with autonomous exploration. Steps defined once and replayed with adjustments are easier to reason about, but reliability still has to be measured against the target UI variants.

Pattern 4: Human-in-the-loop for consequence. Any action with significant consequences (financial, legal, customer-facing) goes through human review.

Pattern 5: Aggressive monitoring. Every action logged. Anomaly detection. Kill switches. Operations team watching dashboards.

Pattern 6: Cost discipline. The economics matter. Many “AI does it all” theoretical deployments don’t pencil out against human or RPA alternatives.

Pattern 7: Specialized vs general. Production deployments tend to use specialized models (or specialized configurations) rather than general computer-use models for everything.

Together, these patterns define a conservative candidate architecture for a shadow test.

Candidate workloads for computer use

These categories may justify an experiment when a supported API is unavailable. They are not proof that computer use is the right tool:

1. Data extraction from sites without APIs

Many enterprise tools, government portals, and small B2B services don’t have APIs. Or have APIs with significant gaps. Computer-use agents can extract data by operating the UI.

Candidate examples:

• Pulling invoices from 50+ vendor portals.
• Extracting case data from court system websites.
• Scraping competitor pricing pages.
• Aggregating data from compliance reporting portals.

Compare against an API integration, deterministic browser automation, and the current human process. Computer use wins only if measured quality, risk, maintenance, and cost are acceptable.

2. Form filling at scale

Submitting the same kind of form to many different sites. Each site is slightly different; an API would be ideal but doesn’t exist.

Examples:

• Government applications (each agency has its own portal).
• Compliance filings.
• Customer onboarding into vendor systems.
• Account setup for SaaS tools.

3. UI testing and quality assurance

Computer-use agents make good QA testers. They can navigate apps, attempt user flows, report issues.

Examples:

• End-to-end testing of web apps.
• Visual regression testing.
• Accessibility audits.
• User-flow validation across multiple devices.

This is RPA-adjacent but with AI’s flexibility to handle UI changes.

4. Cross-application workflows

Tasks that span multiple applications without a single integration point.

Examples:

• Pulling data from a CRM, formatting it, uploading to an analytics tool.
• Taking customer support tickets, creating tasks in a project tool, updating status in a CRM.
• Aggregating reports from multiple internal tools.

When you can’t or won’t integrate the apps directly, an agent provides a flexible bridge.

5. Repetitive multi-step processes

Tasks the same human keeps doing.

Examples:

• Onboarding new customers through a 30-step process.
• Reconciling data between two systems weekly.
• Generating periodic reports that require pulling from multiple sources.

If a process is well-defined, repeats often, and is currently done by humans clicking — it’s a candidate.

Where they still fail

The other side: tasks that require especially strong evidence before production use.

1. Tasks requiring judgment

“Find me a good vendor.” Agents can navigate vendor sites; they can’t make the judgment about which is good for your specific needs.

2. Tasks with novel UI patterns

A new site, never seen before. Agents struggle to discover unusual UI conventions. They work better on common patterns (forms, lists, navigation menus) than on bespoke designs.

3. Tasks with strong anti-bot measures

Many sites actively detect and block automation. Agents can sometimes circumvent (with effort), but it’s a perpetual cat-and-mouse. Often not worth it.

4. High-stakes individual actions

Sending a payment, signing a legal document, posting publicly on behalf of someone. The blast radius of a wrong action is large; human review is essential.

5. Tasks requiring real-world context

The agent only sees what’s on screen. It doesn’t know your relationship with that customer, your team’s recent context, the political situation. Context-poor tasks fail.

6. Open-ended exploration

“Find the best deal” or “research this person thoroughly” — tasks without clear completion criteria. Agents either spin or stop too early.

The architecture

A production candidate should have these layers:

┌─────────────────────────────────────┐
│ Orchestration                       │ Schedules, retries, escalations
├─────────────────────────────────────┤
│ Task definition + scope             │ What the agent does and doesn't do
├─────────────────────────────────────┤
│ Agent runtime (Computer Use SDK)    │ Anthropic / OpenAI / Browserbase
├─────────────────────────────────────┤
│ Browser / desktop environment       │ Isolated, sandboxed
├─────────────────────────────────────┤
│ Authentication and session          │ Credentials, cookies, MFA handling
├─────────────────────────────────────┤
│ Result handling                     │ Capture, validate, store
├─────────────────────────────────────┤
│ Monitoring + alerts                 │ Real-time observability
└─────────────────────────────────────┘

We’ll go through each.

Task definition

The single most important step. Define what the agent does, narrowly.

A good task definition includes:

Trigger. What initiates the task? (Schedule, event, manual.)

Inputs. What data does the agent have? (Specific record, structured form data.)

Scope. Which sites, which actions, which paths through the UI.

Success criteria. What does completion look like?

Stop conditions. What ends the task early?

Output. What data does the agent return?

Error semantics. How are failures categorized and reported?

A poorly defined task: “Submit our weekly compliance report.”

A well-defined task:

Task: Submit weekly compliance report to portal X.

Trigger: Cron, every Monday at 9 AM.

Inputs:
- Report data file (CSV) from /reports/weekly.csv
- Submitter info from environment variables (name, ID).
- Credentials from secrets manager.

Scope:
- Site: https://portal.example.gov/submit (and subpaths)
- Allowed actions: navigate, click, type, upload, submit, screenshot.
- Forbidden: visit external sites, change account settings, navigate away from submission flow.

Success criteria:
- Receive confirmation page with submission ID.
- Capture submission ID.

Stop conditions:
- Confirmation received: success.
- CAPTCHA: escalate to human.
- Login failure: escalate to human.
- Form validation error: report and stop.
- Timeout 5 minutes: report and stop.

Output:
- Submission ID.
- Screenshot of confirmation page.
- Timestamp.

Errors:
- Validation: log, notify owner, do not retry.
- Auth: log, notify ops, do not retry.
- Network: retry once, then escalate.

This level of specificity is what production looks like. “Submit the report” is what demos look like.

Scope enforcement

The scope isn’t just a description; it’s enforced at runtime.

URL allowlist. The agent can only navigate to URLs matching a defined pattern. Outside the allowlist, navigation is blocked.

Action filtering. Only certain action types are allowed. Wholesale “operate the computer” gives way to specific allowed actions.

Element filtering. Some pages have elements the agent should never interact with (settings, logout, dangerous buttons). These can be filtered out of the perception layer.

Time limits. Tasks have hard maximums. If not done in N minutes, abort.

Step limits. Tasks have step maximums. Same logic as agent loops.

Implementation varies by platform — Anthropic Computer Use, OpenAI Operator, Browserbase all have different mechanisms. The principle is universal: enforce scope at the runtime, not just describe it in the prompt.

Authentication

A perpetual challenge. Production deployments need to authenticate the agent’s session.

Pre-authenticated sessions. A human logs in once; the session cookies/tokens are captured; the agent operates within that session. Refreshes when needed.

Service accounts. Dedicated accounts for the agent (where the site supports them). Scoped permissions, audit logging.

Credential injection. The agent receives credentials at runtime, uses them to log in, then discards them. Secure storage and handling required.

MFA handling. A real challenge. Options:

• Use TOTP secrets the agent can compute.
• Route MFA to a human for approval.
• Use accounts/sites that allow API tokens instead of MFA.

OAuth. For modern sites, OAuth flows work well — the agent gets a token from a flow approved once by the human.

The pattern: agents should never have humanlike access to your accounts. They should have scoped, auditable, terminable credentials.

Result validation

When the agent claims success, validate.

Capture artifacts. Screenshots, downloaded files, output data. Don’t trust the agent’s report; check the evidence.

Verify success conditions. Did the form actually submit? Is there a confirmation? Was the data correct?

Cross-check. If you can verify success through a different channel (an API, an email confirmation, a database check), do it.

Anomaly detection. Was this run unusually long, unusually short, unusually costly? Investigate outliers.

The pattern: assume the agent might be wrong. Have verification independent of the agent’s self-report.

Error handling

Computer-use tasks fail in many ways. Categorize and handle each:

Network errors. Site down, timeout. Retry with backoff.

Auth failures. Login failed, session expired. Refresh credentials or escalate.

UI changes. Site changed; expected element not found. Stop, alert maintenance.

Validation errors. Form input rejected. Log, notify, do not retry blindly.

Anti-bot detection. CAPTCHAs, blocks. Escalate; potentially blacklist the site.

Agent confusion. Agent stuck, looping, going off-script. Kill, log, investigate.

Quota / rate limit. Site rate-limited the agent. Backoff and retry, or schedule for later.

Each category has different response semantics. Bad pattern: “agent failed, retry.” Good pattern: “agent failed in category X, follow recipe X.”

Monitoring

Every action logged, every run tracked, every anomaly surfaced.

Per-run logs:

• Start/end timestamps.
• All actions taken.
• All screenshots.
• Outcome (success/failure/escalation).
• Cost.
• Performance metrics.

Per-run dashboard: Operations team can see active runs, recent failures, queue depth.

Aggregate metrics:

• Success rate per task type.
• Latency distribution.
• Cost per run.
• Anomaly rate.

Alerts:

• Success rate drops below threshold.
• Cost per run spikes.
• Specific failure types increase.
• Site UI may have changed (multiple recent failures on same step).

This monitoring is what catches issues before they become incidents.

The economics

The blunt question: is computer use cheaper than the alternative?

Costs to measure: model input/output, screenshots, browser runtime, proxies, storage, retries, failed runs, human review, incident handling, and engineering maintenance.

Alternatives:

• Human process: use the organization’s fully loaded role cost and measured handling time.
• RPA tools: lower per-run cost but require structured automation.
• Direct API integration: much cheaper per-call, but requires the API to exist.
• Outsourced process: use actual contract cost, quality, lead time, and privacy constraints.

The economics favor computer-use when:

• The site has no API.
• The task is long enough that automation pays back fixed costs.
• Volume is high enough that human time accumulates.
• Site is relatively stable (low maintenance burden).

The economics don’t favor computer-use when:

• An API exists (just use it).
• The task is short and infrequent.
• The site changes constantly.
• The task has too many edge cases (high maintenance).

A useful exercise: estimate cost per task in computer-use vs human. Multiply by volume. Compare.

Production patterns that work

Patterns to evaluate:

Pattern 1: The “recorded recipe” approach

For high-volume, narrow tasks: record the workflow once with explicit steps, then have the agent replay it on each input with minor adjustments.

This is closer to traditional RPA but with AI’s flexibility to handle minor variations (e.g., a button moved slightly, an extra confirmation dialog).

Compare reliability with autonomous exploration on the same test cases; do not assume the recorded recipe wins every variant.

Pattern 2: The “extract and submit” split

Many workflows have two phases:

• Extract data from somewhere.
• Submit data somewhere.

Splitting these into separate agent runs (or recipes) is cleaner. Each phase has clearer success criteria. Failures in one don’t compound into the other.

Pattern 3: The “human checkpoint” pattern

The agent does prep work autonomously, then surfaces a “ready to act” state for human approval. Human reviews, approves, agent executes.

Used for: payments, public posts, sensitive submissions. The agent saves time on prep; the human catches errors.

Pattern 4: The “specialist agent” pattern

Rather than one general agent, have specialized agents for specific tasks. Each is tuned, tested, and maintained for its specific workflow.

A general “operate any website” agent is hard to maintain. A “submit our weekly compliance report” agent is straightforward.

Pattern 5: The “fallback to RPA” pattern

For tasks where AI flexibility isn’t actually needed (the site is stable, the workflow is fixed), fall back to traditional RPA (Playwright scripts, Selenium). Cheaper, faster, more reliable for those cases.

Use computer-use specifically when AI’s flexibility adds value.

Pattern 6: The “batched run” pattern

Don’t run agents on-demand for high-volume tasks. Batch the work; run agents in parallel on a schedule.

E.g., instead of “user submits a request; agent runs immediately,” queue requests, run agents on a 15-minute batch. Smooths the load, simplifies the architecture.

What can go wrong

A short list of common failure modes:

The site changed. A redesign can invalidate selectors, visual assumptions, or recorded steps. Detect this in canaries before customer-facing execution.

Anti-bot detection caught up. The site implemented bot detection. Agent runs increasingly fail. Eventually account is banned.

Cost spiral on a stuck task. An agent loops on a confusing page and continues making billed model or browser calls. Enforce external step, time, and cost limits.

Wrong action taken. Agent clicked the wrong button. Cancelled an order instead of confirming. Or sent a message to the wrong person.

Stuck on MFA. Agent can’t get past MFA. Production runs back up. Queue grows.

Account banned. Site detected unusual activity, suspended the account. All similar tasks broken until account is restored.

Credential leak. Agent accidentally exposed credentials in a log or screenshot. Security incident.

Privacy issue. Agent inadvertently captured PII in screenshots that were logged.

The controls above reduce risk but do not make every failure preventable. Test each failure mode and keep a recovery owner.

The economics: a worked model

This is a modeled scenario, not a client case, and we label it that way on purpose: an ROI model you can rerun with your own numbers beats an “anonymized case” you cannot verify.

Task: submitting recurring compliance reports to 12 different agency portals. In Estonia, think of the portals that still require form-by-form entry: e-MTA filings, Statistics Estonia questionnaires, and EU-level submissions.

The following entries demonstrate the formula; they are not evidence about Estonian or EU portals.

Manual baseline: portals × measured handling time × fully loaded labor rate.

Automated:

• Successful runs: volume × measured successful-run cost.
• Failed runs: volume × failure rate × combined run and recovery cost.
• Human review: reviewed volume × review time × labor rate.
• Maintenance and incidents: recorded engineering and operations time.
• Compliance and vendor cost: security review, data handling, browser infrastructure, and contract commitments.

Now the two numbers that decide whether any of this is real.

First, the success rate and error severity. Derive the acceptable threshold from the manual baseline and risk tolerance; there is no universal break-even percentage. Measure it in a representative shadow period.

Second, maintenance drift. Track how often target interfaces change and how long recovery takes. Name an owner and a service objective before relying on the automation.

A deployment checklist

If you’re deploying a computer-use system to production:

• Task is narrow and well-defined.
• Scope enforced at runtime, not just described.
• Step / time / cost budgets in place.
• Authentication strategy with secure credentials.
• Anti-bot considerations (use legitimate accounts; respect rate limits).
• Error categorization and handling.
• Result validation independent of agent self-report.
• Monitoring and alerting.
• Kill switches.
• Human-in-loop for consequential actions.
• Audit logging.
• Privacy/PII handling.
• Cost economics make sense vs alternatives.
• Maintenance plan for when sites change.

Each is non-trivial. Skipping any creates a risk.

Match the technology to the task

Production readiness requires narrow tasks, runtime guardrails, monitoring, human checkpoints, and measured economics.

Candidate tasks include extraction from API-less sites, repetitive form filling, and cross-application workflows. A representative shadow test must establish whether the system saves time or money without increasing error or risk.

For the wrong tasks — open-ended judgment, novel UIs, high-stakes individual actions — they’re not yet ready. Don’t try to make them.

The engineering work is matching technology to task and keeping a deterministic or human fallback. Promotion should depend on acceptance evidence, not a demo.

Pick narrow tasks. Build the guardrails. Monitor relentlessly. Maintain consistently. That’s how computer-use agents earn their place in production systems.