Before accepting an update
The average score improves, but the user still cannot rely on the agent
Consider an agent that reads deadlines from documents and enters them in a calendar. This is a hypothetical example for explaining evaluation design. A new model handles more documents and improves the overall success rate. Yet it still saves some deadlines in the wrong time zone or creates two copies of an event when retrying an interrupted operation. Should the higher average be sufficient to deploy the update?
What troubles the user is missing their deadline or having to investigate which event is correct. My judgment is that evaluation for deployment needs to explain improved capabilities separately from unacceptable failures. Public benchmark scores cannot establish a passing grade applicable to every business process.
This requires reading what the grader accepts as correct, alongside what the agent did. Solving a task once, solving it consistently under the same conditions, and producing something the user can use without repair are different questions.
What the score covers
Public evaluations help reveal the distance to the work being delegated
The original AgentBench paper evaluates reasoning and decision-making in eight interactive environments. The original GAIA paper builds 466 questions combining reasoning, inputs such as images, web browsing, and tools. The former examines multistep actions in environments, while the latter examines reaching answers using multiple sources of information. Neither directly grades an internal requirement to save exactly one event at the correct time.
The original OSWorld paper supplies initial states and execution-based evaluation for 369 tasks involving web and desktop applications and file operations. Such evaluation helps move attention from plausible explanations to the state that remains after execution. Actions completed on public tasks may still fail with a different application version or permission arrangement.
Using a benchmark involves more than sorting candidates by an overall score. Examine which capabilities required by the intended work are included and which conditions are absent. For a calendar job, check the target calendar, time zone, and treatment of duplicates as well as document reading. Understanding the test’s scope reveals what needs additional inspection.
The difference across repeated trials
One success in eight attempts and success on all eight answer different questions
The original τ-bench paper tests agents supplied with APIs and business rules in simulated retail and airline environments, interacting with users played by language models. It compares the database after a conversation with the expected state. This research environment tests dialogue, tools, and rules together; it is distinct from deployment testing with actual customers.
The paper distinguishes pass@k, success at least once across multiple trials, from pass^k, the probability that every trial succeeds, averaged across tasks. In its retail experiment, a GPT-4o function-calling agent exceeds 60% average success, while pass^8, success on all eight trials, falls below 25%. Generated dialogue introduces variation even for the same task, so one success does not establish consistency.
The significance of the two measures depends on the work. When a person selects one of several drafts, obtaining at least one usable candidate has value. For changes to an existing calendar, repeatedly saving until something works and choosing the best result afterward is unsuitable. Repeated evaluation should restore the initial state and inspect duplicates and partial changes alongside success.
An action that prevents the next one
A polite response can conceal the loss of a required operation
The τ-bench appendix describes a user seeking to exchange two items from one order. The agent forgets the rule allowing only one exchange per order and exchanges one item first. It should have gathered the necessary information and handled both items in one operation. The first action works, but the remaining request becomes impossible to execute. Tone and the ability to call a tool do not expose that failure.
The calendar example likewise needs separate checks for reading the document and preserving existing events. A higher average from improved reading does not reduce the user’s burden if the agent saves to the wrong calendar or adds duplicates. Before deployment, decide whether success on other, easier tasks can compensate for a consequential failure.
I would investigate an update that produces a prohibited change independently of its average score. Conversely, seeing no such failure during evaluation does not prove it will never occur. Recording the tested conditions, trial counts, and untested areas permits a narrower explanation of what deployment is justified.
Inspecting the grader
A low score can also reflect a defect in the test
The SWE-bench Verified announcement illustrates the need to examine evaluation itself. OpenAI identifies underspecified issues, tests specific enough to reject valid solutions, and problems configuring execution environments, then releases 500 tasks screened by developers. The requirement, grader, and environment need inspection before their score is trusted.
In its example, an issue about an unused library parameter is graded by a test requiring a particular warning and its exact wording. If that condition is absent from the issue supplied to the agent, a reasonable fix can still fail. This provides a concrete reason why stricter grading does not automatically mean more trustworthy grading. Human screening also does not guarantee measurement of every quality dimension.
Calendar grading needs the same scrutiny. Does the test reject an equivalent instant saved in another time-zone representation merely because the strings differ? Or does a correct description of the time allow an event saved to the wrong calendar to pass? Inspect both the rejection of valid changes and the acceptance of invalid ones.
Even with tasks from actual work
Historical payouts do not directly establish the benefit of deployment
The original SWE-Lancer paper collects over 1,400 software engineering tasks from Upwork, with historical payouts totaling $1 million. Implementation tasks use end-to-end tests triple-verified by experienced engineers; managerial tasks compare the model’s choice among technical proposals with the original manager’s choice. The research measures implementing code and selecting an approach through different grading methods.
This makes the work more understandable than an abstract count of problems. Historical payments, however, are not revenue actually earned by an AI system. Matching a manager’s choice does not establish that the proposal was optimal under every condition. An economic assessment needs to distinguish measured outcomes from operating costs and time spent on human review.
For calendar work, do not count an automatic entry as the entire benefit. Include time spent checking the date and destination, removing duplicates, and rereading the document when necessary. Does delegation reduce effort compared with direct human entry or extracting the date for a person to save? That comparison provides a reason to expand the scope entrusted to the agent.
Comparing updates
Hold conditions steady and retain scores separately from explanations
The original BrowserGym paper presents common observation and action interfaces for several web evaluation environments. AutoGen v0.4 describes message tracing and debugging, among other features. Consistent experimental conditions and inspectable execution support an explanation of the deployment decision. Even detailed records still require a separate definition of a correct saved time.
Compare the old and new configurations using the same documents, existing events, permissions, time zone, and application version. Changing multiple conditions together makes improvements and regressions harder to attribute. Alongside completion counts, retain the source consulted, interpreted time, destination, submitted arguments, and saved state. These records should distinguish misreading the document from converting the time incorrectly or selecting the wrong calendar.
Good results on tasks used for tuning leave performance on new documents unresolved. Separate retesting previous failures from testing cases that were not used for adjustments. Define clarification or withholding a change as correct when a document lacks a time or calendar permission is absent. Count necessary deferrals separately from unnecessary returns to a person, so the evaluation distinguishes useful restraint from a system that delegates everything back.
Completion and remaining changes
Was exactly one correct event saved? Record duplicates, wrong destinations, and partial changes separately.
Consistency by condition
Separate document and permission conditions and restore initial state for repeated trials. Read the failures hidden by an average.
Effort returned to the user
Distinguish necessary clarification from unnecessary handoffs. Compare investigation and correction time as well.
Making a deployment decision
Pair the work delegated with the conditions that should stop an update
Building effective agents describes adding complexity as needed and choosing between predefined paths and decisions delegated to a model. If evaluation finds that unrestricted retries increase duplicates, returning to input preparation is an available choice. Deployment can be granted for tested work and conditions, rather than for the entire agent at once.
The proposed acceptance conditions extend beyond a success-rate target. Define the work, repeated tests, unacceptable failures, reasons to defer, and user effort in advance, then compare updates against the same criteria. Adequate trial counts and passing thresholds depend on the consequences of failure and the scope actually examined. A few successful examples cannot establish long-term consistency.
Connecting a higher score to work the user can rely on requires showing what improved, which failures remain, and who repairs them. Evaluation used for that explanation can recognize progress while identifying updates that should be withheld. Deployment is supported by a high score together with an understanding of the work that the score leaves unresolved.