A problem introduced by a change
Fixing a new inquiry lost an established escalation
A team changes its routing instructions to answer questions about a new product. The questions it tries now receive detailed answers. Yet the agent also starts answering return cases that it previously handed to staff. This is a hypothetical example from this article. Improvement on new work and deterioration on established work arise from the same change.
The reviewer needs more than an increase in the average score. Can the team use the new-product answers while reliably receiving return cases with unclear conditions? Deciding which change to accept, and what to hold until corrected, requires examining improved tasks separately from behavior that must remain.
This article connects evaluation design, versions and execution records, and action restrictions to that decision. It does not present measured evidence that organizations with evaluations necessarily deploy faster. The priority is to make the effect of a change explainable, without leaving users responsible for discovering regressions.
Improvement and preservation
Separate capabilities to improve from behavior to retain
Anthropic's January 9 guide to agent evaluations distinguishes capability evaluations for difficult work from regression evaluations that preserve work an agent already handles. It reports that Descript's video-editing agent uses separate suites for quality benchmarking and regression testing. This is a vendor-reported operational example, rather than a controlled comparison showing that two suites accelerate deployment.
For support, reading the new product's conditions and giving appropriate guidance can belong in the capability suite. Escalating unclear return conditions, avoiding another customer's order, and not reporting an unprocessed refund as complete can belong in the regression suite. Answer quality and permission to proceed with an action should not be blended into one score.
Some changes cannot be accepted merely because improvements outnumber regressions. Ten better product explanations would not offset one refund to the wrong customer. Those counts are hypothetical. The business owner needs to define in advance which failures cannot be traded off and must block acceptance.
Ask about capability improvement
Can the agent solve a previously difficult inquiry using the necessary material? Examine where it improved.
Ask about retained behavior
Did escalation, customer matching, and stopping on failure survive? Examine regressions that new successes cannot offset.
Comparable conditions
Record what changed and what stayed fixed
Anthropic distinguishes tasks, trials, graders, transcripts, and outcomes. I would apply these distinctions to a comparison between the candidate and current version. Use the same request, initial order state, available material, and expected outcome, and run each version repeatedly. Customer information or processing results left by one trial should not become the starting point of the next.
The comparison record should connect versions of the tasks and grading criteria, model and instructions, tools, sources, branch conditions, and environment. Even if the team intended only to change routing instructions, a change in the return policy read by a tool prevents attributing the score difference to the instructions alone. Conversely, a grader using an outdated policy might reject a correct answer.
This record does not aim to freeze every condition indefinitely. It explains what changed and what is being compared for the purpose of this update. If evaluation needs a new policy, evaluate the current version under that policy too. Do not directly treat an old score and a new score under different criteria as the size of an improvement.
Judging the outcome
Grade a good reply separately from a completed action
A courteous “Your refund is complete” does not complete a refund if the external order system has no refund record. In this scenario, I would separate checks of the actual customer, order, amount, and processing state from judgments of explanation accuracy and clarity. High-quality prose cannot substitute for successful processing.
Anthropic explains that code graders can be brittle to valid variations, while model graders are non-deterministic and need calibration against human judgment. Here, defined procedures can compare order IDs and refund states; criteria checked against human assessment can judge the clarity of the guidance. Every item does not need the same grader.
Before correcting a candidate that failed grading, establish whether a known correct action or answer passes the grader. If it rejects a valid paraphrase of an escalation message, the expected string might be the problem. But even an excellent explanation does not meet this scenario's outcome requirements if no case was created for the recipient. A reviewer should be able to recheck the evidence used to declare failure.
Returning to the cause
Which step produced the difference?
The Foundry multi-agent workflows public preview describes immutable versions on saving and traces of variables and branches. Its initial observability focuses on individual Foundry agents within the workflow. The AgentKit announcement also presents procedure versioning and trace-based evaluation. Agent Builder is in beta, however, with availability different from ChatKit and Evals.
These features provide material for investigating causes that the outcome alone cannot explain. When a return case is not escalated, examine whether it was reclassified as an ordinary inquiry, whether the branch changed despite the same classification, or whether the escalation tool failed. Depending on the cause, the correction belongs in the instructions, procedure definition, or external connection.
The coverage of an execution record matters too. Seeing a tool call on screen does not necessarily prove that the external system saved a case. Connect the call's input and response to the external outcome. A product's observation features cannot establish complete verification of every external action.
Restrictions and evaluation
With a stopping mechanism, still test whether valid cases proceed
The AgentCore Policy and Evaluations preview announcement describes restricting tool calls through an associated Gateway and evaluating execution quality. Policy offers enforcement and logging-only modes. The role and amount below 200 in its refund example are conditions for deciding whether an action is permitted.
Those conditions alone do not detect a refund to the wrong customer that is within the amount limit. Conversely, a restriction that blocks every valid refund makes the job unusable, even with zero prohibited actions. This scenario should evaluate both proceeding on eligible cases and stopping on ineligible ones.
Pre-execution restrictions, subsequent quality evaluation, and handoffs to people serve different purposes. Quality evaluation of sampled executions in the same announcement is not evidence that every call was authorized beforehand. Even a candidate that passes evaluation will not enforce restrictions in production if configured for logging only. Check that the evaluated configuration matches the one being released.
Updating criteria too
Preserving behavior differs from preserving an outdated policy
Regression evaluation need not freeze every historical output. Suppose the return window changes from fourteen to thirty days. Retaining an old answer that rejects an otherwise eligible day-twenty request could pass a regression check while violating the new business rule. Preserve conformity with the approved rule, rather than the old reply itself.
Record the policy change and establish product conditions and the starting date before updating expected outcomes. Distinguish this from relaxing criteria merely because the candidate failed. A request for clarification when the deadline cannot be established should not count as failure if it is appropriate stopping behavior. If reviewers cannot explain a unique acceptable judgment, specify the task's conditions first.
Anthropic's guide to writing tools for agents recommends representative tasks with verifiable outcomes and checks using held-out evaluation tasks. Here, I would test unused wording and conditions alongside the return cases used to revise the instructions. This distinguishes fitting the answer to a known failure from consistently preserving the required behavior.
The acceptance decision
Explain the remaining burden before users discover a regression
The initial candidate should not be accepted unchanged if it improves product explanations but loses necessary escalation. Correct the cause, then recheck improvement and preservation tasks under the same conditions. Also examine excessive handoffs, latency and cost, and the investigation staff must repeat. Optimizing automation rate alone can reduce necessary handoffs along with unnecessary ones.
Creating and maintaining evaluations takes time. A team can begin with work affected by the change and conditions where failure imposes a large burden, rather than grading every possible job immediately. Add real inquiries that automated evaluations missed from production records and staff reports. Passing pre-release evaluation does not guarantee performance on every unknown input.
Aligning versions, execution records, and grading does more than make scores manageable. It enables an explanation of what improved, what remains, and which conditions changed, so a team can accept or hold a candidate. Can it reduce the work of waiting for user complaints and investigating again after each change? Answering that question connects having evaluations to better support work.