Use AI agent evaluation to decide what work needs repair

The same “changed” answer can conceal different work

Compare two AIs that return “Your booking has been moved to Friday.” Both change a verified customer's booking R42 and save the same time and price. This service requires presenting the time and price and obtaining approval before saving.

The first run checks availability, presents the change, obtains approval, and saves. The second saves before asking for approval. This is a hypothetical evaluation example, not a measurement of a particular product. Although the answer and saved state match, the second run violates the service's conditions.

OpenAI, Google, AWS, and Anthropic material available through July 11, 2026 describes evaluating execution records and tool use alongside answers. My recommendation is to compare records with business expectations and choose repairs by the condition violated. Treating a better answer as better work requires evidence that explains actual operations.

Separate the request, execution, and remaining result

An aggregate booking score can let a natural answer offset an approval-order violation. This article records three distinct checks. They are an organizational choice for explaining verdicts, not an industry-wide taxonomy.

A changed price can justify another lookup and renewed approval. If the customer refuses, leaving the booking unchanged can be correct. Define results from requests and conditions rather than requiring identical operations every time.

OpenAI tracing collects events for subsequent judgment

OpenAI Agents SDK tracing documentation records generations, function-tool calls, handoffs, and guardrails. It connects an operation's identifier with individual spans carrying start and end times, and supports custom events.

Those records do not automatically establish a correct booking change. The service needs to connect presented content, the approval target, save requests, and saved results. Do not assume standard SDK instrumentation captures every business approval or external-system write.

Even with the correct booking ID, a price changed after approval is not covered by the earlier approval. Alongside names and timestamps, inspect which version of which content was approved. Execution evidence becomes useful when it identifies decisions and corresponding updates as well as order.

Traces can contain inputs and outputs, so design the necessary evidence together with its capture scope. The SDK provides settings controlling sensitive-data inclusion. Consider reviewed reproduction data or sufficient identifiers where retaining entire original contents is unnecessary.

Google's matching modes change which paths are allowed

Google ADK’s evaluation criteria describe comparing expected and actual tool calls. It offers exact matching, required calls appearing in order, and required calls appearing in any order. Exact matching is the default; answer similarity and model-judged quality are separate metrics.

The matching implementation compares arguments as well as function names. A save to R43 does not match an expected save to R42 even with the same function name. This goes beyond checking names, but depends on supplying correct expected arguments.

Exact matching can reject a repeat lookup needed after a price changes. In-order matching can allow that addition while checking required calls. However, a save to another booking among the allowed extras may still leave the required subsequence matching.

“Approval preceded saving” therefore differs from “no unapproved save occurred.” Independent reads may not require a fixed order, while prohibited writes need their own checks. Looser matching should come with explicit decisions about allowed alternatives and additional inspections.

AWS separates expected answers from session behavior

The AWS AgentCore SDK evaluation client accepts expected responses, lists of tool names, and assertions about required behavior. Responses can be associated with a particular trace, while tool trajectories and conditions apply to a session. The client collects CloudWatch spans and submits them to specified evaluators.

Its trajectory input is a list of names, not structured arguments containing booking IDs or amounts. Do not equate that input with Google ADK's argument comparison. The booking case needs separate reconciliation of approved content, save parameters, and actual reservation data.

The general-availability announcement describes evaluating specified interactions on demand and sampling production records online. Session-wide, response-level, and tool-call-level scopes expose different possible violations. An evaluation capability does not establish that every action is authorized before execution.

Inspect why the grader declares a pass

Rules can check explicit identifiers and tool ordering. People or model judges can help assess whether a price change was explained clearly. The choice depends on what is being judged and which paraphrases are acceptable.

Anthropic's evaluation explanation distinguishes execution transcripts, graders, and final outcomes, including grader errors. If a grader rejects a correct alternative phrasing or accepts a plausible success report, increasing its score may not improve the work.

For the scenario, prepare a correct save of approved values to the right ID and a confusable record that saves before approval. Compare the owner's expected judgments with the grader and investigate disagreements. Recheck the same records after changing rules or the judge model. This is a proposed check, not a measured accuracy result for a particular grader.

Measure saved results and repeated reliability

τ-bench version 1 compares the final database with an annotated goal state. It also distinguishes success in at least one attempt from consistent success across repeated attempts. Fluent answers or success once make it difficult to judge whether the same work can be delegated repeatedly.

Final-state evaluation still does not replace approval-order checks. The opening runs leave identical data yet differ on order. Conversely, correct call ordering can still end with an unsuccessful save when a tool fails. Inspect appropriate execution and actual outcomes together.

Restore the same initial data for repeated runs, separating correct changes, appropriate refusals or stops, wrong actions, and missing evidence. Evaluating only successful runs hides how often difficult cases stop. With an interrupted save response, also inspect whether the agent resends before establishing the first operation's result.

Compare which conditions improve after a change

Include customer refusal, changed prices, missing permissions, similar booking IDs, and interrupted save responses alongside ordinary changes. Record versions of the model, instructions, tools, and grading rules, then compare before and after with matched initial data and action budgets. Changing everything together makes the cause harder to explain.

Use the failure breakdown to choose a repair. Wrong targets suggest inspecting lookup and identification; postapproval changes suggest binding approval to saved content; false completion reports suggest checking results. Do not reduce every failure to a need for a larger model.

Report answer quality, trajectory violations, saved results, missing evidence, latency, and cost separately. A high score for a concise answer must not cancel a wrong-booking update. Measuring the owner's investigation time and user clarification requests also reveals whether effort increases despite maintained success.

Turn production observations into reproducible cases

Production records help find failures absent from a fixed evaluation set. A violation absent from a sample may still occur elsewhere. Show which actions were observable, including capture scope and missing records.

Review and anonymize necessary information from a new failure, then create a reproduction with initial state, request, expected operations, forbidden conditions, and outcome. Verify the next repair and check whether later changes bring the failure back. Passing accumulated cases cannot guarantee safety in situations not yet collected.

More traces can also mean more reading for owners. Supporting a repair decision requires reaching the violated condition and its evidence. Showing which booking, approval, and save ceased to correspond can make investigation easier than presenting every record.

Make evaluation evidence useful for repairing work

The four providers support different methods for recording execution and comparing it with expected behavior. Their materials do not establish a product ranking under shared conditions or a common complete audit capability. Select records and judgments from the booking service's own requirements.

Start with a small set of cases expressing the success and forbidden conditions owners consider consequential. Include disagreements among answers, execution, and saved results, with inspectable reasons for verdicts. Use existing procedures and deterministic rules where they suffice.

The goal extends beyond watching scores rise. Explain what improved, where it worsened, and what to repair next. A booking-change “completion” should connect to work the customer and owner can establish.