Comparing AI agents: models, retrieval, and trial allocation

Choosing a configuration

A higher score does not explain its cause

A team replaces the model in an agent answering from internal procedures and adds another search when evidence is insufficient. More answers are correct, so it concludes that the new model was better. But a comparison with an old configuration that searches only once also includes the effect of additional retrieval. This is a hypothetical example for examining comparison methods.

There are two questions: which configuration is better for the job, and which change caused the improvement? Comparing complete configurations helps answer the first. Answering the second requires separating the model, information presentation, search, and retries. Mixing the questions obscures both what to improve next and why costs increased.

This article uses BrowserGym, RE-Bench, and development tools to make those comparison conditions concrete. Instead of ranking architecture and models by general importance, it asks which change improves the same job at what burden. That question provides a basis for the next improvement beyond a configuration's complexity or a model's novelty.

Available observations

The same page can show different information to a model

The first version of the BrowserGym paper provides observations including the page's DOM, accessibility tree, and screenshots. The agent implementation decides how to process and present them to the model. It is a research environment offering a common interface for observations and actions, rather than a product that makes different business tasks equally difficult.

One example in the paper shows a search button obscured by an autocomplete menu, causing a click error. The error returns in the next observation. Images showing overlaps and feedback about the previous failure provide information for the next action beyond page text. An action failure alone cannot establish that model planning was the sole weakness.

Likewise, in the internal-procedure scenario, a model cannot use a necessary warning omitted from the search result. A comparison giving the new configuration full text but the old one only a short excerpt does not compare models with the same information. Align which source version is provided, how much of it, and in what form.

A comparison for deployment may instead use presentation suited to each model. In that case, describe the comparison as one between combinations of models and observations. Distinguishing controlled information from usable configurations helps avoid both unnecessarily weakening one candidate and attributing all improvement to its model.

Tools and responses

A shared search name does not establish shared conditions

Observation differences extend beyond browser screens. Answers can change depending on whether a search tool returns source text and citations or only a short summary. Coverage of older material, access permissions, and the number of results are also comparison conditions.

The MCP tools specification dated June 18, 2025 describes structured output and results indicating tool failures. Explicit output formats help inspect a connection. But identically shaped responses need not contain equivalent content or coverage. A common connection method does not establish equivalent search conditions.

In the scenario, the old search could turn retrieval failure into an empty result while the new search exposes the failure, changing the decision to stop. Calling that difference greater caution from the new model would miss a condition required to reproduce it. Record the actual text, gaps, and failure information passed to the model, and treat tool replacement as a configuration change.

Time and trial allocation

The same total budget can favor different trial lengths

The first version of RE-Bench studies time and trial allocation across seven ML research-engineering environments. In comparisons allocating eight total hours, thirty-minute trials scored best for the Modular scaffold, while two-hour trials scored best for AIDE, among the allocations tested. The favorable allocation differed by scaffold even with the same total time.

The comparison includes selecting good results across multiple trials. Selection is random for Scaling Law Experiment, where the score is unavailable. The best obtained score cannot be treated as performance available from an arbitrary single attempt. Nor do these bounded research tasks establish that thirty minutes or two hours is optimal for internal inquiries.

The lesson I would adopt is to compare how a budget is spent alongside the model. Repeating the same search, finding a different source, and generating several answers for selection spend extra time differently. Recording only the retry limit cannot explain what improvement the extra processing bought.

Also distinguish user waiting time from total compute time. Parallel trials may shorten the wait while retaining compute and selection work. Examine time from the request to a usable answer, call and search counts, and usage cost together. Ranking peak scores obtained with different budgets moves away from choosing a configuration suitable for the job.

Separating causes

Compare four combinations of model and additional search

Return to the internal-procedure scenario. With the same questions, source snapshot, grading conditions, and total budget, combine the old and new models with either one search or another search when evidence is insufficient. Examine a model change, a procedure change, and finally both together. Reset the initial state and repeat each configuration rather than judging a single success. The four configurations permit comparison without assigning improvement to one factor prematurely.

Additional retrieval helps only if the model identifies insufficient evidence and chooses another useful search. A new model might handle this decision well while an old one repeats the same query. That is a mechanism-level hypothesis, not a measured result for a named model. Because the factors can interact, do not add their isolated gains to predict combined performance.

For deployment selection, each configuration may instead be tuned within the budget allowed for the job. That compares the output of complete tuned configurations, rather than model capability alone. Separate tuning questions from final evaluation questions to help detect improvements fitted only to familiar inputs.

Grading and selection

Can the system select the good candidate it generated?

Anthropic's guide to writing tools for agents recommends verifiable outcomes, evaluations representative of real work, and checks on held-out tasks. For answers from procedure documents, I would verify the applicable product version and supporting evidence alongside the setting value. Examine stopping when a source cannot be retrieved separately.

When generating multiple answers, distinguish selection by an evaluation grader that knows the correct answer from selection available in production. If a user who does not know the answer must choose among five candidates, that checking effort is also a cost. Having a correct candidate in the set leaves another verification step before delivering correct guidance.

Read which questions improved and which deteriorated alongside the average score. Distinguish answers supported by additional evidence from wrong answers combining instructions for older versions. Counting appropriate stops only as unanswered questions risks tuning away necessary stopping behavior. Output requirements for the use case determine the configuration's evaluation.

Recording configurations

Keep versions that lead back to a failed execution

The AgentKit announcement presents procedure versioning and evaluation using traces. Agent Builder is in beta, and having those features does not prove improved performance. AgentLab, used with BrowserGym, also describes experiment version information and tools for inspecting observations and actions. Both provide material for linking configurations to results.

In this scenario, retain the model and instructions, sources and search conditions, time and retry limits, actual calls, and grading and selection methods. When a reviewer finds an incorrect answer, they should be able to return to the product version it described, unavailable material, and additional searches. The purpose is to explain what to change next, beyond demonstrating configuration complexity.

Records cannot guarantee exact reproduction. The BrowserGym paper describes limitations from changing live-site content and regional settings, among other factors. Distinguish comparison on a fixed internal corpus from checks on changing live sites. An improvement under one fixed condition cannot establish performance on every future screen or source.

The adoption decision

Should the next change improve capability or provide missing information?

If additional retrieval explains the initial improvement, trying that procedure with the old model has value. If the new model improves with the same information, examine the value of changing reading or judgment capability. If only the combination improves, carry that interaction and its additional cost into the decision. No particular configuration can be declared the winner before the comparison.

Alongside correct-answer counts, include user waiting time, time to find errors, and the burden of locating evidence again. More automatic answers may offer little improvement if staff must recheck every product version. Comparing models, tools, and procedures ends with whether the recipient can proceed to the next task, as well as whether the score can be explained.