From AI-written code to comparable experiment results

Can the result be used?

A smaller forecast error can still leave adoption undecided

Asked to improve a demand forecast, an AI changes code, trains a model, and reports a smaller error. But it evaluated a different period from the previous model. Did the forecasting method improve, or were only easier weeks compared? This is a hypothetical example for examining experiment support. Working code alone does not establish a reason to adopt its result.

August’s GPT-5 for developers announcement described support for code changes and tool calls. Looker MCP Server provided a way for AI to access predefined metrics. MLE-bench and RE-Bench evaluate work that uses data and compute to try solutions and compare outcomes. More material connects code, data, and analysis, but the capabilities each establishes differ.

This article argues for using the ability to produce candidates while separately establishing evidence for adopting them. Connect code, data meaning, execution conditions, results, and interpretation. A deliverable that lets the next person compare under the same conditions makes experiments without improvement useful to subsequent decisions too.

Connecting tools

Combine tools for changing code with tools for reading metrics

The GPT-5 developer announcement addresses code repairs and work using tools. Custom tools allow plaintext input instead of JSON, with grammar constraints supplied by developers. This adds a format option for passing material such as lengthy code to a tool. Easier input handling remains distinct from correct experiment design.

The August 9 Looker MCP Server announcement describes AI querying a model in which business metrics and calculations are defined. Looker generates SQL according to those definitions. The post describes use through Gemini CLI, Claude Desktop, and Cursor, providing a path from development environments to data with shared definitions.

Combining these tools suggests an application that investigates a metric, proposes possible causes, and changes aggregation or forecasting code. Combining available features, however, is not a measurement of improved business results. The handoffs need to preserve what was inferred from which data and which experiment will test it.

Separate execution from performance

MLE-bench measures valid submissions and good predictions separately

The first MLE-bench paper evaluates data preparation, training, experimentation, and prediction submissions across 75 Kaggle competitions. Its main setup allows at most 24 hours per competition and one A10 GPU. Averaged over repeated runs, o1-preview with AIDE produced valid submissions in 82.8% of attempts and medal-level performance in 16.9%. These are results for a 2024 configuration, not GPT-5’s performance on machine-learning experiments.

The submission-validation tool returns whether the format is valid, without a prediction-performance score. This illustrates the verification needed between producing runnable code and producing a good result. The interpretation here is to keep producing predictions in the correct format separate from improving the comparison metric.

Many tasks reconstruct training and test splits and compare with historical competition leaderboards. They do not measure the rate of successful business deployments. Evidence of strong competition performance needs to be read with its evaluation target, rather than transferred unchanged to other data or conditions of use.

Read the research objective

RE-Bench tasks require more than faster execution

The first RE-Bench paper evaluates seven machine-learning research and engineering tasks. Optimize LLM Foundry, for example, asks for less runtime without changing a training script’s behavior. Omitting required work to reduce time would miss the objective. The paper also reports loopholes that looked good to automated scoring but violated task rules under human inspection.

The principle worth carrying into experiment support is to define both the metric to improve and the conditions that must remain intact. A forecasting experiment that excludes required periods or uses information unavailable when predictions would be made is no longer solving the same task. Claims of a better score need to be interpreted against those conditions.

RE-Bench does not treat short tasks with clear objectives as covering months-long research in full. Choosing a research question or evaluation metric differs from improving a supplied task. Fast experimental iteration should not be evaluated as an ability to determine research objectives correctly by itself.

Preserve data meaning

Align periods and populations even when both tables say “revenue”

Sharing metric definitions through a system such as Looker offers a way to reduce different people creating different aggregation formulas. Access to a defined metric does not make the compared populations or explanations of differences identical. Data retrieval needs to remain connected to the user’s intended decision.

Consider a hypothetical comparison of store revenue. Order dates versus recognized-revenue dates, treatment of returns, and included stores can change conclusions even when tables have the same label. Recording metric definitions, periods, and filters lets a reviewer check whether a code change still compares the same question.

Separate observed changes from causal explanations too. An increase in revenue does not by itself establish that a campaign caused it. Seasonality or the composition of stores may offer other explanations. If the agent proposes causes, retain a distinction between observed differences and untested hypotheses.

Compare forecasts fairly

Restore the conditions available when a forecast would be made

For the opening forecasting example, first decide the data cutoff and future period to predict. A simple baseline using sales on the same weekday of the previous week could be compared with the candidate on identical periods. Do not give only the candidate actual future sales that would be unknown at prediction time. This is a proposed comparison, not an account of a company’s deployment.

Retain missing-data treatment, features, training and validation periods, metrics, and a compute-time limit with the results. A comparison using mean absolute error would calculate the magnitude of prediction errors on the same population and period. Even if the average improves, investigate deterioration on busy days or in particular stores. For ordering stock, the average alone may not settle adoption.

Recorded sales of a sold-out product may not equal the number of people who wanted it. Without explaining the difference between observed sales and the demand to be estimated, a good score can still diverge from the intended decision. Human examination of what the data measures remains necessary alongside the AI’s ability to process it.

Handing over results

An experiment without improvement can still inform the next step

The experimental deliverable should connect code version, data-retrieval conditions, dependencies, model settings, compute, and results. Retain randomization settings where used, while recognizing that a seed alone does not guarantee identical results. Check rerun differences if environmental or execution variation remains. The record should let another person read the result and continue necessary validation.

A candidate worse than the baseline can still help decide whether to try the same idea again if its conditions remain available. Explain where performance deteriorated and whether more time or resources might be worth testing. Distinguish a failed run from a correctly executed experiment that did not improve. Removing the latter and retaining only good outcomes hides the experimental picture.

Adoption also considers cost, update frequency, consequences for populations that worsened, and further validation. These are proposed decision criteria, not a claim that the sources establish successful automatic adoption. A starting design can delegate candidate creation and comparison records while assembling evidence for people to verify the conclusion.

A deliverable for decisions

Connect code, data, and analysis to the same question

Development tools and data connections increase the options for combining previously separate work. Benchmarks establish execution and improvements within supplied conditions. Applying them to business or research requires data meaning and comparison conditions that fit the objective.

The value of experiment support includes the ability to question results, compare under the same conditions, and retain reasons for declining adoption. Connecting code, data, and analysis in one record turns an agent’s speed into decisions a person can verify.