Notes ·
Terminal-Bench-Science Is Testing Agents on Actual Scientific Work
Joshua MorrisTerminal-Bench-Science 0.1 launched today, bringing the approach behind Terminal-Bench into scientific research.
I like the premise immediately.
Instead of asking AI systems textbook questions and calling the result scientific capability, Terminal-Bench-Science evaluates agents on workflows contributed by practicing researchers.
The first release contains 70 tasks across the life, physical, Earth, mathematical, and engineering sciences.
Those tasks include things like scientific data analysis, statistical inference, simulation, optimization, theorem proving, image reconstruction, signal processing, inverse problems, sensor calibration, model fitting, classification, and scientific machine learning.
The agent has to actually do the work.
And the output is graded through concrete artifacts such as analyses, simulations, proofs, code, and data products using reproducible, task-specific tests.
That seems much more useful than asking whether a model can answer a scientific question that may already exist somewhere in its training data.
Scientists are setting the bar
The part I appreciate most is that the benchmark is deliberately putting scientists in charge of deciding what should be measured.
The project received 920 task proposals.
Of those, 464 were approved for implementation.
386 became pull requests.
Only 70 made it into the first release.
That is a pretty aggressive filter.
Tasks have to be scientifically grounded, objectively verifiable, difficult enough to challenge current frontier systems, and realistic enough that researchers actually care about the workflow.
Domain reviewers check the science.
Technical reviewers inspect the construction and verification.
Then there is another final quality review before a task is merged.
The project says 376 contributors across 22 countries participated through proposals, reviews, or pull requests.
That gives me much more confidence in the benchmark than something assembled entirely by a model company trying to demonstrate that its newest model is good at science.
The people doing the science should have a large say in defining what useful scientific AI actually means.
The models are still bad enough for this to be interesting
The results are also refreshing.
The strongest system tested, Claude Opus 5 with Claude Code, resolves 30% of the tasks.
GPT-5.6 Sol with Codex reaches 22.4%.
Claude Fable 5 with Claude Code reaches 21.4%.
Everything else is below that.
That means roughly seven out of ten tasks still defeat the best system in the benchmark.
Good.
A benchmark where the frontier model scores 97% on launch is mostly measuring how quickly we need another benchmark.
Thirty percent gives researchers somewhere to go.
It also gives us a much clearer picture of the gap between models being impressive at scientific conversation and agents being dependable at scientific work.
Those are very different things.
An AI can explain an algorithm convincingly and still fail to run the analysis correctly.
It can describe an experiment and still make a subtle mistake processing the data.
It can produce plausible-looking code that does not reproduce the expected scientific result.
That is why evaluating the artifact matters.
I also like that they expect the benchmark to decay
Terminal-Bench-Science is being treated as a continuous benchmark rather than a paper that gets published once and slowly becomes irrelevant.
Tasks will be added.
Tasks that become saturated can be retired.
Underspecified tasks can be fixed.
New frontier models can be evaluated against new versions.
The benchmark itself is versioned so old trials can be regraded or rerun as the test suite evolves.
That seems important because AI benchmarks have an unusually short useful life.
Once models get good enough, or once the benchmark becomes widely known enough, the number stops telling us very much.
A benchmark should probably be expected to become obsolete.
Designing that into the project from the beginning makes sense.
This is closer to the agent future I care about
I am much more interested in agents that can take difficult, tedious chunks of real work off someone's plate than I am in increasingly polished chat.
Science is a particularly good place to test that.
A useful research agent does not need to replace the scientist.
It needs to do enough of the expensive mechanical work that the scientist can spend more time deciding which questions are worth asking, interpreting what happened, challenging the result, and figuring out what to do next.
That is also a much harder capability to fake.
If the task is to produce a simulation, reconstruct an image, prove something, fit a model, or analyze a real dataset, eventually the result has to work.
The announcement describes the goal as freeing researchers to spend more time on the parts of science where human judgment matters most.
That is exactly the division of labor I would like to see.
Let the agent grind through the work.
Let the human decide whether the work means anything.
And keep building benchmarks hard enough to tell the difference.