Notes ·
Terminal-Bench-Science Is Testing Agents on Actual Scientific Work
Joshua MorrisTerminal-Bench-Science 0.1 launched today, bringing the approach behind Terminal-Bench into scientific research. I like the premise immediately. Instead of asking AI systems textbook questions and calling the result scientific capability, Terminal-Bench-Science evaluates agents on workflows contributed by practicing researchers.
The first release contains 70 tasks across the life, physical, Earth, mathematical, and engineering sciences. The agent has to actually do the work — data analysis, simulation, theorem proving, image reconstruction, model fitting, and more — and the output is graded through concrete artifacts using reproducible, task-specific tests. That seems much more useful than asking whether a model can answer a scientific question that may already exist somewhere in its training data.
The part I appreciate most is that scientists are setting the bar. The project received 920 task proposals, approved 464 for implementation, and turned 386 of those into pull requests. Only 70 made it into the first release. Tasks have to be scientifically grounded, objectively verifiable, difficult enough to challenge frontier systems, and realistic enough that researchers actually care. Domain reviewers check the science; technical reviewers inspect construction and verification. The project says 376 contributors across 22 countries participated. That gives me much more confidence than a benchmark assembled by a model company trying to demonstrate that its newest model is good at science.
The results are also refreshing. The strongest system tested, Claude Opus 5 with Claude Code, resolves 30% of the tasks. GPT-5.6 Sol with Codex reaches 22.4%, Claude Fable 5 with Claude Code reaches 21.4%, and everything else lands below that. Roughly seven out of ten tasks still defeat the best system. Good. A benchmark where the frontier model scores 97% on launch is mostly measuring how quickly we need another benchmark. Thirty percent gives researchers somewhere to go, and it clarifies the gap between models being impressive at scientific conversation and agents being dependable at scientific work. Those are very different things. An AI can explain an algorithm convincingly and still fail to run the analysis correctly, or produce plausible-looking code that does not reproduce the expected result. That is why evaluating the artifact matters.
I also like that they expect the benchmark to decay. Tasks will be added, saturated ones retired, underspecified ones fixed, and new models evaluated against new versions. AI benchmarks have an unusually short useful life. Designing obsolescence into the project from the beginning makes sense.
I am much more interested in agents that can take difficult, tedious chunks of real work off someone's plate than I am in increasingly polished chat, and science is a particularly good place to test that. A useful research agent does not need to replace the scientist. It needs to do enough of the expensive mechanical work that the scientist can spend more time deciding which questions are worth asking, interpreting what happened, and figuring out what to do next. Let the agent grind through the work. Let the human decide whether the work means anything. Keep building benchmarks hard enough to tell the difference.