Anthropic’s 64%: choosing a next step where humans went off course

Anthropic reports that Mythos Preview’s next-step suggestions were judged better than a human researcher’s choices 64% of the time. The researchers had deliberately selected moments where the human went off course. In a separate sample where the human’s next move was already strong, model suggestions were preferred only about 20% of the time. The footnote does not give individual model scores for that second sample. The two percentages therefore need their sample descriptions attached: neither is an overall score for who is better at research. Anthropic’s test and control.

The original test used 129 moments from real Claude Code research sessions recorded between January and March 2026. The work included investigating why a training run kept crashing or why a model performed poorly on a benchmark. Anthropic found points where a researcher chose a direction that led the session sideways before it eventually recovered. The evaluation returned to the point just before that choice:

1. The work already done supplied the context. The human’s next move was preserved in the session record.
2. A candidate model received only that earlier context and proposed what to do next. It could not see the later course of the session.
3. A separate Claude received the eventual session outcome and judged the candidate’s suggestion against the human’s recorded choice. The reported procedure.

The information boundary matters. The proposing model had to choose from the past; the judge could use hindsight. A preference for the model’s suggestion does not establish what would have happened if that alternative had been carried out. The 64% counts judgments of next-step suggestions, rather than measured research gains from executing both routes.

Editorial explanation: arrows show information available to each model. Blue bars count model suggestions judged better than the human choice within each selected sample. The 64% belongs to Mythos Preview; the approximately 20% control has no per-model breakdown. Gray remainder is not labelled as human wins.

Anthropic introduced the 127 already-strong human decisions as a check on judge bias. This second group also makes the boundary of the headline result easier to understand. Beating a human choice selected because it was a detour is a different comparison from improving on an already-strong choice. Neither group represents the ordinary mix of research decisions, so averaging their scores would not produce a general human-versus-AI win rate. Nor can the difference between them be attributed entirely to sample selection when the control lacks per-model results. Selection rationale and control.

There is a useful model comparison inside the original sample. Opus 4.5, Anthropic’s best model in November 2025, scored 51%; its April 2026 model, Mythos Preview, scored 64% on the same evaluation. Those dates identify the compared model generations; the recorded sessions came from January–March 2026. The increase is positive evidence of better judged next-step suggestions on this particular yardstick. It answers a narrower question than whether AI generally directs research better than humans. The 51%–64% comparison tracks models on the original sample; the 64%–approximately 20% contrast concerns differently selected samples. The comparison across models.

That distinction connects to three responsibilities Anthropic separates in its account of research work. Carrying out a specified experiment means following an approach already chosen. Designing an investigation means deciding which experiments to try within a supplied question. Setting research direction means choosing which question is worth pursuing in the first place.

Its weak-to-strong supervision project shows agents taking on the second responsibility. Humans supplied the question—could a weaker model reliably supervise a stronger one?—and created the scoring rubric. Within those bounds, agents proposed hypotheses, designed every experiment, tested them, shared findings with parallel agents and iterated. This is autonomy over how to investigate a problem, while the problem and the framework for judging results remain human choices. Project and division of responsibility.

Anthropic reports that the agents recovered 97% of the project’s defined performance gap. The lower reference point was the weak supervisor’s performance on its own; the upper reference point was the strong model’s performance when trained on correct answers. The percentage describes how much of the distance between those performances was closed, not 97% accuracy. The result did not transfer cleanly to production-scale models. The essay’s account also does not provide an experiment-by-experiment explanation of which interventions produced the recovery. It supports the reported outcome and division of responsibility, without supplying a detailed causal reconstruction. Gap definition, result and transfer limitation.

Choosing the research goal remains a separate responsibility. Anthropic says large gaps persist in Claude’s judgment there. In this essay, it uses recursive self-improvement to describe an AI system fully autonomously designing and developing its own successor, and says it has not reached that state. Better next-step suggestions and agent-designed experiments are meaningful advances in research work. They do not establish that the system can choose its own development goals and carry successor development through autonomously. Goal-selection gap and the essay’s RSI description.

This 14 October 2026 retrospective uses frozen essay snapshots including the 18 September update; the essay’s first publication date remains unverified here. It presents Anthropic’s reported research and editorial explanations, without independent replication or audit results.