ScientistTwo: why five improved metrics did not earn the architecture credit

Five of six metrics favored ScientistTwo's DMC-TeCh candidate over its baseline. The candidate nevertheless failed the system's contribution check. According to the authors, the ablation critic attributed the gains mainly to EMA and label smoothing—general training controls—rather than the proposed multi-core architecture. Here, rejection describes an internal judgment about the research contribution. The DMC-TeCh case in Appendix B.

The candidate contained two kinds of changes: an architectural proposal and changes to training. Comparing the complete candidate with the baseline measured the performance of that package. It did not, by itself, tell which change deserved credit. The narrower architectural claim needed evidence connecting the gains to the architecture. The critic's reported attribution prevented the package's better numbers from automatically becoming support for that claim.

“Primarily” matters here. The passage reports where the critic placed the main credit; it supplies neither numerical effects for individual components nor a finding that the architecture contributed nothing. The improved measurements and the rejected contribution can coexist because they answer different questions.

Editorial explanation of the reported decision. Five green circles count improved metrics. Solid and dashed attribution branches distinguish credited and uncredited main-cause explanations; the geometry does not estimate effect sizes or numerical shares.

ScientistTwo's component-ablation procedure gives this distinction an operational role. A planner specifies experiments for individual components, and a coding stage modifies the validated implementation to execute them. The critic can then request changes to the hypothesis and code. A revised candidate is compared with the previous one: the retained state is updated only if the system's result-comparison agent prefers the new results, after which ablations run again. Criticism can therefore change the method being proposed, with another check on the revised explanation. Component experiments and retention rules, §3.4.

The manuscript stage adds another route back to evidence. ScholarPeer reviews the draft's strengths and weaknesses, poses targeted questions and gives a score. When that score falls below the workflow's threshold, a rebuttal planner turns the concerns into supplementary experimental tasks. A coding stage implements and executes those tasks; the manuscript is then revised using the findings, including changes to claims, tables and figures. Review and experimental response, §3.5.

Meta-review can reach further back, requesting a revised hypothesis and implementation. If the new results are preferred, the workflow returns through ablation, drafting and simulated review. If the refinement fails to improve on the previous results, it is discarded and the previous outputs are retained. An internal Accept ends this process and exports the manuscript and code. These are described routes for revising research; they do not establish that every criticism was resolved successfully. Meta-review and return to research, §3.6.

The review-round experiment measures a different outcome: how AI evaluators judge the resulting manuscripts. ScholarPeer's feedback already shapes the drafts, so the authors call it an in-distribution evaluator. Stanford Agentic Reviewer was unseen during development. That makes Stanford's response useful for checking whether improvements are recognized beyond the reviewer used in production, while remaining an AI assessment. Evaluator roles, §4.

On 49 target problems derived from ICML 2026 Spotlight papers, Table 5 reports the following conditions and AI acceptance rates. This experiment varies review and rebuttal settings; the DMC-TeCh example above is a separate contribution decision in Appendix B.

Sample, complete conditions and Table 5.

Both evaluators give higher acceptance rates after the first round, including the evaluator unseen during development. After the second, ScholarPeer rises again while Stanford falls, although Stanford remains above its zero-round rate. The results support that descriptive pattern. The three settings alone do not establish statistical significance, identify the best number of rounds or diagnose reward hacking.

The first contrast also bundles interventions: moving from zero to one round introduces both review and the Rebuttal Agent. The workflow can add experiments and revise claims, and meta-review can change the method. Table 5 does not apportion the acceptance-rate changes among these actions or changes in presentation. It also does not explain the second-round divergence. Its percentages describe manuscript judgments; they cannot supply component attribution for a proposed mechanism.

The word “acceptance” needs its object kept visible. Accepted conference papers provided the input problems and codebases. The workflow's internal Accept is a simulated meta-review decision, and Table 5 records AI evaluators' judgments of generated manuscripts. Actual conference decisions on those generated manuscripts remain unverified in the presented materials. Input papers and output evaluations, Table 3 and Appendix A.1.

Human assessment adds evidence on a separate sample. The authors organized nine experienced reviewers to evaluate 33 papers generated from NeurIPS-derived problems. They used standalone ratings across six dimensions and paired comparisons with accepted human-authored papers. Both scales run from 1 to 5, but their meanings differ: a standalone value above 3 indicates positive endorsement; comparative 3 means parity, and comparative values above 3 favor ScientistTwo. Human procedure and Table 10.

The overall standalone mean was 3.7, giving a favorable judgment under that scale. The overall comparative mean was 3.0, at the parity point. That overall value sits alongside different judgments within the manuscript: baseline and benchmark experiments averaged 3.5 in comparison, while method soundness and algorithmic rigor averaged 2.9. The reported means favor the benchmark work more than the method's rigor. An overall mean at parity does not establish statistical equivalence or make the dimensions interchangeable. These human means also cannot be combined with the 49-task AI percentages into one quality score. Reviewer independence and blinding remain unverified in the presented human-evaluation procedure.

The artifact audit addresses another set of properties. For the full system's 49 ICML-derived outputs, Table 7 reports 49/49 passes for score verification, which compares reported scores with repository re-execution; 0/49 task-rule violations; zero hallucinated bibliography entries among 1,814 references; and 49/49 passes for agreement between the manuscript's method description and its code. These are the authors' reported checks on the research artifacts. Audit definitions and complete Table 7.

Each check has a concrete use, but its scope matters. A real bibliography entry does not establish that it supports the argument attached to it. Agreement between a method description and executable code does not, by itself, establish which component caused a gain. The audit therefore complements the attribution test illustrated by DMC-TeCh; it cannot replace it.

DMC-TeCh makes the distinction consequential. The reported numbers supported improvement of the complete candidate, while the critic withheld contribution credit from its architectural explanation. The later review rates, human ratings and artifact checks assess other parts of the output. The workflow's capacity to return to experiments matters because feedback can change the research claim that survives, even after the measurements have improved.

This analysis, dated 14 October 2026, discusses arXiv v1, submitted on 17 September 2026. That is the submission date. The article presents the authors' reported research and editorial explanations, without independent rerun experiments or replication results.