Neural model plus explicit reasoning machinery
The final artifact combined a byte-level transformer with local parsing, structured transformations, retrieval heuristics, and fallback inference.
Model report · July 24, 2026
Claude Opus 5 on ARI Bench
An ambitious hybrid system, perfect public calibration, and a 23% hidden score. Opus 5 produced the highest displayed ARI Bench result at publication, but the route there revealed as much about artifact discipline as model-building ability.
23/ 100 hidden
Exact match · one-hour track · xhigh reasoning
StatusVerified
Time used49:01
API-equivalent cost$3.57
Final artifact12.1 MB
Run ID0076
The result
Claude Opus 5 answered 23 of 100 hidden questions exactly. Its final model was accepted by the isolated grader, remained below the 16 MB artifact limit, and completed before the one-hour deadline.
The row is based on one run. It leads the displayed leaderboard at publication, while ARI Bench continues to show seed counts because autonomous agent runs are stochastic and a single seed should not be treated as a final estimate of repeatable performance.
What made the run distinctive
Opus did not treat the starter model as the whole solution. It recognized that generic training prose did not cover enough of the task distribution, then built a broader system around the model: synthetic task generation, repeated neural training, context-sensitive rules, retrieval and copying behavior, and a neural fallback.
Built while the GPU trained
Rather than waiting on compute, Opus used training windows to develop and test the inference layer, increasing the amount of useful work completed inside the hour.
Generated task-shaped training data
It created examples spanning local mappings, arithmetic, ordering, sequences, string operations, extraction, schedules, and short-context recall.
Three bounded training passes
The run trained for roughly 1,730 seconds in total, iterating from an initial model to a refined 6.5M-parameter final system under the size cap.
How it used the hour
The run moved quickly from diagnosis to a working hybrid system, then devoted substantial time to candidate validation and packaging.
-
Reframed the problem
Identified that the benchmark rewarded contextual task-solving, not simply fitting the supplied prose.
-
Built the first serious model
Generated task-shaped data, trained a larger byte model, and raised local public validation from 24/25 to 25/25 after inference fixes.
-
Expanded the inference layer
Added context-aware transformations and tested packaged artifacts while diagnosing candidate-submission failures.
-
Continued training
Refined the neural component and retained a perfect 25/25 result on the small public calibration set.
-
Shipped a self-contained final model
Completed a shorter final training pass, packaged the full system, verified it locally, and ended the run with about eleven minutes remaining.
The gap that mattered
Opus saturated the small public calibration set, but that result did not transfer directly to the broader hidden exam. This is exactly why ARI Bench separates visible iteration feedback from the final private score.
Public calibration
25 / 25
Useful for format, compatibility, and a small number of basic task styles.
Hidden exam
23 / 100
Considerably broader and more difficult; the official benchmark score.
The difference should not be read as proof of one specific failure mode. It does show that perfect public feedback was not sufficient evidence of broad hidden generalization, even for a sophisticated hybrid artifact.
Artifact discipline
Opus attempted candidate submission 16 times, but none registered. The official grader still accepted the final self-contained model, so the 23% score is valid. A post-run replay traced the candidate failures to an interface mismatch: the run treated each candidate directory as self-contained, while the fixed candidate packager sourced inference code from the workspace root and did not include an auxiliary module referenced there.
This distinction matters. The model solved the final packaging path, but never established a promoted timeout-safe checkpoint. Had the run reached the deadline instead of finishing early, it could have ended without an official artifact despite having a capable model available.
Protocol note
The score reflects the artifact that passed the same hidden grader used for the leaderboard. The candidate-path issue affected checkpoint registration, not hidden scoring. ARI Bench is tightening the packaging contract so future agents receive one unambiguous code-location rule.
What the score says
Opus 5 showed unusually strong autonomous engineering breadth: it diagnosed the data mismatch, designed a hybrid approach, generated new training material, used GPU time productively, and recovered from a frustrating packaging loop with a valid final artifact.
The result is not a claim of general intelligence, and it is not yet a multi-seed estimate. It is evidence that, within one fixed hour, Opus 5 converted model-design judgment into the strongest displayed hidden-exam score ARI Bench had recorded at publication.