Opus 5.5 Cleared 34.8% of Argo-Bench

The abstract does not name a winner. The score does not look like a winner’s score. arXiv:2610.02122, submitted on Oct. 1, 2026, introduces Argo-Bench: 210 data-science and analytics tasks inside a simulated New York City food-delivery company. The strongest of 14 frontier and open-weight models scores 95 or higher on 34.8% of those tasks and averages 59.5 points. Priya Anand’s Oct. 2 writeup says that strongest model is Claude Opus 5.5, citing AI Weekly. I am keeping those two sentences apart. The paper’s abstract, as posted, does not print the model name. The secondary piece does.

We already wrote the price cut in Opus 5.5 reset the API prices, and Sol halved them. We wrote the bill that does not match the sticker in Grok 4.7 kept the price list. We wrote the earlier Sonnet ship in Claude Sonnet 5 at one-third the Opus price. This is not a rate card. It is a test that grades what the agent did to a fake company, not whether the SQL looked right.

Argo-Bench’s warehouse is not a demo table

The authors on the arXiv listing are Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, and Joseph J. Ma. Anand’s piece calls them a five-person team at TextQL Labs. The abstract I can read does not include the lab name in the byline block, so the affiliation is the writeup’s, and the names are the paper’s.

The company is not a toy table. The abstract says the simulator draws on public data, peer-reviewed industry literature, and regulatory filings, and that it stands in for a New York City food-delivery platform in 2024 with 81 million orders. The economics, the fraud patterns, and the marketplace incentives are part of the build, not a caption. That world is then exported into an ERP warehouse: 235 tables, 7.5 billion rows, modeled on an Oracle E-Business Suite schema.

Two design choices matter more than the row count. First, the simulator’s ground truth is withheld from the warehouse the agent sees. The agent has to reconstruct facts by navigating the warehouse before it acts. Second, every task has an executable reference solution that demonstrates the task can be solved using only that warehouse. The benchmark is not asking the model to know a secret the data does not contain. It is asking the model to find the fact in 235 tables and then do something with it.

Anand adds a paper-shape detail the abstract page does not: 41 pages, 4 figures, 18 tables, under the title “Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows.” She also says the full 14-model results table was not reproduced in the indexed coverage available on Oct. 2. If you want the other 13 scores, the paper is the place. This article will not invent them.

The grade is the consequence, not the query

Established text-to-SQL benchmarks grade the query. The abstract is blunt about why that is a weak proxy: audits have found their answer keys frequently wrong, and because real enterprise warehouses are too sensitive to release, those benchmarks get built on public datasets where a business event fits in a single table. Argo-Bench’s complaint is structural. A delivery platform’s week does not fit in one table. Fraud, courier pay, and a promotion budget are spread across an ERP schema on purpose, because that is how the source systems were sold.

The agent does not stop at a SELECT. The abstract’s examples of filed actions are banning fraudulent accounts, allocating courier incentive budgets, and issuing back pay. The grader scores each action by its consequences in the simulator. Anand uses a close pair: banning a fraud ring, or approving a courier-incentive budget. Same idea. A query can be syntactically fine and still ban the wrong accounts. A query can return the right cohort and still approve a budget that the simulator treats as the wrong size. The 95-point bar is a bar on that outcome, not on the prettiness of the SQL.

That is why 34.8% is a harsher number than a leaderboard that measures exact-match queries. The abstract says the strongest model clears 95 or higher on only 34.8% of tasks. Thirty-four point eight percent of 210 is about 73 tasks, if you do the multiplication yourself. The paper does not print “73” in the abstract. I am going to cite 34.8% of 210, and note that a reader who wants a count can multiply.

The mean is the other number, and it is worse for anyone selling “the agent runs the company.” 59.5 points on a 100-point scale, averaged across the full set, is a partial-credit story. The model is not scoring zero. It is also not sitting near the 95-point line on a typical task. A mean that far below the pass bar means a lot of tasks are producing an action the simulator does not accept as solved, even when the run is not a total blank. Anand’s reading is that the gap between a correct-looking query and a correct consequence is large. The abstract supports the mechanism.

Naming Opus 5.5 is a citation, not a footnote in the abstract

Anand writes that Claude Opus 5.5 led the 14 models, averaging 59.5 and clearing the 95-point bar on 34.8% of tasks, “according to AI Weekly’s reporting on the release.” The numbers match the abstract’s “strongest model” sentence exactly. The name does not appear in the abstract text I extracted. Until the paper’s own table is what you are reading, the honest line is: the paper says the best of 14 hit those two figures, and an Oct. 2 secondary report identifies that model as Opus 5.5.

That distinction is tedious. It is also the whole point of an analysis piece about a benchmark that is one day old. Labs will quote 34.8% next to a product name by the end of the week. Some of them will be right. The abstract, on submission day, was careful enough not to make the name part of the claim you can cite from the landing page alone.

If the name holds, it still does not rescue the price conversation we already had. A model can be the least-bad agent on a 210-task warehouse test and still be the expensive one on the rate card. The September price reset was about what a token costs. Argo-Bench is about what an action does. Those are different bills. A buyer who hears “Opus won Argo-Bench” and hears “Opus is safe to let approve courier bonuses” has skipped the mean. 59.5 is not a green light. 34.8% solved is a minority of the task list.

Anand also says no pricing, licensing terms, or commercial roadmap for Argo-Bench itself have been disclosed. She describes it as an open research benchmark, not a paid product. I am saying the Oct. 2 coverage did not find a price, and the arXiv submission is a paper, not a SKU.

What you still cannot claim from the coverage

You cannot rank the other 13 models from this article, because Anand says the full table was not in the indexed secondary coverage on Oct. 2, and the abstract only describes the strongest. Anyone publishing a bar chart of 14 names this week is either reading the PDF or drawing. Ask which.

You cannot treat 7.5 billion rows as a claim that the agent scanned 7.5 billion rows. The warehouse is that large. A task can be solvable, by the paper’s own rule, from the warehouse alone, without a full scan. The reference solution exists to prove solvability, not to prove that the winning run was efficient. There is no cost-per-task figure in the abstract. The Grok piece was about a bill that drifts from the sticker. Argo-Bench, as released in the abstract, does not even give you the sticker for a solved task.

You cannot smuggle in a safety incident from the same week and call it a finding of this paper. I saw that move in the Oct. 2 writeup’s framing and I am leaving it out. The paper’s claim is about consequence grading on a simulated delivery platform. It is not a writeup of a lab’s sandbox escape. Mixing them makes the 34.8% sound like evidence for a story the authors did not run.

You also cannot lean on the text-to-SQL audits as if Argo-Bench re-scored those old sets. The abstract uses those audits as motivation: answer keys on public single-table sets have been found wrong often enough that query-match is a shaky target. That is a reason to build a different test. It is not a result from the 210 tasks.

What I would ask before anyone cites 34.8% in a pitch

I would ask whether the pitch is quoting the paper or quoting a recap that named a model. If it names Opus 5.5, I want the table, not the recap. The numbers can be right and the citation still sloppy.

I would ask what “solved” means in the sentence. On this benchmark it means a score of 95 or higher on the simulator’s consequence grade, not a query that matched a gold string. A vendor who translates 34.8% into “a third of enterprise workflows are automated” has changed the noun. The noun is 210 constructed tasks in one simulated 2024 delivery business, with fraud patterns and marketplace incentives built in, and with ground truth hidden from the warehouse view. That is a serious test. It is not your accounts-payable queue.

I would ask what the mean is doing in the same paragraph. A leaderboard that prints only the solved rate hides the 59.5. The solved rate says how often the best model cleared a high bar. The mean says the typical task, including the ones it did not clear, landed far below that bar. If you are deciding whether an agent may ban an account or issue back pay, the miss rate is the operating fact. The paper’s own examples of actions are exactly the kinds of actions a company would rather not have wrong: fraud bans, incentive budgets, back pay.

The reference-solution rule is the part I trust. Every task can be solved from the warehouse. The agent is not being failed for lacking a private document. When it fails, the failure is navigation, judgment, or an action the simulator scores as the wrong consequence. That is a cleaner failure than a benchmark whose gold SQL was wrong. The abstract spends a sentence on those wrong answer keys for a reason.

Thirty-four point eight percent, 59.5 points, 210 tasks, 81 million orders, 235 tables, 7.5 billion rows. Those are the figures the Oct. 1 submission actually puts in the abstract. The model name is one day newer, and it lives in a secondary report. Use the figures. Cite the name as a name someone else attached. Do not let a pitch deck collapse them into “Opus solves enterprise work.” It solved a minority of this test, at a mean that would not pass the test’s own bar.