Six AI models, ten business tasks: why the cheapest one won

For one night I had six AI models compete on work that every company knows. This is the full report: the setup, the rules, how each round went, the results on speed and on cost, what went wrong, and what we are doing with it in the Virtual Company Builder.

The question behind this

Every week a new AI model sits at the top of a leaderboard. Those scores come from a lab: carefully chosen tasks, polished prompts, ideal conditions. That is valuable work, but it does not answer the question I face every day.

I build autonomous organisations: companies where agents do the work. Invoices, email, bookings, reports, code. For each of those processes, an agent has to pick a model. So the question is not "which model is the smartest", but: which model does this particular work best right now, under the conditions my agents actually work in?

And those conditions are never ideal. An agent gets a short assignment and a few examples. It has a clock running. And when something goes wrong, it usually hears only that it is wrong, not why. Just like a customer who tells you the invoice is not right.

So I built a tournament that imitates those conditions. I call it Arena Routing: automatic tournaments that decide which kind of work goes to which model.

The contestants

Six models, deliberately mixed:

• Claude Opus 5.5 and Claude Haiku 4.5 from Anthropic: the premium model and its cheap sibling.
• GPT-6 Astra and GPT-6 Luna from OpenAI, via Codex: the frontier model and the fast, cheap one.
• Gemini 3 Flash from Google: Google's low-cost model.
• Qwen3.5 35B, an open model running on our own server, on the CPU, without a graphics card.

Each model worked with its own tooling, the way you would use it in practice. Claude through Claude Code, GPT-6 through Codex, Gemini through the Gemini CLI. Qwen got a simple harness of its own: write a program, run the examples, improve, submit.

The ten tasks

No puzzles, but work that shows up in any office, ranging from one minute to an hour of work for a good developer:

1. Till total: add up a day's transactions, returns included, to the cent.
2. VAT summary of an invoice: per rate the tax base and the VAT, rounded per rate and not per line.
3. Top words in customer feedback: which words appear most often, with capitals, punctuation and ties handled.
4. IBAN check: supplier bank account numbers typed by humans, with spaces and lowercase letters.
5. Working days between two dates: with public holidays, dates in random order and impossible dates.
6. Revenue per hour of a coffee bar: a real till export in CSV, with quotes inside quotes.
7. Warehouse route with access passes: the shortest route for an order picker who must first fetch the right pass.
8. Price formula engine: calculating with formulas from a quoting tool, including the right error messages.
9. Meeting room booking system: smallest room that fits, a waiting list, and a waiting list that is replayed after every cancellation.
10. Budget spreadsheet: formulas, ranges, exact fractions, error values that propagate and circular references.

Every task came with public examples and a set of hidden tests. Those hidden tests existed only in the referee's memory, generated from a fixed seed. No model could see them.

Before the tournament I had the tasks checked blind: a separate agent solved them using only the text, and compared its answers with mine on all hidden tests. That found one real error. In the warehouse task the letter E was both the packing station and a door. That sentence was rewritten before the contest began. A task that is itself wrong measures nothing.

The rules

• Every model got the same short prompt, the same task and the same examples, in its own working folder, at the same moment.
• With ./check a model ran its program on the examples. With ./submit it handed in.
• The referee answered only approved, or rejected with the number of hidden tests passed. Which test failed, nobody was told.
• At most eight submissions per round, and a time limit per round that grew with the task.
• Every correct program scored by order of speed: six points for the fastest, then five, four, three, two, one. Wrong, nothing or too late: zero.
• A second referee, itself an AI model, read every approved program afterwards looking for cheating, such as hard-coded answers. Nobody was disqualified.

The working folders were deliberately placed outside our own project folder. Otherwise Claude Code would have read in the instructions of our project, and that would have been unfair.

What it cost

Every token was counted. For Claude I took the amount Claude Code itself reports. For the other models I multiplied tokens by the published price, converted to euros at the ECB rate. For Qwen, on our own server, I calculated the way you do for your own machine: ninety euros a month, divided by the seconds in a month, times the seconds the run lasted. These are approximations, but approximations made the same way for every model.

How it went

The first rounds were a sprint. In the till total, five correct programs were in within half a minute. GPT-6 Luna was fastest, Claude Opus one second behind. The difference was not in speed but in the bill: the most expensive correct program cost more than two hundred times the cheapest.

In the VAT round and the word count, Claude Opus won: the first time by a fraction of a second, the second time by two ticks. From the IBAN round on, GPT-6 Luna took over and never gave up the lead. Opus stayed close, usually a few ticks later, and always correct.

The most interesting things happened with the models that did not make it.

Claude Haiku submitted eight times in the working-days round and got the same answer eight times: fifteen out of sixteen. Afterwards I rebuilt the tests and worked out which one it missed. The task says a date in the wrong format is invalid. Haiku treated 2026-1-5 as the fifth of January. A person with a tester next to them would have found that in a minute. With only "fifteen out of sixteen" as feedback, Haiku did not.

The same pattern showed up in the price formula round (a negative power that should give a different error message) and in the final (a formula that cannot be parsed, and the order in which errors take precedence). Each time one or two edge cases that were in the task, and each time the difference between all the points and none.

That is exactly why you run this kind of test. In a lab a model gets detailed feedback or a carefully tuned prompt. In an organisation an agent reads the task once and has to manage with that.

The final, the spreadsheet engine, was the hour-long job. GPT-6 Luna delivered a correct program after a little over half a minute, for half a cent. Claude Opus and GPT-6 Astra followed within ten seconds, Gemini after more than two minutes. Haiku and Qwen did not make it.

Those four programs barely resemble each other. The longest is over twice as long as the shortest. They all pass the same tests.

The result on speed

GPT-6 Luna won, with all ten tasks correct first time. Claude Opus came second, also ten out of ten, always just a little later. GPT-6 Astra was third, Gemini fourth: those two also solved everything, only more slowly. Claude Haiku missed three tasks, each time on an edge case. Qwen solved seven tasks correctly, but missed the time limit three times.

And then the bill

Here it gets interesting. The fastest model was also the cheapest: GPT-6 Luna solved the whole tournament for a few cents. The premium models cost tens of times more per solution, for exactly the same result. The most expensive per solution was Claude Haiku, the "cheap" model: its rejected attempts cost money too.

All models together cost about six euros for this tournament. The AI referee that reviewed the programs cost more by itself than the winner's entire tournament.

What if you count on cost?

The points went to speed. But not every process is in a hurry. A report that runs overnight can take a minute if it costs a tenth. So I recalculated the same tournament: the same correct programs, but now the cheapest gets six points.

Then the standings tip. Luna stays first, but Gemini becomes second and our own Qwen jumps to third. Opus drops to the middle, and Astra, fast and always correct, ends up last.

Add both together and Luna is well ahead, followed by Opus and Gemini. So "the best model" does not exist. There is only the best model for what a process needs at that moment.

What we do with it

In the Virtual Company Builder, each process states what matters. Is a customer waiting? Then speed counts. Is a nightly report running? Then cost counts. Everything else gets the best balance. So the tournament does not write one routing table, but one per priority. The same kind of task can go to a different model depending on the moment.

For the balanced table I used a simple rule: the cheapest model that was correct, within twice the winning time. This time that gave almost the same answer everywhere, because one budget model won almost everywhere. That is a result of this night, not a law. Next month, with a new model or a new price, the table may look very different.

How exactly to weigh speed against cost is a choice I am still working out. The data to make that choice is now there.

The agents organise their own tournaments

The best part: nobody has to start these tournaments. In the Virtual Company Builder the agents organise them themselves. When an agent sees a new model appear, a price change or its own results declining, it takes tasks from its own recent work, sets up an arena like the one this night, lets the models compete, reads the new routing table and switches over.

That means we never pick a model once and stay stuck with it for years. No choice based on gut feeling, no choice based on a lab leaderboard, no dependence on a single supplier. The organisation measures for itself, on its own work, under its own conditions, and adapts. After launch it keeps getting better and cheaper that way, without a person having to sit there.

Limitations, stated honestly

• One night, one round per task. Speed fluctuates per attempt. A second tournament may give a different winner per round. That is why this has to be repeated, and that is exactly the idea.
• Qwen was at a disadvantage. During the tournament the video editing was also running on the same server. Qwen runs on the CPU, and as a result dropped from five to seven tokens per second to two or three, sometimes less. Its times are therefore too pessimistic, and in three rounds it hit the limit. That should not have happened. The working instructions now say that nothing heavy runs alongside a tournament.
• Python. All tasks were solved in Python. In another language the result may differ.
• Familiar kinds of work. Tasks like an IBAN check resemble what models saw during training. That is exactly why the next step is to take tasks from our own processes.
• Costs are approximations, based on published prices and a fixed exchange rate. The ratios are more reliable than the exact amounts.
• Tooling counts. Each model worked with its own CLI. What you measure is therefore the model plus its tooling: just as you would deploy it in practice.

How it is built

The whole tournament is a reusable setup. A task set is one file: per task the text, examples, a reference solution and a generator for hidden tests. A new tournament is a new set plus one command. Everything is kept: every line of output of every model, every submitted version, every result, the recordings of the dashboard. A new weighting, a new ranking or a new video is therefore possible without replaying the rounds.

In closing

This tournament does not say which model is the best. It says which model did the best work this night, on these tasks, under these conditions. And that is exactly the question an autonomous organisation has to keep asking itself.

The video of the whole contest shows all six models side by side, live, with the code, the rejected attempts, the cost per round and the final standings on speed and on cost: https://youtu.be/P76ZpwlJHtY

Floris Knol is a software engineer. He builds AI agents that automate business processes and trains teams to work with them.

All pieces