AI model routing is a fix for AI over spending and that’s a big issue for big AI labs

A new spending discipline is taking hold inside corporate America, as Chief Financial Officers and boards start cracking down on inefficient Artificial Intelligence (AI) spending. The change has the potential to reshape the AI trade.

For the past two years, the playbook has been to default to the most powerful AI model and direct all queries through it, regardless of complexity. Now, with AI bills running far ahead of budgets, companies are starting to ask whether every task actually needs the top or frontier models. Two leaders at the centre of the AI buildout told the WSJ this week that a solution is emerging: Model Routing.

Routing is a tool that matches the right job to the right model – a trend known as Compound AI - sending hard problems to the expensive frontier models and easy ones to cheaper, faster alternatives which helps reduce costs.

Scott Wu, CEO of Cognition, which makes the coding agent Devin which was a huge hit a while ago, said the gains on routine work are enormous. For a lot of the boilerplate work, he said, companies can get five to 10 times better cost efficiency using models that are still good enough for the task.

Most companies today aren’t routing at all. Glean CEO Arvind Jain has estimated that roughly 95% of enterprise AI usage is still running on the most expensive frontier models, even for tasks that cheaper alternatives could easily handle. Wu gave the example of asking a model to name the third US president. Each one, no matter how expensive, will tell you it was Thomas Jefferson.

The pressure behind the shift is a cost curve that has surprised even the biggest tech companies. Jeetu Patel, Chief Product Officer at Cisco, laid out the math. At roughly $200 of token usage per employee per week, that’s about $10,000 a year per person. With 90,000 employees, a company is looking at $900 million annually. Tokens are blocks of data that models use to generate information. Usage is billed by the number of tokens processed.

Patel said Cisco came in well over its own budget and has had to adjust, with 30,000 engineers now building products written largely with AI. Cisco has reallocated resources, prioritising tokens over other spending, and AI companies recognise the anxiety – and ignore it at their peril.

Cognition announced what it calls an AI productivity guarantee. If Devin delivers less engineering value than a customer is paying for, Cognition will fund usage up to $10 million until it’s up to par. Wu framed it as a way to cut through the noise on a metric that’s dogged the industry: Return on Investment (ROI).

Rather than measuring activity like tokens consumed or lines of code, Wu said, Cognition estimates the number of human engineering hours its agent actually saves and backs that estimate with a refund. You can spend billions of tokens and be doing nothing with it, he said. Companies should be striving for output, not activity.

If companies begin steering easy, high-volume work to cheaper open-source models out of China or elsewhere, then OpenAI and Anthropic stop getting paid for every task. They only get the more complex jobs. Both companies have built their businesses, and the IPO expectations around them, on the assumption of enormous demand at premium prices.

Patel doesn’t think that sinks the frontier labs, and says that cutting-edge technology will remain valuable. But he sees the pricing model shifting. The labs will have to get more efficient with how the models are used rather than simply charging more, which Patel predicts will lead to a concerted industry effort.

The question had been whether companies would keep spending as their AI bills climbed – as we saw in Ubers highly publicised case. It now appears that many will simply find a way to spend smartly. Pricing power is shifting from the companies selling premium AI toward the companies buying it.

The frontier labs will still command a premium for the hardest work. But how much of the market is the other stuff? The answer could go a long way to determining the valuations of the leading AI companies.

---