Introduction

I wanted to talk a little bit today about small models and routing.

So background on our company, we're an open source language modeling company. We made a model family called Glina. Gets about 2 million downloads a month on Hugging Face. It's really cool, you should check it out.

And we recently launched a product called Pioneer. So Pioneer is a small language model router, and that's what we're going to talk about today, which is routing.

The Token-Maxing Trend and Why It Hurts

First of all, there's this industry trend of token maxing, right? So this idea that the correct way to be productive as a developer is to just be very, very kind of token inefficient

and just use as many tokens as you can. So like famous examples of this is like the founder of OpenCore spent one and a half million dollars

in a month on tokens, which is nice for him, I guess. Pretty expensive.

So we're looking at ways that we can now like make them, I guess, cheaper.

Why Cost Suddenly Matters More Than Accuracy

So historically, when we've asked customers in the past what they've cared about with language models, models, they've always said accuracy is number one, followed by latency, followed by cost.

And there's been a very sudden change in which that's now changed to cost being the number one focus. And I think the answer to why that is the case is because language models are now super good enough.

They do the thing that we're asking them to do, and now we actually need to make them a little bit more cost -efficient. A great example of this is that we do a lot of analytics on top of Cloud Code.

I found that when we were reviewing a pull request, So this was me reviewing like a 20 file pull request. It cost $41, which is super expensive. It's like the cost of breakfast to review a pull request.

It's pretty insane. Expensive breakfast, but it's the base.

Ways to Cut LLM Spend

So there are ways to save money, right? So like, for example, caching is a huge one, or you can batch prompts together, or you can try and trim or like compress prompts.

That's another option to save money.

Model Routing: Using the Right Model for the Job

But what we're going to talk about today is model routing. So, the idea of model routing is basically that instead of you using Opus 4 .8 every single time for every single prompt, you try to use the correct model when you need it. You basically only use the really large models when the complexity of the task makes sense to do so.

How Routing Works in Practice

And the way that you do this is you basically feed lots of different prompts into lots of different language models and you record, like, basically a pass or fail. So, was this language model good enough to solve the problem?

And if it was, then in the future what we do is when a new query comes along, we look at similar embeddings, there's a little bit of complexity there because we're not looking at semantic embeddings, we're looking at embeddings to do with associated complexity. And then we can kind of match that to a correct model.

Routing by Task Complexity (Not Just Semantics)

So a quick example of what that means is that two prompts, for example, build a front -end in JavaScript and build a front -end in C++ are semantically very similar, but it's really really hard to build a front -end in C++.

So you need to train a custom embedding model to rank by complexity.

The Counterintuitive Economics of Routing

The interesting thing about routing is that there's a very unintuitive

cost saving.

So if you sacrifice a 4 % accuracy decrease on an Opus model versus a bunch of routed models, you might expect a 4 % cost saving, but actually it's closer to a 90 % cost saving.

So if you're willing to take a very small margin of like decrease in accuracy you can get a huge benefit for cost saving.

Where This Is Headed: Fine-Tuned Model Suites

So roughly like what the future looks like now is we're trying to build fine -tuned models that we're routing to instead of just running to a bunch of generic base models we want to actually route to models that have been fine -tuned for specific tasks.

So the dream situation here and I think you know enterprise has already started to get there there now, is that maybe you have a suite of 10 or 15 models, all fine -tuned for specific small parts of your thing that your agent or whatever you're building is solving, and then just a router to route efficiently between them.

Routing Across Specialized Models

Next one is it's really nice if you can pass the cache between those models you're routing to.

Sharing Cache Between Routed Models

So the way that Anthropic works, for example, is that you get a 70 % discount on tokens if they have been cached from previous outputs. inputs.

You lose all of that cache if you route between models. Ideally, we want to pass the cache between models.

And then, yeah, I guess the theme here is try not to use frontier models when you don't need them every time.

Pioneer: A Practical Router Example

We also have a model router, pioneer .ai. So this is an example in which we've sent in a query, and it's picked Claude's Sonnet 4 .6 and saved the user 68 % of their costs.

And yeah, this product came out like five weeks ago, just Just hit 1 ,000 customers. Go check it out. Pioneer .ai.

Conclusion

That's about it. Thanks, guys.