Kojo Osei

 undo All posts
Subscribe

Not all inference is text generation

• 2 min read

I've written before about how the inference market can be segmented by intelligence saturation. In that post, I talked about how different tasks need different model capabilities. As a result, distinct markets emerge along the Pareto frontier of cost and intelligence.

I've been thinking about how the inference market might also segment along the shape of intelligence a task requires.

Today, we shoehorn almost every intelligence task into an autoregressive LLM. But not every task needs open-ended generation. Some tasks are classification problems where all you want is a label or score. Others are ranking problems where all you need is an ordered list.

As more specialized models emerge, I expect the inference market to fragment not just by how much intelligence a task needs, but by what kind of output it requires.

Jev is a good early example of this. It's a decision model that constrains the output of a query. You give it a prompt and a predefined set of possible outputs, and it gives you back a decision: a selection from your list, a score on your scale, or the probability that an answer is yes.

This is a new shape of intelligence that has been quite useful for me. It turns out a good bit of what I use LLMs for amounts to a series of if statements. For example, I have a series of scripts I run periodically that do a bunch of sorting tasks like sorting my highlights in Obsidian automatically with pre-defined topic tags.

Before Jev, I was using a combination of Gemini and Opus for this. It worked fine, but in hindsight it was a pretty odd way to solve a classification problem. I was using general-purpose generative models to do something that didn't require generation at all. It was slow and relatively expensive.

Jev, on the other hand, is wicked fast. And cheap, very cheap. It's not necessarily the smartest model, but for these tasks it's a much better fit. It's made clear that there's another dimension of model selection that the traditional cost-versus-intelligence Pareto frontier doesn't fully capture.

This makes me wonder how much of today's inference demand is being awkwardly forced through the autoregressive paradigm simply because that's the interface we have available.

If we look at OpenRouter's breakdown of high-volume tasks, we can start to imagine a roadmap for this fragmentation. How many of these tasks actually require open-ended generation?