Intelligence Saturation

I've been trying to understand how intelligence saturation affects model selection. Do certain tasks really need frontier models or can we get by with a previous generation of closed source models or open-weight models?

I've found it easiest to reason through this question bottoms up so I built my own benchmarks from my Claude & ChatGPT usage data and then looked at how various models perform on the benchmark tasks. Each model's response was graded on a set of rubrics using an LLM-as-judge approach.

Let's walk through some observations.

On my bench of tasks related to general Q&A and explanation questions, the top performing model is an open-weight model. What's more interesting, though, is that the difference in scores across models is marginal. This low variance hints at intelligence saturation for this bucket of tasks. I'm just not getting that much more squeeze from a frontier closed model over open-weight models like Qwen, Kimi, or Nemotron.

I notice similar marginal performance differences on tasks related to lifestyle, health & everyday advice.

And again, a similar dynamic for tasks related to design & product ideation.

There are of course some categories where there is much more variance. For example, Fable, from the brief period I had access, is much better on my software related queries. This discrepancy points to the fact that intelligence saturation is a task specific, not global, phenomenon. There’s a lot of value in building benchmarks and knowing what tasks are better matched to specific models.

While this benchmark isn't perfect (e.g. it isn't entirely fair to a model like Fable that is maxed out on tool calling capabilities since my benchmark is created only from my chat transcripts), there's a lot of evidence that a big chunk of queries in my day-to-day usage don't need the absolute frontier.

One side effect of intelligence saturation is that I've begun to focus more on other factors like cost and latency. I'm not going to burn through my usage credits with a more capable model on a query saturated by a less capable model.

I particularly like this chart from Artificial Analysis that extends this idea by plotting intelligence vs cost for various models. As we move away from a world of tokenmaxxing and intelligence at all costs, I suspect we'll all be paying more attention to cost-per-unit-intelligence.