Tool search is a middleware layer you may not need

Past a certain catalog size, a model gets worse at picking tools. Anthropic's documentation puts the point where selection starts to degrade at 30 to 50 available tools. Plenty of production agents carry three or four times that.

The answer the industry converged on is a middleware layer. The agent is given a search tool and an execute tool instead of its real catalog. It searches, gets back a shortlist, then asks the layer to run the one it picked. Several companies now sell this as infrastructure, and it works: the prompt shrinks and selection improves.

It also changes something that is easy to miss until it bites. The model stops calling tools and starts calling a layer that calls tools.

A dispatcher has to return one uniform shape, whatever the underlying tool produced. In practice that means serialising the result into text.

For a tool that returns a row count, that is invisible. For a tool that returns an image, the image stops being an image.

We watched one of our agents take a screenshot and then tell the person who asked for it that it could not see the screenshot. Its exact words: "I received an image but I can't actually see/interpret it, I only have base64."

It was right. The picture had been flattened into a 22,653 character string before the model looked at it. Nothing was broken in the browser, the tool, or the model. The layer we had added to manage the size of the tool list had quietly changed what the tools were allowed to return.

This isn't anyone's bug in particular. Anthropic's own Programmatic Tool Calling documents the same shape, describing a function that "returns a string: the text of the tool_result". It is an honest trade that comes with routing calls through a wrapper. It's just rarely a trade anybody makes on purpose.

The second cost is subtler: a search step and an execute step are turns. The agent spends part of its reasoning finding its own tools rather than doing the work.

The middleware approach treats tool selection as something the agent does. It doesn't have to be.

The request already passes through your own infrastructure on the way to the model. That is a place where the catalog can be narrowed, before the model sees anything, using a separate model that never joins the conversation.

The agent then receives a short list of real tools and calls them natively. No search turn, no execute turn, no wrapper on the way back. Whatever a tool returns is what the model gets, images included.

What makes this practical is a class of model that answers typed questions with a typed value and a probability, and generates no text. Because there is no generation, output tokens are not billed and the answers come back in a single parallel pass, so the number of questions barely affects the time. On our own key, two questions took 501ms and 111 questions took 428ms.

That inverts the usual advice. Normally you ask as few questions as possible, because each costs tokens and time. Here, asking about every tool you have is the cheap option.

The shape of the question decides whether this works at all, and we got it wrong first.

Our first version asked the model to choose the needed tool from a list. A choice returns a distribution that sums to 1, so it comes back effectively one hot:

browser screenshot: 1.0
everything else:    0.0

Take everything above a threshold from that and you admit exactly one tool per turn. An agent that needs three would find them across three turns, failing twice on the way.

The fix was to stop asking which one, and instead ask separately, for each tool, whether that tool is needed. Several answers can then be high at once, which is what selecting a set actually requires.

Two rules came out of running it:

• Admit a tool's siblings with it. One tool's error routinely names another, such as telling the agent to open a tab before taking a picture of one. That hint only helps if the second tool was also sent.
• Fail open, every time. Model error, timeout, storage failure: all of them send the full catalog. A component that cannot decide must never be the reason an agent loses a capability. Sending too many tools costs tokens. Withholding one costs the task.

Tool definitions sit in the cached part of the prompt. Change them mid conversation and the cache for that whole conversation is invalidated. Anthropic states the rule plainly: never add or remove tools mid session.

That rules out re-picking the set on every turn, which is what a straightforward retrieval approach does.

So the set only ever grows. A tool admitted for an agent stays admitted, and new turns can add but never remove. Here is what that looked like on one of our own agents, in September 2026:

The model was shown 23 tools instead of 131, and finished the task on that set. The selection settled on the first turn and added nothing after it.

The result that mattered most is the easiest to miss: the fingerprint of the tool block stayed byte for byte identical across every request after the first. One cache invalidation at the start, then a stable prompt. That is the entire reason for the never remove rule, and it is the part only real traffic can confirm.

Cost sits near four hundredths of a cent on a turn carrying a new request, and nothing on the follow up steps inside it. Against roughly 9,200 input tokens saved per request, that is about eight times its own cost.

Two things are still open, and both need weeks rather than turns.

The first is how often this withholds a tool the agent genuinely needed. A tool that is never admitted cannot be asked for, so the failure is quiet: the agent simply reports that it cannot do something. We haven't seen it. We also haven't run it long enough to claim we wouldn't.

The second is whether a set that converges inside one conversation stays converged across a month of them. A tool admitted for a one off task stays admitted, so the set can only drift upward, and far enough upward it stops being a filter.

There is also an inefficiency we left in deliberately. Once the set has settled, each new request still asks about the tools that have never been admitted, and the answer is always no. It is cheap, and we'd rather keep paying it than stop asking and miss a genuine change of subject.

The wider case for this class of model, including where it is unreliable, is in System One models know which work to hand back.