When my coding assistant shows automatic in the model selector, my reaction is mixed. I want the convenience. I also want to see and influence the decision, especially when an investigation gets stuck or a new model gives me something else to try.
In this episode, I explain what would make me comfortable handing over routine model selection. A cheaper model can leave more of my budget available for a difficult task. Repeated explanations and corrections can consume that saving, along with my time. I want to compare what it costs to reach a change I can accept.
I use a hypothetical Java service that creates duplicate orders after a timeout to make the tradeoffs concrete. Finding the retry configuration may be straightforward. Establishing what happened after the receiving service saved an order needs more investigation. That gives a router more information to act on, while showing why a stronger model also needs the right code, context, and tools.
I also question my own expectations about the latest models. Research on novelty and confidence gives me a reason to check whether a response handled the difficult constraint, even when I like the answer. Those studies have limits, which I discuss alongside what a developer can reasonably judge in a familiar service.
Then I share experiments I’m driving at IBM to connect context information with recorded activity. I want to follow what the assistant read, which claims it checked, and where an assumption entered the conversation. Being able to inspect that history would help me decide whether to provide more information, request a more capable model, or accept the change.
The episode is narrated locally with Fish S2 Pro and my trained LoRA voice adapter.
Chapters
00:00 — Intro
00:12 — Why my reaction to automatic routing is mixed
02:45 — More help from the same budget
05:22 — Routing a Java duplicate-order investigation
08:15 — Hidden choices and the control I want
10:43 — What depends on the harness
12:49 — New models, expectations, and judging quality
15:39 — Making conversations easier to inspect
18:28 — The controls that would make me comfortable
20:47 — Outro
Links and references
RouteLLM: Learning to Route LLMs with Preference Data — Research on choosing between stronger and cheaper models using preference data. Its reported cost savings describe the evaluated benchmarks, rather than a guaranteed saving on a development task or subscription.
SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks — A cheaper model takes exploratory turns before the router decides whether to continue or escalate. In this implementation, the stronger model restarts from the original prompt; the exploration counts toward the cost.
Quantifying the Novelty Bias when Evaluating Interactive Prototypes — A study of 48 participants using functionally identical technology pairs, including chatbots, presented as old and new. The brief interactions with general users do not establish an effect size for professional developers doing sustained coding tasks. Author-hosted paper.
Do Users Write More Insecure Code with AI Assistants? — An earlier security-task study in which assistant users produced less secure code overall while expressing greater confidence in its security. It used an older model and specific tasks; it is not a ranking of current assistants.
