The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers.
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
These are cool results and I'm glad that tokens are being invested to eval harness capabilities.
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.
I think Conway's law applies. In an ideal situation your code mimics your organisational structure.
The places where you let an agent do its thing is bound by the (implicit) interface your prompt creates. In an ideal situation this interface is a simple isolated thing in your application.
If you just let it do its thing, expect a monolith. You shouldn't rely on it coming up with interfaces that mesh well with what you had in mind, and each of those interfaces represents a decision that it took, restricting future iterations of the code. So a monolith is not only what you can expect, but also the best you can hope for.
If you want to let it cook, create more places for code that does a thing and you don't care how.
This comment could be interpreted at least two ways, and I’m not sure which it is:
- tongue in cheek: in topics you know very little about, you don’t have the ability to tell whether the LLM output is trash or not
- very serious: LLMs allow you to create things you could not have created before due to not having the knowledge
Sort of like a Dunning-Kruger or Gell-Mann amnesia.
LLMs are allowing people to create things with very little effort compared to before, and those creations are of questionable quality - indeterminable by the LLM-user due to a lack of experience.
Not every problem should (or even could sometimes) be solved the same way. Are you planning to maintain the produced code for more than a year? No, then wave away. But if you're expecting others to depend on it, and you want to be able to continue evolving what it is, then a tight leash and full understanding of how it actually works, is very much needed, even if you don't actually type characters into an editor anymore.
Otherwise you still end up with balls of spaghetti where each change takes longer and longer to work into the codebase. With a bit of planning, you can keep tackling changes and they keep taking the same amount of time more or less.
The cost of AI is rapidly falling while its engineering capabilities keep rising. This implies that code is becoming more fungible, and you should expect any code you write today to mostly be replaced in the near future.
The requirements will be more stable than the code. Focus on them instead.
> The requirements will be more stable than the code.
These kinds of changes to the way teams are formed and managed, and how products are developed is what really fascinates me about the impact of coding agents.
Even if it works, for some definition of works, to dump a huge spec into a coding agent context window, I suspect this is where a lot of horror stories about agentic coding come from.
Teams, and the management structure above them, are going to have to get used to more collaborative and iterative product definition and development in order to extract the most value from coding agents.
Robust systems experience very little turnover in nearly every non-software discipline that involves systems. Why the hell would we insert risk into our system-building processes by needlessly turning over code?
How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.
I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
I don't think this is common, though it would be interesting if ThinkingMachines/HuggingFace/OpenRouter/BaseTen offered fine-tuned variants of the most popular open models that worked with specific harnesses
However, the hardware requirements are still quite enormous.
I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.
Taking the open weight model and altering those weights in order to overfit for success on the tool shape / quirks of the harness you are running the model within.
As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.
Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.
I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.
>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title.
The page's title is: "Introducing FrontierHarness Eval v1.1"
Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps.
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found:
The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3.
Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost.
The model still sets most of the bill (93% of cost variation). The harness is the second lever.
Starting prompt size doesn’t predict cost. Output tokens do.
64% of inference spend went to runs that didn’t pass.
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
This is why for open-weights models at least harness aware post-training is becoming a thing. Also closed source harnesses that aren't fully editable are basically an anti-pattern. The labs should just hurry up and productionize self-evolving harnesses, and have the models internalize harness engineering, so they can optimise the harness for the given task. These are solvable problems.
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
https://www.stet.sh/blog/cost-per-successful-task
The places where you let an agent do its thing is bound by the (implicit) interface your prompt creates. In an ideal situation this interface is a simple isolated thing in your application.
If you just let it do its thing, expect a monolith. You shouldn't rely on it coming up with interfaces that mesh well with what you had in mind, and each of those interfaces represents a decision that it took, restricting future iterations of the code. So a monolith is not only what you can expect, but also the best you can hope for.
If you want to let it cook, create more places for code that does a thing and you don't care how.
The "objective" nature of code is revealing a flaw in the human division of labor.
I find that only works with topics I know very little about.
LLMs are allowing people to create things with very little effort compared to before, and those creations are of questionable quality - indeterminable by the LLM-user due to a lack of experience.
Otherwise you still end up with balls of spaghetti where each change takes longer and longer to work into the codebase. With a bit of planning, you can keep tackling changes and they keep taking the same amount of time more or less.
The requirements will be more stable than the code. Focus on them instead.
These kinds of changes to the way teams are formed and managed, and how products are developed is what really fascinates me about the impact of coding agents.
Even if it works, for some definition of works, to dump a huge spec into a coding agent context window, I suspect this is where a lot of horror stories about agentic coding come from.
Teams, and the management structure above them, are going to have to get used to more collaborative and iterative product definition and development in order to extract the most value from coding agents.
Wouldn't count on it. There likely is a ceiling, we might or might not be near it.
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
However, the hardware requirements are still quite enormous.
I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.
As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.
Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.
https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)
I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
The page's title is: "Introducing FrontierHarness Eval v1.1"
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
https://news.ycombinator.com/newsguidelines.html