Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
I'm curious, you say "super interesting engineering" but then they say "it will often just tell you, even though we’ve asked it not to" and to me that seems like extremely shit engineering.
Where are the interesting engineering parts at? Seems to be an interesting idea and perhaps design, but to call the implementation/engineering itself bad seems to be an understatement.
"turn -> FINAL -> schedule wake-up", this is where my excitement has faded unfortunately. Many of us are probably wondering about the same idea: bridging the gap between a reactive agent and my daily workflow or existence. But, this still feels too close to how Claude (or any other agent) runs as a process in the background (always ON), where you can use a custom channel to feed the dialog with external signals like chat, CI/CD events, whatsapp, etc.
Humans aren't scheduling a wake-up to the next thought. Ideally, a sub second agentic loop with no FINAL / wake-up, always "spinning" would get closer. I'm conscious about the waste of resources this would drag with it (because of current architectures), but exciting still.
PS. I love the take on using bash instead of Python (one less abstraction layer!) and using UNIX fundamentals as stepping stone when composing tools as agents are naturally drawn to using it on a box anyways.
Don't use any public benchmarks, every single one is worthless for your own use cases essentially.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
andy here (headlong post author).
terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them.
w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
The language composition is interesting. The source is half Shell, a quarter Python, almost a fifth Typescript. Among the rest is 2.3% Rust and 1.3% Swift.
If you wonder what the Rust is for: It is the Ratatui TUI.
Sub Question : IS there a real successful agent product today that uses a library for harness(like langgraph etc)? Building our own worked for us. Works with our components(postgres, events ...) and scales naturally with our system.
I don't know about real successful. Since you mentioned Langchain, you could look at https://www.langchain.com/dcode which is a CLI harness build off Langchain deep agents.
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
> Audel designed experiments to spawn recursive shellm sub-runs to work on subproblems. Most of the experiments failed, because shellm has a safety watchdog that kills any command that stays silent for 30 seconds. Audel fought the watchdog for about 40 minutes and mostly stopped using shellm sub-runs. Results from recursive sub-runs of shellm merged back into Audel’s mind 64 times in its first two days and 12 times in the twelve days since. We’ve since revamped the watchdog, and we’ll see if we can convince Audel to give recursion another shot.
This is why "I made it think in a loop" doesn't result in significant improvement in LLM performance. It's not learning. You need RLAIF, STAR, IDPO, etc to retrain the model to learn from its mistakes. And you need a human to review it so it's not compounding mistakes. It's expensive and time-consuming. Doing it wrong leads to bad outcomes. But not doing it leads to no significant improvement.
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
Where are the interesting engineering parts at? Seems to be an interesting idea and perhaps design, but to call the implementation/engineering itself bad seems to be an understatement.
Humans aren't scheduling a wake-up to the next thought. Ideally, a sub second agentic loop with no FINAL / wake-up, always "spinning" would get closer. I'm conscious about the waste of resources this would drag with it (because of current architectures), but exciting still.
PS. I love the take on using bash instead of Python (one less abstraction layer!) and using UNIX fundamentals as stepping stone when composing tools as agents are naturally drawn to using it on a box anyways.
There are just so many now that it's hard to personally test them all or just trust the vibes.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
If you wonder what the Rust is for: It is the Ratatui TUI.
https://github.com/exoharness/exo/
https://github.com/laude-institute/headlong
https://github.com/microsoft/agent-lightning
and now https://github.com/PrimeIntellect-ai/prime-agent
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
This is why "I made it think in a loop" doesn't result in significant improvement in LLM performance. It's not learning. You need RLAIF, STAR, IDPO, etc to retrain the model to learn from its mistakes. And you need a human to review it so it's not compounding mistakes. It's expensive and time-consuming. Doing it wrong leads to bad outcomes. But not doing it leads to no significant improvement.
Wow. So, be nice or I'll replace you with a very large shell script?