I'm not sold; that's not to say I can't be swayed, but I've spent a lot of time building RAG solutions as a "one-size-fits-all" against codebases and I am not convinced embedding models or any of the myriad BERT implementations work well enough that they can beat out grep.
Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.
You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.
And if you do use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
Agreed, but I wonder how that plays out without them entering the frontier model space. I just downloaded the new foundation models cli and ran some inference on my Mac, it wasn't great, but it was neat to see. I asked which model it was using and it said gpt 4o mini.
An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
I think it'd be foolish to assume that they aren't already - they have a reputation of doing a lot more internally than ever comes to light. Smaller companies like mistral or thinking machines show that you don't even need that much resources to be semi-competitive, and comparatively Apple is a vertically integrated behemoth.
That said in a pinch they could also acquire / acquihire both of those companies without much thought.
I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.
Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.
all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.
That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.
Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.
You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.
And if you do use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
That said in a pinch they could also acquire / acquihire both of those companies without much thought.
bet could be that this space will be commoditized: many good enough open weight models and multiple competing provides.
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...
"why not use grep?"
https://github.com/lightonai/next-plaid/tree/main/colgrep
With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.