around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.
I’ll look into adding harnesses. The benchmark sources usually don’t contain harness so the problem is limited data. What models would you like to see?
He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.
This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
https://github.com/ggml-org/llama.cpp/issues/27330
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.