It’s Not About Benchmarks
When I started testing different language models, I wasn’t looking for benchmark numbers or theoretical benchmarks. I wanted to know what it actually feels like to use them. What it’s like to run a model locally, how quickly it responds, and whether it delivers consistent, useful results without drifting into hallucinations.
This article focuses on real-world use. What worked for me. What failed. Where the limits are. And how each model behaved when given tasks like summarizing text, comparing examples, or answering layered questions. This is not a ranking. It’s a personal, hands-on perspective based on repeated use.
Models That Stayed in My Toolkit
After plenty of back and forth, I’ve ended up using Ollama, OpenGPT, and DeepSeek regularly. These models have given the most stable experience in my day-to-day tests. They tend to keep the conversation thread going better than others and maintain relevance in their responses longer.
They’re not flawless, but they consistently handle basic and mid-level tasks without needing constant resets or corrections. That alone makes them valuable.
A Closer Look at BOLT
BOLT deserves its own section. I use it mostly through LM Studio because it hasn’t integrated smoothly with OpenWebUI in my experience. But once it’s up and running, it’s fast. One of the quickest to respond.
BOLT has done well in tasks that involve structured output. Things like formatting logs, simple JSON structures, or YAML samples have worked surprisingly well. The performance is reliable when you’re running focused prompts.
That said, the smaller versions of BOLT do suffer from the same issues as other compact models. They hallucinate and lose context when the prompts become more complex or when the conversation gets longer.
The Breaking Point of Small Models
Every time I’ve tested models smaller than 7 billion parameters, the result has been the same: fast startup but very limited capability. These small models forget context quickly, respond with vague or generic text, and are easily thrown off track. They’re useful if you just want to experiment or see if a setup works, but they shouldn’t be trusted for anything critical.
If the task requires depth or memory, anything under 7B is not a real option. And even at 7B, there are still compromises. I’ve had better consistency when using 13B models, especially when asking multi-step or layered questions.
How I Test in Practice
My testing isn’t about synthetic prompts. I use models on actual tasks: summarizing documents, analyzing notes, testing error messages, and occasionally drafting responses or comparisons.
I also use repetition. I give the same task to multiple models and compare the outputs. This approach gives me a clearer view of where a model falls short or surprises me. Over time, it becomes obvious which ones you can depend on and which ones just look good on paper.
Size Matters – But Not Always the Way You Think
I’ve worked with both 7B and 13B models, and while 13B models are more capable and consistent, they also require more from your system. You need enough free memory to load and run them properly, or they’ll crash or lag.
If your hardware is limited, 7B models can be made to work. But you need to stay within their limits. For quick summaries or one-off responses, they’re fine. But once you ask them to think across multiple steps, they often break.
My Conclusion After All This Testing
Testing local models is not about finding the smartest model. It’s about finding what works for your use case and your system. I care less about how clever a model sounds and more about whether it can finish a task without needing a reset.
Local AI gives you the tools to choose. You can test, delete, switch, and tune things in a way that cloud services don’t allow. That’s the power. But it also means you need to be realistic.
Start simple. Be patient. And let your own use cases guide you. That’s how I’ve learned the most.
This isn’t about replacing cloud AI. It’s about building something that’s yours, that runs on your terms, and that fits your daily needs without compromise.
Markku Arvekari
Digital Transformation Expert
