Why It Matters Where AI Runs
The growing use of artificial intelligence in business and personal tools has created a question that many are just starting to ask: where should my AI run? Most people are used to cloud-based services like ChatGPT, Copilot, or Gemini. But for some industries and cases, cloud isn’t an option. That’s where local AI comes in.
This article kicks off a 10-part series where I share my own journey into local AI – testing models like LLaMA4 and BOLT, setting up tools like LM Studio and OpenWebUI, and building my own AI lab at home. I’m not here to convince you to ditch the cloud. I’m here to show that there are real reasons why local matters – and how it can work, today.
The Use Case That Got Me Started
I didn’t begin this because I wanted to move away from cloud AI. I started because I wanted to understand how to run AI where the cloud is not allowed.
Think about sectors like healthcare, finance, defense, or government. These industries often have strict rules: sensitive data must stay on-premises, AI models must be explainable, and the infrastructure must be under direct control.
As someone helping businesses navigate digital transformation, I saw a growing demand: ”How can we use AI safely and privately in our own environment?”
So I decided to get hands-on.
Cloud AI Is Powerful – But Not Always the Answer
Let’s be clear. Cloud AI offers many benefits:
-
Scalability
-
Easy integration
-
State-of-the-art model performance
-
No hardware requirements
But it also comes with trade-offs:
-
Privacy concerns
-
Data residency issues
-
Cost per use
-
Dependency on external uptime and licensing terms
Not all companies can afford to take those risks. And not all workflows need the power of a 175B-parameter cloud model. Sometimes, local is better.
What Surprised Me About Local Models
When I started running models locally, I expected the worst. But here’s what I found:
-
Smaller models can be useful: Even 4GB or 7B parameter models can summarize, help with idea generation, and simplify documents.
-
BOLT gave surprisingly good results, better than I expected from a locally hosted system.
-
Offline = Peace of mind: There’s no API watching your inputs. No one logging your queries. It’s just your machine and your ideas.
But it wasn’t all smooth. In more complex tasks — like holding a long conversation, analyzing mixed data, or understanding tricky instructions — these lightweight models often struggled.
Hallucination and Context Loss in Complex Tasks
What surprised me early on was how quickly some models lost context. You might start a multi-step task or a detailed discussion, and suddenly the model answers something completely unrelated. This isn’t a bug, it’s a limitation. Small models often:
-
Forget what was said just two messages ago
-
Invent facts or names when under pressure
-
Cannot keep structure in more advanced workflows
This doesn’t make them bad. It just sets a limit on how you use them.
My Goals for the Series
This isn’t about proving a point. It’s about:
-
Showing how I built my setup (and how you could too)
-
Comparing tools like OpenWebUI and LM Studio
-
Testing models like Ollama, LLaMA4 and BOLT
-
Exploring real use cases for business, learning, and innovation
If you’ve been curious about local AI, but don’t know where to begin, this series will show you the path — from one practical step to the next.
Markku Arvekari
Digital Transformation Expert
