Self-hosted AI became a vendor market this year. The open-source projects that make local models work now have companies behind them: vLLM's creators raised $150 million as Inferact, SGLang's raised $100 million as RadixArk, Ollama closed a $65 million Series B, and the llama.cpp team joined Hugging Face. NVIDIA, Red Hat, HPE, Dell, Nutanix and VMware all sell managed stacks on top. We mapped the whole field in a roundup of self-hosted LLM platforms on wetheflywheel.com.
I'm a solo operator with a modest local AI setup, and I pay none of them. This is where my setup sits on that map, why the free layer is enough for me today, and what would change that.
What is my local AI setup?
It's small, and I've written about each piece before:
- One laptop. A MacBook Pro with an Apple M4 Max and 64 GB of unified memory.
- Local speech. A push-to-talk dictation agent on parakeet-mlx, reachable from my phone over Tailscale, and mlx-whisper for longer recordings. My voice input setup covers how that works day to day.
- Ollama, installed through Homebrew.
- One Linux server that hosts my sites and runs an AI gateway, which routes model calls to cloud APIs. Nine agentic coding hosts sit behind it.
- One planned box. A Mac Studio M5 Ultra for batch transcription, video tagging, local coding models and image generation. I haven't ordered it. The reasoning is in why I'm buying a Mac Studio M5 Ultra for local AI.
Where does that sit among the vendors?
The roundup sorts the market into six layers. Mapping my own setup against them was a useful exercise, because it showed how much of the market is aimed at problems I don't have.
| Layer | What I use | What vendors sell here |
|---|---|---|
| 1. Desktop runtime | Ollama, parakeet-mlx, mlx-whisper. Free. | Ollama's hybrid cloud, LM Studio, Lemonade |
| 2. Inference engine | Nothing separate. One user doesn't need batching. | vLLM and Inferact, SGLang and RadixArk, TensorRT-LLM |
| 3. Cluster and cache | Nothing. I have one machine. | NVIDIA Dynamo, llm-d, LMCache and Tensormesh, exo |
| 4. Model optimization | The quantized model builds Ollama downloads by default | Unsloth, LLM Compressor, Pruna |
| 5. Enterprise platform | Nothing | NVIDIA AI Enterprise, Red Hat AI, the OEM private AI stacks, Cohere North, Zylon |
| 6. Gateway | My own gateway on the Linux server | LiteLLM, Ollama cloud routing, Red Hat's model-as-a-service gateway |
Two of six layers are in use, and both run on free software. The layers where the money is, enterprise platforms and cluster orchestration, solve coordination problems: many users, many GPUs, many teams asking for access, and an auditor asking who saw what. A solo setup has none of those.
Why is the free layer enough for me?
- The engines are the same code. Red Hat's inference server is built on vLLM. NVIDIA has pushed its optimizations into vLLM and SGLang. Inferact and RadixArk are commercializing the projects anyone can download. When I pay a platform vendor, I'm mostly paying for support and governance, not for faster tokens.
- My local work is speech. Parakeet and Whisper are small models that Apple Silicon runs quickly through MLX. There's nothing for an optimization vendor to squeeze.
- My heavy work isn't local. The agentic coding work goes to hosted frontier models through the gateway, because the strongest models aren't downloadable. No self-hosting vendor changes that.
- I already built the one coordination piece I need. The gateway is my layer 6. It gives every caller one API and lets me switch providers in one place. The reasons are in what I rent versus what I run.
Distance running teaches a version of this. A coach, a lab test and a bike fit are all worth paying for, but each one solves a specific limiter, and you hire them once you've hit that limiter, not before. Buying a lactate test in your first month of running doesn't make you faster. Buying an inference platform for one laptop doesn't either.
What would make me pay a vendor?
These are conditions, not plans. Each one maps to a layer where paying for the product would make sense.
- More than one local machine. If the Mac Studio arrives and the laptop still runs local models, I'd want the two to share load. exo pools Macs into one cluster and is free. Jeff Geerling's test of four Mac Studios over Thunderbolt 5 also found the setup unstable, so I'd treat it as an experiment first.
- Other people depending on the local models. If a team or a client used the Mac Studio's models, single-user runtimes would stop being enough. A proper engine such as vLLM or SGLang, plus a chat front end like Open WebUI, would be the next step. That's still free software. A support contract would follow only if the system became something others couldn't work without.
- A data requirement I have to prove. If a client required data to stay on hardware I control, with access logs and a named vendor, a packaged platform would make sense. Cohere North and Zylon are built for that. I'd still check whether my own gateway's logging covers it first.
- Agents running locally at volume. Agents send long, mostly identical prompts over and over. If the coding agents moved to local open-weight models, KV cache reuse is exactly what LMCache does, and that would be the first optimization I'd look at.
What would I tell another founder?
Start with a runtime and a gateway. Run the work that is private and repetitive locally, keep the work that needs the best model in the cloud, and route both through one front door so switching is a configuration change. Don't buy an inference platform until you can name the coordination problem it solves for you. If you can, the roundup lists who solves it.
For an engineering team's version of this decision, ctaio.dev covers local AI for engineering teams and compares Ollama, LM Studio, llama.cpp and vLLM. For a company deciding whether to bring AI in-house at all, aistrategy.guide covers private AI from the executive side.
Sources
- wetheflywheel.com, Self-hosted LLM platforms 2026, for vendor funding, product and licensing facts and their primary sources.
- Jeff Geerling, 1.5 TB of VRAM on Mac Studio: RDMA over Thunderbolt 5 (Dec 18, 2025).
Frequently Asked Questions
What do you need for a local AI setup?
For one person: a machine with a lot of fast memory, a runtime such as Ollama or LM Studio, and models that fit the memory you have. On a Mac, MLX-based tools run speech and language models well. That is the whole stack for most individual work. A gateway, a shared server and an enterprise platform only become necessary as more people and workloads depend on it.
Do I need to pay a vendor to run LLMs locally?
Not for personal or small-team use. The runtimes and engines are free and open source, or free to use at work in LM Studio's case. Vendors such as NVIDIA, Red Hat, Inferact or the OEM private AI stacks sell support, cluster management, access control and compliance. Those matter when many users, many GPUs or regulated data are involved.
Should I run AI locally or use cloud APIs?
Both, split by workload. I keep frontier-model work, such as agentic coding, on hosted APIs because the best models are only available there. Work that is private, repetitive and tolerant of a queue, such as dictation and transcription, runs locally.
What would make a solo operator buy a local AI platform?
In my case: several machines that need to act as one, other people depending on the local models, or a client requiring data to stay on hardware I control with audit trails. None of those is true today.
Need Expert Technology Guidance?
20+ years leading technology transformations. Get a technology executive's perspective on your biggest challenges.
