Why Run a Local LLM If ChatGPT Already Works?

A local LLM runs inference on hardware you control, not on a shared cloud API. Ollama and LM Studio wrap llama.cpp for that job.

Ollama raised $65M in July 2026. CEO Jeff Morgan told TechCrunch it is used by over 8.9 million developers every month.

This page is the unbranded job: when on-box inference wins, which VRAM class you need, which layer of the stack you are on, and why published break-even numbers disagree by orders of magnitude.

You are not installing ChatGPT weights, not running ComfyUI image gen, and not ranking Claude Code or a Mac Mini “personal agent” SKU.

Key Takeaways

• Local means inference on infrastructure you operate. It does not automatically mean offline, on-device, or private.
• Pick hardware by peak VRAM at Q4, then speed, then watts. A 24 GB card is a 30B-class box. 70B Q4 peaks at 46-48 GB.
• Treat the stack as weights, then engine, then wrapper. Ollama is a llama.cpp wrapper on port 11434. vLLM is a production scheduler. They are not peers.
• Default desktop quant is Q4_K_M. “Best model” is family × size × quant × VRAM × job, not a ranked ten.
• Total cost is utilization math. GPUs bill idle hours. APIs bill tokens. Do not print one crossover as fact.

What Is a Local LLM?

A language model is “local” when inference runs on a machine you control: a laptop, a GPU workstation, a home server, or an on-prem box. Training is a different job. Almost every “run it locally” walkthrough means serving tokens, not fine-tuning a base checkpoint.

Five labels get collapsed in search results. They answer different questions.

A home-server chat is self-hosted, but it is not on-device from your phone. A phone model can still phone home.

NAT plus an allowlist is isolated, not air-gapped. Air-gap means zero egress, no external DNS, and weights staged before the box goes dark.

After a model is on disk, LM Studio’s offline docs are blunt. Chat, document RAG, and the local server do not need internet.

“Nothing you enter into LM Studio when chatting with LLMs leaves your device.”

Discover, model download, runtime download, and app updates still do.

Ollama’s FAQ says prompts are not sent to ollama.com when you run locally. Optional Cloud and web search do send the request. Default bind is 127.0.0.1:11434.

Yagil Burowski, who built LM Studio:

“Private LLM” as an iOS app is a store listing, not this stack.

Google AI Edge and Apple Foundation Models are on-device phone NPUs. This hub is workstation and server inference.

Why Local Inference Matters in 2026

llama.cpp joined Hugging Face on 20 February 2026. The projects stay MIT, with a new front door at llama.app.

Ollama’s Series B added a cloud SKU for larger open models. LM Studio’s 0.4.0 split the GUI from the llmster daemon.

Bionic is a separate agent app. LM Studio says its optional Secure Cloud uses zero data retention by default.

Operators who arrived for “nothing leaves the box” now have to read which button they pressed. If you already pay a hosted OpenAI API bill, on-box inference is the other side of that trade: you buy VRAM and ops instead of tokens.

How Local Inference Works: Weights, Engine, Wrapper

Tim (Tech With Tim) in Local AI Explained (5:07):

“Almost every tool that I'm going to show you today, so LM Studio, Ollama, Docker Model Runner, they're all basically just a wrapper around engines like this.”

Three layers. Confusing them is how “Ollama vs vLLM” became a four-figure monthly search.

Weights

The file comes from Hugging Face Hub: GGUF for desktop llama.cpp-family runners, safetensors for vLLM and SGLang. Hugging Face is the registry, not a runner.

Open-weight is not the same as open. Hugging Face engineer Merve Noyan: “just because weights are on Hugging Face, it doesn't mean a model is fully open!”

Llama 3 (Meta’s Llama site) uses a Community License with a 700M MAU restriction. Qwen 2.5 ships Apache 2.0 on most sizes (not 3B or 72B).

Gemma 4 is Apache 2.0. Mixtral is a MoE: active parameters are not total parameters.

Engine

llama.cpp (MIT) runs GGUF across Metal, CUDA, Vulkan, and SYCL, including CPU+GPU hybrid. MLX is Apple-only.

vLLM is PagedAttention plus continuous batching (Apache 2.0). SGLang’s RadixAttention wins when prefixes overlap. It does not automatically win on unique prompts.

Wrapper

Ollama is a CLI and daemon on 11434. LM Studio is a GUI on 1234 (2026: llmster headless). Jan and Msty sit in the same GUI bucket.

Open WebUI and AnythingLLM sit on top of wrappers. They are UIs, not engines.

Ollama wraps llama.cpp (and MLX on Apple). “Ollama vs llama.cpp” is wrapper vs engine. “Ollama vs vLLM” is laptop daemon vs production scheduler.

When On-Box Inference Beats a Cloud API

On-box inference wins on a data boundary, offline or air-gap work, predictable high token volume, control of the runtime, and tail latency on a LAN. Cloud wins on frontier quality, burst latency, zero capex, and zero GPU ops.

Do not promise that a 27B Q4 “beats ChatGPT.” Syntax’s CJ said the split out loud in Local AI Explained (20:13): basic question answering, search, and summarizing replaced his hosted chat. A from-scratch coding project did not, so he will not cancel his Claude subscription.

On r/LocalLLaMA, u/Joozio framed the agent case without the thumbnail claim:

"This isn't 'local AI competes with Claude.' It's 'not every agent task needs a frontier model.' A lot of what agent systems do is genuinely simple: read a file, format output, summarize a short note, route a request."

u/Joozio in r/LocalLLaMA (Mar 2026)

On r/LocalLLM, the failure mode that sends people back to a hosted product is not tokens. It is state.

"What nobody mentions is what breaks it around month three. Not tokens or model quality. State. The agent starts every session from zero and you become its memory and that is the point where people drift back to a hosted product with a chat history."

u/saltexx in r/LocalLLM (Aug 2026)

Put memory on disk (a git repo, a notes vault) if you want the model to be swappable. Pair that with LLM observability if more than one person hits the endpoint.

Quality comparisons against frontier chat still live on Claude vs ChatGPT.

Air-gapped is a one-paragraph constraint, not a compliance title. Zero egress is the test. Hugging Face, Docker Hub, and ollama pull all fail in a real gap.

Pre-stage GGUFs and container images. Isolated (NAT + allowlist) is not air-gapped.

Talk to counsel if you have a DPA to satisfy. This is not a HIPAA runbook.

Hardware Fit: VRAM, Unified Memory, and Power

Fit first. Speed second. Power third.

Peak VRAM is weights plus KV cache plus buffers plus CUDA or Metal overhead, not the GGUF file size. 9bench treats Q4_K_M as the default most local users run. A 7 GB GGUF does not fit 8 GB VRAM once KV cache and overhead land.

9bench reports these text-LLM peak-VRAM rows (Q4, skip image models):

Apple unified-memory rule of thumb from the same 9bench write-up: macOS hands the GPU about 75% of RAM. An M3 Max 64 GB box is ~48 GB usable. sudo sysctl iogpu.wired_limit_mb=… is the Mac setup step people skip, then they wonder why the 70B never loads.

Speed Crossover: 4090 vs Apple Silicon

On small models a discrete 24 GB NVIDIA card wins on speed. On 70B Q4 it is offload: ~40 GB of weights do not fit 24 GB, so a 64 GB Apple Silicon box is the class that actually loads.

Tim’s 4090 matches that ceiling, not a shopping-list winner:

"I have 24 GB of VRAM in my 4090, and it's very fast and can generate, you know, 200 tokens per second for some of the models that I run. However, I can't run models that are 70 billion parameters like I might be able to on my Mac."

Tim in "Local AI Explained" (Tech With Tim, 8:25)

If a bench claims 70B Q4 at 12 tok/s on a single 4090, treat it as offload, a different quant, or a different card count.

Without a Discrete GPU

CPU llama.cpp and Apple Silicon both count. Small Q4/IQ quants on 7B-9B are the honest path. A Raspberry Pi is a demo, not a self-host default.

A Mac Mini belongs here as an Apple Silicon / MLX box, not as someone else’s agent appliance.

On r/LocalLLaMA, 16 GB is viable for a 27B only with aggressive weight plus KV quant and one slot. A 24 GB 4090 is a coding daily-driver if you keep context in check and restart.

Copying NVIDIA speculative-decoding recipes onto an M4 Max can slow the box:

"For those of you on MacOS, benchmark MTP before you enable it. I tested, and at least on my M4 Max MTP makes everything slower, not faster."

u/jwr in r/LocalLLaMA (Aug 2026)

Fine-tunes still prefer CUDA.

Pick a Runner: GUI, Daemon, Engine, Serving

Three paths. Not twelve product reviews.

Tim’s split in the same video (23:09): never see a terminal → LM Studio. Scripts and apps → Ollama. On r/LocalLLaMA, power users then leave Ollama for raw llama.cpp once they know the model.

"Ollama is a great jumping on point for local LLM. Much easier to get going, 'ollama pull' takes away a lot of hugging face headaches for beginners. But once you know what model you like, switching over to llama.cpp will generally see better performance and much easier to customise."

u/PaxUX in r/LocalLLaMA (Sep 2026)

The other camp bounced off Ollama because it would not reuse GGUFs already on disk and called LM Studio the easier GUI. Both can be true.

LM Studio does not use Ollama. Both wrap llama.cpp.

Ollama Context Is Not the Model Card

The FAQ default is 4096 tokens (OLLAMA_CONTEXT_LENGTH, /set parameter num_ctx, API num_ctx). A newer context-length page publishes VRAM-tiered defaults: under 24 GiB → 4k; 24-48 GiB → 32k; 48 GiB+ → 256k.

Agents and coding should set at least 64k. Check ollama ps CONTEXT. Parallel slots multiply that reservation.

OLLAMA_NUM_PARALLEL defaults to 1. The Processor column tells you 100% GPU, 100% CPU, or a split (offload).

ollama ps

vLLM --api-key Is Not a Full Lock

On the OpenAI-compatible server, --api-key covers /v1, /v2, and /inference. /invocations can sit unauthenticated. Put a reverse proxy in front if the port is reachable.

Georgi Gerganov flagged VS Code’s custom OpenAI-compatible endpoints as the unlock: any local provider, no vendor lock-in.

Pick a Model Class, Not a Ranked Ten

A “best local LLM 2026” listicle is already rotting. Satisfy that query with family × size × quant × VRAM × job.

Start at 7B-8B Instruct on 6+ GB VRAM (Llama, Qwen, Mistral). Coding wants a coder-class checkpoint. Chat wants Instruct, not base.

RAG wants a model that follows short grounded context, not a 70B that spills out of VRAM.

Default desktop quant: Q4_K_M. 9bench calls it what most local users run. Step up only when VRAM has real headroom. Q3 is the squeeze path if you are forcing a 70B onto 24 GB.

Ollama’s KV cache default is f16. q8_0 cuts that roughly in half. Avoid q4_0 KV for code or long context.

GGUF is a container, not a precision. Filename literacy:

Llama-3.1-8B-Instruct-Q4_K_M.gguf

Family, size, Instruct vs base, ~4-bit, K-quant, Medium. Publisher (Unsloth, Bartowski) matters as much as the letter.

A bad GGUF will lose a Q4-vs-Q8 eval to sampling error. Treat vendor “+10% accuracy / 1-bit on 8 GB” posts as vendor posts.

Version strings (Llama 3.2, Qwen 2.5, Gemma 4) are news. They do not belong in the H1.

Coding on a Local Endpoint

Point Continue, Cline, or a similar harness at base_url on :11434 or :1234. Qwen-coder and DeepSeek-coder class models are the local coding bet. Hosted AI coding assistants still win at unguided multi-file work.

Claude Code and Cursor are products that call a model. They are not an on-box weight file.

Keep tests and lint as checks the on-box model cannot skip.

Local RAG

Ingest, embed on-box, retrieve, generate. AnythingLLM collapses runner plus vector store plus UI into one workspace.

Use it as an example app, not a ranking. Local RAG is ordinary RAG with a local embedder and a local generator.

Serve One User or Many

Ollama, LM Studio, and llama.cpp are consumer, single-user, CPU-fallback tools. vLLM and SGLang are shared-GPU schedulers. Alex Ziskind (1:55): llama.cpp by itself cannot handle as many concurrent connections as vLLM.

Attach GPU, model, and date to every serving number. Do not crown a laptop winner from an A100 graph.

Winner depends on load shape: time-to-first-token vs saturation vs prefix cache. vLLM’s own note on speculative decoding matches the Mac MTP report: no universal winner across MTP, EAGLE-3, DFlash, or DSpark.

Set base_url to Ollama :11434, LM Studio :1234, vLLM :8000, or LocalAI. Swap the process behind the socket without rewriting the client.

Evaluate Two Different Jobs

Model quality. EleutherAI lm-evaluation-harness is the backend behind the Hugging Face Open LLM Leaderboard (HF / vLLM / OpenAI-compatible backends). Wrong tool for RAG or agent application eval. GGUF eval needs a matching tokenizer or the run can hang.

Runtime. Tokens per second, time-to-first-token, and ollama ps (is it actually on GPU?). LocalScore is a pointer, not a leaderboard to republish. MMLU will not pick your coding model.

What On-Box Inference Actually Costs

Weights are free. GPUs bill idle hours. An API is $0 at idle.

Self-host beats per-token API only with steady traffic.

Published crossovers disagree by orders of magnitude because they answer different questions. Show the shape. Do not pick one number.

Track hardware, electricity, labor, depreciation, high availability, and idle reserve. Hardware dollar tables date. Closed-model SKU names date faster.

A GPU at low utilization is several times the per-token cost of a busy GPU. Spendark puts the self-host line around ~50% utilization, on the order of 2 billion tokens a month for a single H100.

Codersera talks about 2-5 million tokens a day on a reserved GPU over 12 months. Those are not the same claim.

Hidden costs (HA, engineering time, idle reserve) push break-even up. Privacy and residency can still justify on-box inference below the cost line.

Hobbyist wall-watts are a third measurement. Towards Data Science (28 Jul 2026) measured an M3 Ultra 96 GB at $0.31/kWh: Qwen3.5-4B dense at $0.063 per 1M output; a ~80B MoE coder near $0.103; a 27B dense at $0.554.

Electricity cost of a token is watts divided by throughput. A large MoE can be cheaper and faster than a smaller dense model because cost tracks bytes streamed per token.

Real traffic is lumpier (plan 2-3×). Do not average Mac wall-watts with H100 break-evens.

On r/LocalLLM, the lived-in version is not a formula either. After about $3,000 of H1 2026 API spend, u/Bertoluci_669 cancelled the big hosted models once a local 27B sat as the daily driver.

The counter-camp stays on a ~$20/mo coding SKU because a 5090-class box is thousands. Both are utilization stories.

Best Tools for On-Box Inference

Jobs, not a ranking. Cloud add-ons exist on Ollama and LM Studio. They are optional and they send data.

Common Setup Mistakes

Treating Ollama and vLLM as the same layer

Ollama is a llama.cpp wrapper with a 4k default context and a parallel cap. vLLM is a continuous-batching server.

Red Hat’s 793 vs 41 TPS is an A100 / 8B number. Pasting it onto a 4090 laptop is the mistake.

Trusting GGUF file size as VRAM

Peak VRAM includes KV cache and runtime overhead. Size the box with 9bench’s peaks, then confirm with ollama ps. If Processor is a CPU/GPU split, you are already offloading.

Assuming the model card’s 128k window

Ollama will not silently give you 128k. Set num_ctx. Watch CONTEXT in ollama ps.

Two parallel chats can double the reservation.

Printing one TCO crossover

2 billion tokens/month/H100 and 2-5 million tokens/day are different studies. Mac wall-watts are a third. Use utilization, then decide whether privacy pays for the idle hours.

Calling any localhost bind “private”

127.0.0.1 is local. Binding 0.0.0.0 on a LAN is self-hosted.

Optional cloud and web-search buttons on Ollama and LM Studio leave the box. vLLM --api-key does not cover every path.