Local-First AI Tooling: When Does Local Inference Win?

Local inference is best understood as an architectural optimization with a narrow but real winning region — not as a privacy default, not as a latency win, not as a moral stance against hosted models. This article maps the runtimes (llama.cpp, Ollama, MLX, vLLM), the quantization landscape, the agent-harness integration story (which has been widely mislabeled), and the conditions under which local actually beats a frontier API. The honest summary, defended below: cloud frontier models remain the default for open-ended agentic work, and local wins only when a specific constraint — boundary, offline, latency-sensitive narrow task, or amortized high-utilization workload — has been demonstrated by measurement, not assumed by reflex.

Coverage note: verified through May 2026.

1. What "local-first" actually means

The phrase "local-first AI" is used loosely enough that articles routinely conflate four distinct boundaries:

Boundary	What is local	What is not local
Local execution	Agent harness, file system access, tool calls, shell	Model inference (may still be a hosted API)
Local inference	Model weights, decoding, KV cache	Pretraining, fine-tuning data, sometimes embedding services
On-prem deployment	Inference inside organization-controlled infra	Public internet, third-party datacenters
Air-gapped	Everything from setup onwards: no package mirrors, no model registries, no telemetry	Anything requiring network access — including, in many setups, the initial model download

These are not interchangeable. Claude Code, for instance, "runs locally" in the sense that the harness, the repo, the shell, and the tool calls live on the developer's machine, but its data-usage documentation is explicit that prompts and model outputs are sent over the network to interact with the model and that local plaintext session caches are kept by default (Anthropic Engineering). A user who treats "Claude Code is local" as a privacy claim has misread the boundary.

The same precision matters in the other direction. A model whose weights live on the developer's machine but whose inputs are still streamed to a hosted observability service has not preserved confidentiality; it has merely changed which vendor sees the data. "Local-first" only earns its claims at the boundary where the boundary actually holds.

For the rest of this article, "local inference" means the decoding pass — token generation given a prompt — happens on hardware the user controls. Everything else is named explicitly.

2. The runtime layer: four tools, four niches

The four most cited local-inference projects do not compete for the same job.

Project	Primary niche	Hardware focus	What it gives you
llama.cpp	Single-machine inference, CLI/server, broad portability	CPU SIMD, NVIDIA, AMD, Apple Metal, Vulkan	GGUF runtime, 1.5-bit through 8-bit quantization, perplexity/benchmark tooling, an OpenAI-compatible HTTP server (llama.cpp README)
Ollama	Local model management and a stable local API	Whatever llama.cpp and friends support beneath it	Model pulls, named tags, a default API at http://localhost:11434/api, OpenAI- and Anthropic-shaped compatibility shims (Ollama API docs, Ollama OpenAI compatibility)
MLX	Apple-silicon-native array framework	M-series Apple Silicon	Unified-memory CPU/GPU execution, Python/Swift APIs, used by mlx-lm and adjacent projects (MLX docs)
vLLM	High-throughput GPU serving for production-grade workloads	NVIDIA (CUDA), AMD (ROCm), Intel (XPU), TPU paths	PagedAttention KV management, OpenAI- and Anthropic-compatible servers, multi-tenant request scheduling; the original paper reports 2–4× throughput at comparable latency over prior serving systems on evaluated workloads (Kwon et al., 2023, vLLM quickstart)

Treating these as a single column in a comparison table — as if they were direct substitutes that you'd score on the same axes — obscures the point. llama.cpp is the underlying inference engine and a server; Ollama is a user-facing model manager that wraps it (and other engines) in an ergonomic API; MLX is an Apple-stack array library that supplies a different runtime path entirely; vLLM is a production serving system that solves throughput and concurrency problems neither of the others addresses.

The first decision is therefore not "which of these four," but which architectural slot you are filling:

• Single developer, single model, mixed hardware: llama.cpp directly, or Ollama on top of it.
• Apple Silicon laptop or workstation: MLX-backed runtimes outperform CUDA emulation paths and use Apple's unified memory natively.
• Multi-tenant serving, predictable concurrent load, GPU fleet: vLLM. Nothing in the llama.cpp family is designed to schedule hundreds of concurrent decodes against a shared KV cache.
• Mixing all of the above: this is the brittleness section. Skip ahead.

The runtime layer is where most local-first articles stop. The harder questions live in the layers above and below it.

3. The capability layer: frontier-versus-open-weight is the dominant variable

For most real agentic AI engineering workloads, the deciding factor is not latency or cost but whether the model can do the task at all, end-to-end, with acceptable failure rate. A model that runs at 200 tokens/sec on the developer's GPU but fails 60% of multi-step tool-use traces is not faster than a 50 tokens/sec frontier API call that succeeds 90% of the time, because the local model's wall-clock includes retries, debugging, supervision, and silent error correction by the human.

The empirical base rates for serious agentic coding work are sobering even for frontier models. SWE-Bench++ reports 36.20% pass@10 for Claude Sonnet 4.5 and 34.57% for GPT-5 across a 1,782-instance subset of repository-level tasks (Sun et al., 2025). SWE-Bench Mobile finds 12% success for the best configuration across 22 agent-model setups on production iOS tasks, and reports up to 6× variance from the same model under different agent designs (Wei et al., 2026). These numbers are the best case: hosted frontier models, well-tuned harnesses, public benchmarks. Open-weight local models score below them on the same suites under the same harnesses.

This is not a permanent gap, and it is not uniform across tasks. Open-weight models in 2026 are now genuinely competitive — often equivalent — for narrower task classes: classification, structured extraction, embeddings, reranking, autocomplete, single-turn code completion, summary generation over short contexts, and well-bounded RAG over indexed corpora. They lag furthest on long-context multi-turn tool-use loops, which is the workload most agent harnesses are designed for.

The decision rule is therefore not "is the open-weight model good?" but "is the open-weight model good enough on the actual workload, evaluated on actual tasks, against an actual harness?" The wide-task case for cloud is strongest; the narrow-task case for local strengthens with every quarter.

4. Quantization: not a clean ladder

Most surveys present quantization as a neat quality ladder: FP16 > Q8 > Q5 > Q4, with a footnote about memory savings. This is the wrong shape.

The empirical literature supports a more precise claim: for a fixed model architecture and a fixed task, lower precision usually preserves most of the quality while shrinking memory and increasing throughput, until it doesn't, and the inflection point is task- and format- and context-dependent.

The k-bit scaling-laws work — 35,000+ experiments across model families and bit widths — finds 4-bit precision often optimal for the bits-per-parameter / zero-shot-accuracy tradeoff (Dettmers & Zettlemoyer, 2022). LLM.int8 demonstrates that 8-bit quantization can preserve full-precision performance on tested large models while halving memory (Dettmers et al., 2022). These are real results — but their domain is academic zero-shot evaluation, not production tool-use loops or long-context retrieval.

The newer llama.cpp-focused study is explicit that quantization comparisons are difficult precisely because formats (GGUF k-quants, AWQ, GPTQ, EXL2, NF4, KV-cache quantization) have been evaluated inconsistently across tasks, and the right framing is context-aware decisions across downstream tasks, perplexity, throughput, and compression rather than one universal rule (Petrov et al., 2026).

What this means for an architecture document, not a benchmark report:

Format family	Where it shines	Where it breaks
FP16 / BF16	Maximum quality preservation, fine-tuning baselines, evaluation runs	Memory-bound on consumer hardware, slowest per-token throughput
Q8 (8-bit)	Near-FP16 quality on most tasks, 2× memory savings vs FP16	Diminishing returns on memory; slower than aggressive quantization on bandwidth-limited hardware
Q5 / Q5_K_M	Good middle ground for capable hardware that's still memory-constrained	Quality degradation on tool-call schema adherence and long-context retrieval becomes visible on harder tasks
Q4 / Q4_K_M	Most popular consumer-hardware choice; runs 70B-class models on a single 24GB card	Quality cliffs on reasoning-heavy, tool-heavy, or long-context workloads — varies sharply by base model
Sub-4-bit (Q3, Q2, 1.58-bit)	Memory pressure relief on extreme deployments	Quality is unreliable across task families; do not deploy without task-specific eval

Three claims to make explicit because they are routinely violated:

Quantization quality is task-conditional, not model-conditional. A model at Q4 may pass a perplexity benchmark, write working Python, and still fail tool-call JSON schema adherence on a fraction of attempts that didn't fail at Q8. Code-heavy and tool-use-heavy workloads are unusually sensitive to small quantization perturbations because a single token error in a structured output can fail the whole call.

KV-cache quantization is a separate axis. A model at FP16 weights with int8 KV cache, used at 32K context, is a different system than the same model at Q8 weights and FP16 KV cache. Long-context performance depends as much on KV-cache precision as on weight precision, and most published quantization evals do not separate the two.

Format-family choice matters as much as bit width. GGUF k-quants, AWQ, GPTQ, EXL2, and NF4 do not produce identical quality at the same nominal bit width; their calibration data, group sizes, and per-channel/per-tensor handling differ. "Q4" is shorthand for at least four mutually incompatible formats.

A wiki article that hands the reader a clean Q4/Q5/Q8/FP16 table without these caveats has lied by simplification. The honest version is the table above with a sentence underneath: evaluate on your task before trusting any of these.

5. Agent-harness integration: a careful re-statement

The most common error in local-first writing is asserting that named agent harnesses have a "local mode" or "offline mode" they do not have, or have under different names. This section corrects the canon.

5.1 Claude Code

Claude Code is an agent harness that executes locally — file access, shell, tool calls, repo state — but invokes a hosted Anthropic model by default. Anthropic's data-usage documentation confirms that prompts and outputs traverse the network and that session caching happens locally. The quick-reference enterprise-deployment docs describe deployment via Anthropic APIs, Bedrock, Vertex, proxies, and LLM gateways — not a first-class local-inference mode.

What does work, and is documented:

• vLLM as a Claude Code backend. vLLM's docs show pointing Claude Code at a vLLM server by setting ANTHROPIC_BASE_URL to the local server, replacing Anthropic's backend if the served open-weight model supports tool calling (vLLM Claude Code integration).
• Ollama as a Claude Code backend. Ollama documents an Anthropic-compatible API surface that Claude Code can be pointed at, with the same caveat about tool-call compatibility (Ollama Claude Code integration).
• LLM gateways and proxies. Anthropic's enterprise docs sanction routing through third-party gateways, which can in principle route to local backends; this is an integration responsibility, not a built-in feature (Anthropic enterprise deployment).

The correct phrasing is therefore: Claude Code can be configured to use a local or private backend through compatible endpoints, but it does not ship with a first-class "local mode." Tool-call semantics, streaming behavior, and system-prompt handling are model-dependent; HTTP-shape compatibility is not the same as behavior compatibility. A local backend that passes Anthropic-shaped requests can still fail the agentic workload silently if the model's tool-call formatting or refusal behavior diverges.

5.2 Codex CLI

Codex CLI's local story is better-grounded but also commonly mislabeled. The relevant flag is --oss, with --local-provider selecting between ollama and lmstudio. OpenAI's own writeup describes Codex CLI with --oss using local Ollama or LM Studio Responses-compatible endpoints (OpenAI Codex loop), and both Ollama and LM Studio document the integration path explicitly (Ollama Codex integration, LM Studio Codex integration). vLLM also documents Codex integration via an OpenAI Responses-compatible local server with ~/.codex/config.toml provider settings (vLLM Codex integration).

There is no "Codex CLI offline mode" by that exact name. The correct phrasing is Codex CLI's --oss open-source-provider mode, which can be pointed at a local Responses-compatible server (Ollama, LM Studio, or vLLM). "Offline" overpromises: in practice the workflow still depends on package registries, model downloads, and often documentation lookups that are not local.

5.3 What actually works at the harness boundary

The integration matrix that the docs support, as of this writing:

Harness	Backend transport	Supported local backends (per docs)	Tool-call caveat
Claude Code	ANTHROPIC_BASE_URL override	vLLM (vLLM), Ollama (Ollama), via LLM gateway/proxy (Anthropic)	Open-weight model must support Anthropic tool-call schema; behavior diverges by model
Codex CLI	--oss + --local-provider	Ollama (Ollama), LM Studio (LM Studio), vLLM (vLLM)	Open-weight model must support OpenAI Responses-shaped tool calling; behavior diverges by model

The structural point: the harnesses themselves do not run inference locally. They are agent loops that can be redirected at a local serving stack. The local-first deployment work is all in the serving stack, not in the harness.

6. When local actually wins

The architectural slots where local inference is the right default, not a curiosity:

6.1 Boundary-constrained workloads

Data that contractually, regulatorily, or contractually-by-customer cannot leave a controlled boundary. Examples: clinical PHI in a non-BAA jurisdiction, classified workloads, customer-deployed embedded products that ship with the model, internal corpora under data-sovereignty rules. The cloud alternative isn't worse here; it isn't an alternative.

The threat model has to be specified. Cloud providers now offer commercial data controls that materially change the calculus — OpenAI documents that API data is not used for training unless the customer opts in, and offers ZDR-like arrangements with documented exclusions (OpenAI data controls); Anthropic documents API retention and ZDR eligibility (Anthropic API retention, Claude Code ZDR). For some boundary-constrained workloads, contractual controls are sufficient and local is overkill. For others — true air-gap, unaudited regions, customer-shipped products — they are not. The decision is workload-specific, not ideological.

6.2 Latency-sensitive narrow workloads

The clearest local-wins-on-latency case is small-to-medium models doing well-bounded work: autocomplete, prompt routing, classifier-as-router, embeddings, reranking, guardrail checks, structured extraction, single-turn code completion. Network round-trip matters most when the inference itself is fast, so a 30ms decode on a local 7B model genuinely beats a 200ms hosted call when the call rate is high.

The case weakens sharply for long-context agentic workloads. FlexGen's strong result for OPT-175B on a single 16GB GPU is roughly 1 token/sec under latency-insensitive batched conditions (Sheng et al., 2023) — a benchmark for throughput-under-constraint, not interactive agent workflows. Long prefills on consumer hardware can erase any network advantage. Latency wins are concentrated in the high-rate-narrow-task corner.

6.3 Offline-after-setup workflows

Field deployments, on-aircraft tooling, secure facilities, customer demos in unreliable connectivity. The rigor here is distinguishing offline-after-setup (the system fetched its dependencies, models, and tools while online; runs offline thereafter) from air-gapped from first boot (the system was never online; everything was sideloaded). The first is common and well-supported. The second is genuinely hard: package mirrors, model registries, dependency caches, base images, signed updates, and license-server traffic all need air-gap-aware substitutes. The article that promises "offline mode" for an agent harness without specifying which of these meanings it means has overpromised.

6.4 Predictable high-utilization workloads

Local hardware amortizes well when utilization is high and stable. Embedding services serving a fixed corpus; classifier farms over predictable input streams; reranking pipelines feeding a known retrieval index; batch processing with night/weekend schedules. The breakeven against API pricing is a function of utilization, not raw token cost.

A serviceable rule of thumb: a 24GB consumer GPU running a Q4 7–13B model 24/7 at 70%+ utilization typically beats hosted-API token cost on the same tasks within 6–12 months, if the tasks are within the open-weight model's competence and the labor cost is correctly accounted for. The qualifications matter more than the number.

6.5 Vendor-independence and reproducibility requirements

When the cost of vendor risk is itself the dominant constraint — model deprecation, opaque regressions, regional availability, rate-limit changes, price changes, or research workflows requiring exact reproducibility years later — local weights are not just a performance choice but a control choice. This is the strongest argument for local that has nothing to do with privacy, latency, or cost.

6.6 Fine-tuning and adaptation loops

Parameter-efficient adaptation (LoRA, QLoRA, adapters) is genuinely local-first. The training loop, the evaluation harness, the adapter artifacts, and the deployed weights all stay on hardware the team controls. The cloud alternative — managed fine-tuning on a hosted model — works but cedes control of base-model versioning and adapter portability. For research-heavy or adaptation-heavy workflows, local is the natural slot.

The caveat: data quality, evaluation gates, license audits, and adapter-deployment paths often dominate the actual work. A team that fine-tunes before they have evals, RAG, or prompt baselines is usually solving the wrong problem.

7. When cloud actually wins

The symmetric statement.

7.1 Frontier capability access

If the workload requires the strongest available reasoning, longest contexts, newest multimodal capabilities, or most-recent training data, the frontier hosted models are still ahead of the best open-weight equivalents on most agentic tasks. This gap is non-uniform across task classes, narrowing every quarter, and irrelevant for narrow workloads — but for open-ended agentic engineering, it remains the dominant fact. The benchmarks cited in §3 are the relevant evidence.

7.2 Burst elasticity

Workloads with peak-to-trough ratios above ~5×, irregular traffic, demo loads, or unpredictable success-conditional retries amortize poorly on owned hardware. The cloud's price for elasticity is paying retail per-token; the local price for elasticity is owning the GPUs at peak and idling them at trough. For most non-steady-state workloads, the cloud number wins.

7.3 Operational simplicity

Local inference at production quality requires CUDA/ROCm/Metal driver competence, Python environment hygiene, model-format compatibility checks, KV-cache sizing, batch tuning, monitoring, log retention, security patching, model-supply-chain provenance, and ongoing eval regression testing. vLLM's GPU install documentation alone exposes Linux, Python 3.10–3.13, CUDA/ROCm/XPU paths, compute-capability constraints, and warnings about binary incompatibility across CUDA/PyTorch builds. None of this is incidental to whether local "works"; it is central to whether local works reliably.

For organizations whose AI workloads are a tiny fraction of their engineering surface, the labor cost of running this stack often exceeds the API token cost. Cloud wins on ops simplicity not because cloud is technically simpler, but because the ops cost is delegated to the provider.

7.4 Lightly-utilized and exploratory workloads

The mirror of §6.4. If utilization is below ~30% and task patterns are still being discovered, owning hardware is buying capacity you won't use. Cloud per-token pricing is the cheaper bet until utilization stabilizes.

8. The hybrid pattern is the most common production answer

Most production AI systems in 2026 are not fully local or fully cloud — they are hybrid, with the boundary chosen at the task class level rather than the system level.

A representative pattern from coding-agent deployments: the agent harness runs locally; embeddings and reranking are local (high-rate, narrow, latency-sensitive); the main reasoning model is a hosted frontier API (capability-dominant); a local guardrail/classifier model gates a subset of cloud calls (privacy and rate-limit pressure relief); fine-tuned adapters serve a domain-specific narrow task locally (utilization-dominant). Each task class is placed where it wins on its own dominant constraint.

The hybrid pattern's risk is that it silently re-introduces every cloud assumption it was trying to escape. A "local-first" deployment that escalates to a cloud frontier model whenever the local one fails has cloud-grade privacy properties, cloud-grade availability dependencies, and local-grade ops cost. The honest version of hybrid declares the escalation paths and the data flows at each escalation, and re-evaluates the threat model whenever a new path is added.

A useful heuristic: the hybrid system is only as boundary-respecting as its loosest path. If even 5% of traffic escalates to a cloud frontier model, the system is not air-gapped. If embeddings call a hosted API, the corpus is not local. If logs are streamed to an observability vendor, the privacy claim is downstream of the observability vendor's policies.

9. Operational brittleness: the cost the local-first articles undercount

This is the section the marketing voice elides. Local inference is brittle in ways that compound.

Hardware variance. Consumer GPUs (RTX 4090, RTX 5090), datacenter GPUs (H100, H200, MI300X), Apple Silicon (M2/M3/M4 Ultra), and CPU-only paths each have different memory bandwidths, supported precisions, KV-cache layouts, and quantization-format compatibility. A configuration that works on a developer's M3 Max under MLX may fail on a deployment H100 under vLLM, not because either is broken, but because the runtime path is different. Reproducibility across hardware requires a containerized, pinned, tested deployment story — the kind of work most teams underestimate by 5×.

Driver and toolchain drift. CUDA/PyTorch/Triton/vLLM version compatibility is real, version-pinned, and easy to break. The vLLM install docs warning about binary incompatibility is not pessimism; it is operational reality. ROCm drift on AMD has been particularly costly historically, though improving. Metal/MLX drift on Apple Silicon is the smoothest of the three but still meaningful across major macOS releases.

Model supply-chain risk. Open weights are downloaded from registries (Hugging Face, model mirrors, organizational artifact stores). Verifying provenance, licensing, and tampering for production deployments is real work — and the trust model is different from a hosted API where the provider attests to the model identity. The local team is the integrator and the auditor.

Dependency drift. Tool docs, integration paths, and CLI flags change on a quarterly-or-faster cadence in the agent-harness space. Articles like this one will be partially wrong within months. A team's deployment notes need versioned citations, pinned model SHAs, and a maintenance budget. Without those, "local-first" silently becomes "local-was-once-first-and-now-quietly-broken."

Evaluation regression. Cloud models change behavior under your feet — but at least the changes are advertised (mostly) and the model identifier shifts. Local models change behavior when you change quantization format, KV-cache precision, batch size, prompt formatting, or harness version. Without a continuous eval harness running against the deployed configuration, regressions are silent. Most teams do not have this. The teams that do are usually the teams for whom local is the right answer in the first place.

Security blast radius. A local agent that can read secrets, execute commands, and call MCP tools has a meaningfully larger blast radius than a hosted model that can only respond with text. The privacy-on-paper of local inference can be undone by the local agent's logs, transcripts, MCP traffic, downloaded artifacts, package-manager traces, and shell history. The threat model has to be the system, not just the model.

The compounding effect is the part the local-first advocate's articles understate: each of these brittleness sources is manageable on its own, but they combine. A team running vLLM on H100s with three open-weight models, four agent harnesses, two quantization formats, and a hybrid escalation path is doing real engineering. A team running one hosted frontier API has delegated all of it to the provider.

This is not an argument against local. It is an argument that local has an ops cost line item that the breakeven calculation must include, and frequently doesn't.

10. Reading benchmarks honestly

A short sub-section because the article cannot stand without it.

Benchmarks comparing local-versus-cloud, or quantization-versus-baseline, or runtime-A-versus-runtime-B, are usually non-comparable for at least one of these reasons:

• Different hardware, with different memory bandwidth and compute capability, dominates the result more than the variable being studied.
• Different model versions or weights are compared as if they were the same model — particularly common across open-weight quantization formats.
• Different context lengths or batch sizes produce different prefill/decode tradeoffs and shift the apparent winner.
• Different prompt mixes and workloads — a benchmark on perplexity is not a benchmark on tool-call adherence is not a benchmark on long-context retrieval.
• Different serving stacks — the same model under llama.cpp, vLLM, and a vendor API will produce different throughput, latency, and quality numbers because the stacks make different scheduling and KV-cache decisions.
• Vendor benchmarks — vendor-published numbers select favorable workloads. This is not necessarily dishonest, but it is selection.

The compact rule: a local-versus-cloud benchmark you didn't run on your workload is directional at best, misleading at worst. The only benchmark that decides anything is the one against the actual task with the actual harness on the actual hardware. The bake-off described in §11 is the smallest version of that.

11. A decision checklist

A pragmatic procedure, applied in order, with cheap stopping conditions before the expensive ones.

Step 1: Capability gate. Can the open-weight model class you would deploy plausibly do this task at acceptable quality? If the workload is open-ended agentic engineering, multi-turn tool-use against complex codebases, or anything requiring frontier-tier reasoning, the answer is usually "not yet." Stop here and use a hosted frontier model unless one of the constraints below forces local.

Step 2: Boundary gate. Does the data have a real boundary that prohibits hosted inference? Specify the threat model: contractual, regulatory, or customer-shipped product. If yes, local is required — proceed to step 4. If no, but the team prefers stronger contractual controls, the cloud commercial-data-controls path may be sufficient.

Step 3: Workload-shape gate. Is the workload narrow, high-rate, latency-sensitive, or predictably high-utilization? If yes, local is plausible. If the workload is bursty, lightly-utilized, or exploratory, cloud is the cheaper bet — stop.

Step 4: Bake-off. Before deploying local, run the smallest decisive experiment. Pick two cloud baselines (one frontier, one cheaper hosted) and four local variants (the same open-weight model at Q4, Q5, Q8, and FP16/BF16 if hardware allows). Use the same harness and the same 20–30 representative tasks: latency-sensitive completions, private-corpus Q&A, small repo edits with tests. Measure p50/p95 time-to-first-token, tokens/sec, wall-clock task success, tool-call failure rate, blind preference, test pass rate, VRAM/RAM use, setup time, and amortized cost per successful task. The last metric is the one the prompt-level comparison usually misses.

Step 5: Ops budget. If the bake-off favors local, allocate the ops budget honestly: a containerized deployment story, a continuous eval harness, a model-supply-chain audit, a versioned-citation deployment doc, and a maintenance owner. If the team cannot fund this, the bake-off result is misleading — the local deployment will degrade silently. Re-run the cost comparison with the ops budget included; the breakeven shifts.

Step 6: Hybrid where it pays. Accept that most production answers are hybrid. Place each task class at the boundary where it wins on its dominant constraint. Document the escalation paths and re-evaluate the threat model whenever a new path is added. The system's privacy properties are those of its loosest path.

Step 7: Re-evaluate quarterly. The capability gap narrows, the toolchain shifts, model licenses change, and prices move. A decision made in Q1 may not survive Q4. Pin the dates of evidence underlying the decision; expire them deliberately.

The procedure is unglamorous on purpose. The romantic version of local-first AI deserves the brittleness it inherits. The boring version — capability gate, boundary gate, workload shape, bake-off, ops budget, hybrid pragmatism, scheduled re-evaluation — is the version that ships.

12. The honest summary

Three load-bearing claims, defended above.

Local inference does not automatically win on privacy. It wins on privacy when the boundary is real, the threat model is specified, the local artifacts (logs, caches, transcripts, MCP traffic) are themselves controlled, and the cloud alternative's commercial-data-controls would not have been sufficient. The crude "cloud sees data, local doesn't" framing is wrong often enough to be misleading.

Local inference does not automatically win on latency or cost. It wins on latency for narrow, high-rate, small-model workloads where network round-trip dominates. It wins on cost for predictable, high-utilization workloads where the hardware amortizes — once labor, ops, depreciation, and reliability costs are honestly included.

Local inference does not automatically win on capability. For open-ended agentic engineering, hosted frontier models remain the default, and the gap is dominated by the model itself, not the deployment surface. The gap narrows for narrower tasks and is irrelevant for the narrowest ones.

The defensible architectural posture is therefore: cloud frontier models are the default; local inference is a measured optimization where a specific constraint forces it or a specific evaluation proves it sufficient. Articles that invert this default — that present cloud as the exception requiring justification — are selling something. The wiki's job is to map the territory, not to root for one of the runtimes.

Companion entries

Core theory:

• Frontier Capability Gap
• Quantization Quality Surfaces
• Context Window Economics
• Privacy Threat Modeling for AI Systems

Practice:

• Multi-Model Agent Orchestration Patterns
• Agent Harness Architecture
• Hybrid Inference Routing
• GPU Capacity Planning for AI Workloads
• Air-Gapped AI Deployments
• Continuous Eval Harnesses for Agent Systems
• Model Supply-Chain Audit

Tools and runtimes:

• llama.cpp Architecture
• Ollama as a Local Model Manager
• MLX on Apple Silicon
• vLLM and PagedAttention
• Claude Code Backend Override Patterns
• [[Codex CLI --oss Mode]]

Counterarguments:

• The Vendor Lock-In Trap
• Benchmark Misreading in AI Engineering
• The Ops Cost Line Item
• Why Open-Weight Models Catch Up Fastest on Narrow Tasks