Local LLM Routing — M1Max

The Agent Host VPS (Hostinger KVM2) cannot run local models — no GPU, insufficient RAM for any useful GGUF. The local tier therefore runs on François’s M1Max (llama.cpp serving a GGUF), reached from the VPS over Tailscale as an OpenAI-compatible endpoint.

The Decathlon M3Pro is ruled out as an inference host — corporate MDM would block Tailscale on the managed laptop. The M1Max carries the local LLM; because it is also François’s dev machine, served inference shares its resources with interactive work (acceptable — routing there is intermittent and explicit).

Reality check — an experimentation tier, not a privacy tier

OpenRouter zero-retention routing (openrouter-zdr) is the accepted default for all inference, including sensitive content. The local tier is 🔵 for-fun — sovereignty, craft, learning — and is used only when François explicitly says “use local for this.”

Why not use the local tier for sensitive content?

Why not…	Answer
…“local is more private than ZDR”?	True, but moot — ZDR already eliminates the dominant risk (training + retention). The residual risk (a compliant provider leaking despite its policy) is speculative, and a 3–9B local model does not fix it
…use it for sensitive work?	Sensitive docs (tax, medical, legal, financial) are exactly where model quality matters most. A light local model is more likely to make a costly mistake than a frontier model
…expect it to save money?	The LLM bill is ~€1/month. Electricity + llama.cpp upkeep costs more than the API it routes around
…rely on “data stays local when it matters”?	The fallback (M1Max asleep → OpenRouter) sends sensitive data external anyway. Either it is sensitive enough to wait (then no fallback), or it is fine for ZDR (then no tier). Can’t have both

The honest framing: Tailscale + llama.cpp is more private than any API (bytes physically never leave your hardware), but that extra privacy is not worth deploying for — the ZDR config already did the job. Running your own inference is worth doing because it is satisfying and instructive, not because it is necessary.

Model choice — keep it light, honesty over theory

The local tier is model-agnostic — any GGUF works. But the honest constraint is the M1Max’s dual role: while it serves, it is also François’s dev machine.

Tested reality: Qwen 27B MoE runs on the M1Max, but it completely slows the Mac down — interactive work becomes painful. A heavy model on an inference machine that also codes is not realistic; it’s a demo.

The realistic rule: pick a light model (3–9B class) so the Mac stays usable while serving.

Model	Size	Fit for an M1Max that also codes
Qwen 3B / 4B / 7B	3–7B	✅ Recommended — fast enough for tool-calling, low memory footprint, dev stays fluid
Ornith-1.5 9B Dense	~9B	✅ Good pick — agentic-coding focus, still light
Qwen 27B MoE	~27B (MoE)	⚠️ Tested — runs, but saturates the Mac; only for when the machine isn’t being used
Anyone else	—	If it has a GGUF and fits with headroom to spare, it runs

The SLO: while serving, François must still be able to code comfortably. Anything that breaks that (27B MoE, long generations on 14B+) fails the requirement — not because it can’t run, but because it isn’t realistic day-to-day.

Where things run

Machine	Role
M1Max (home, primary dev)	Inference host — llama.cpp + GGUF — and interaction machine (SSH → hermes, Slack, dashboard)
VPS (Agent Host)	Hermes itself — routing, memory, tools, cron
M3Pro	Ruled out as host — Decathlon MDM would block Tailscale

Why this design

Constraint	Consequence
VPS is a KVM2 without GPU	Ollama/llama.cpp on the Agent Host is physically off the table
M3Pro is a managed Decathlon laptop	Tailscale would be blocked by MDM → not a viable host
Machines are far apart (datacenter ↔ home)	A tunnel is required — Tailscale (WireGuard) is the zero-config choice
Inference on your own hardware is worth doing for its own sake	The tier exists as 🔵 experimentation — sovereignty and learning, not a security requirement

Synthetic recap — Local vs OpenRouter

	Local — M1Max (experimentation, explicit opt-in)	OpenRouter ZDR (live default for everything)
Inference host	llama.cpp + GGUF on the M1Max	Cloud API restricted to zero-retention providers
Privacy	Data stays on François’s hardware — never leaves the tailnet	Zero retention, no training (accepted risk — see openrouter-zdr)
Quality	Light 3–9B class (mistakes more likely on hard content)	Frontier-grade
Cost	€0 (compute only — electricity)	Pay-per-token (budget cap €15/mo, actual spend ~€1/mo)
Speed	~10–40 tok/s (light model, e.g. Qwen 3–7B / Ornith 9B)	Fast, datacenter-grade
Availability	⚠️ Only when the M1Max is on and awake	24/7
Fallback	OpenRouter if the M1Max is dark	—
Best for	Explicit “use local” requests; experimentation, model tinkering, sovereignty	Everything: lookups, reminders, drafting, coding, sensitive content (after the SOUL warning ask)

Step-by-step setup

1. Tailscale on both machines

# On the VPS (Agent Host)
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# On the M1Max
brew install tailscale
# then open the Tailscale app / run tailscale up from the menu bar

Both machines must join the same tailnet. Verify:

tailscale status          # both appear; note the M1Max tailnet IP (100.x.y.z)

2. Serve the model on the M1Max

# Install llama.cpp (once)
brew install llama.cpp

# Download a GGUF you want to run — keep it light (3–9B class). Example: Qwen 7B, or an Ornith-1.5 9B
huggingface-cli download Qwen/Qwen2.5-7B-Instruct-GGUF qwen2.5-7b-instruct-q4_k_m.gguf --local-dir ~/models
# or: huggingface-cli download ornith-ai/ornith-15-9b --local-dir ~/models/ornith
# (avoid 27B MoE class on a machine that must stay usable — tested: it slows the Mac down completely)
# (swap the file any day — the routing below never changes, only the path/alias)

# Serve — bind to the tailnet IP, port 8080
llama-server -m ~/models/<your-model>.gguf \
  --host 100.x.y.z --port 8080 --alias local

Bind to the tailnet IP (--host 100.x.y.z), not 0.0.0.0 — the endpoint must be reachable only from the VPS, never the public internet.

3. Verify from the VPS

curl http://100.x.y.z:8080/v1/models
# → {"object":"list","data":[{"id":"local",...}]}

An OpenAI-compatible /v1 API is what Hermes speaks natively — no proxy needed.

4. Route Hermes to the local endpoint

hermes config set model.provider custom
hermes config set model.base_url http://100.x.y.z:8080/v1
hermes config set model.api_key local            # llama.cpp ignores the key; a non-empty value keeps Hermes happy
hermes gateway restart                            # gateway reads config once at startup

Or define it as a named provider in config.yaml if we later want it switchable via fallback_providers (the OpenRouter fallback chain stays in place — local is a tier, not a replacement).

5. Confirm the route

hermes chat -q "Reply with exactly: local OK"    # watch tokens appear in the llama-server log on the M1Max

Pitfalls

• Heavy models kill the dev machine — Qwen 27B MoE runs on the M1Max but completely slows it down (tested). The serving machine is also the coding machine: stick to the 3–9B class unless you explicitly accept the slowdown for a one-off task.
• M1Max asleep or away → endpoint unreachable → Hermes must fall back to OpenRouter. Configure fallback_providers so a dark M1Max never blocks a request.
• Memory budgeting — a 3–9B Q4 is only ~3–6GB, leaving plenty of headroom for dev. Verify free unified memory before picking a bigger GGUF (e.g. 14B+ starts eating into usable RAM).
• Model id mismatch — the endpoint ID in /v1/models (here local) must match model.default or the alias used in the route.
• First token latency (~1s on MoE) is fine for chat; heavy agent loops will feel slower than OpenRouter — keep local for explicit “use local” requests and tinkering, not for throughput.
• Don’t let the tier become a privacy story again — ZDR routing is the accepted default. If the privacy rationale starts creeping back into decisions, that is the sign to stop and re-read the reality check above.

Status

• VPS Ollama/llama.cpp LOCAL path rejected (KVM2, no GPU/RAM)
• M3Pro ruled out (Decathlon MDM blocks Tailscale) — M1Max is the host
• Model candidate: keep it light (3–9B class) — Qwen 7B / Ornith-1.5 9B. Qwen 27B MoE tested but rejected as default (slows the M1Max down completely)
• Tier is 🔵 experimentation / sovereignty — not a privacy tier. OpenRouter ZDR is the accepted default for all inference; local only on explicit “use local for this”
• Tailscale installed on VPS + M1Max (worth doing regardless — estate connectivity beyond the LLM)
• llama.cpp + GGUF served, verified via curl /v1/models
• Hermes routed to local endpoint, fallback tested — only when François wants to play with it