$ ninfer --model qwen3.8-27b-nvfp4 --gpu all
[ok] 32 GB VRAM · 64/64 layers on GPU
[ok] context 262144 · KV q8_0 + DDR5 overflow
[ok] 148 tok/s avg · 203 peak (MTP n-max 4)
▮ last report: 100+ pages · every citation checked
I Built a Sovereign AI Stack
A fully local, fully sovereign AI stack — a 27B-parameter model at 148 tokens per second, running on hardware I own. No cloud, no data leaving the machine.
This year I built a local AI stack that does real professional work.
The Machine
Seven Thousand Euros, Fully Utilized.
The workstation I invested in this year — every euro pointed at the two things that matter for local inference: VRAM and CPU bandwidth.
GPU — ZOTAC OC RTX 5090
32 GB of GDDR7 at 1,792 GB/s — enough bandwidth to keep a 27B model fed. The Qwen3.8-27B in NVFP4 (~14 GB) sits fully on the card.
CPU — AMD Ryzen 9 9950X3D
16 cores / 32 threads, up to 5.7 GHz. Batching, tokenization, and host-side tooling run at full speed.
Memory & Storage
64 GB DDR5-6000, 10 TB of NVMe (2 TB PCIe 4.0 + 8 TB PCIe 5.0) — fast DDR5 right next to the GPU takes the spill when a session outgrows the card, while models, datasets, and generated reports sit on fast local media.
Chassis & Power
Lian Li Lancool 216 with a be quiet! Pure Power 13 1000 W unit — a 32 GB card pulling full load stays cool and quiet.
Sovereign AI Local by Design
Sovereign AI means the whole stack is yours: the model, the inference engine, the data. None of it is off-the-shelf — the quantization, the context budget, the DDR5 overflow were all design decisions. That's the architect's part: making one machine do frontier-class work.
Core Stack:
Where Everything Fits
32 GB of VRAM on the card, 64 GB of fast DDR5 right next to it — the whole stack lives in the room.
VRAM — RTX 5090 (32 GB)
~14 GB: Qwen3.8-27B, all 64 layers on the card — the 256K q8_0 KV cache (~9 GB) sits right beside it.
DDR5 — 64 GB, next door
Spill and long-session state — the 256K KV cache and the model live on the card; nothing leaves the machine.
What it runs
Multi-part research reports, long-context analysis over logs and specs, and subagent code chains — at 148 tokens per second, weights and 256K KV on the card, spill next door.
Sovereignty & Hardware
Models, prompts, and documents never leave the machine — the same isolation an air-gapped Kubernetes cluster gives you, applied to the AI stack itself. And it's fast: 148 tokens per second on the card, with a 256K context that spills into DDR5 when it outgrows the 32 GB.
Context Engineering
Context is engineered, not left to chance: compaction with a 60K output reserve and a 30K recent-token window, tool outputs over 50 KB pruned to structured summaries, and think-in-code so a 700 KB log enters the context as a 3 KB summary.
Sequential Subagent Chains
An orchestrator that never does work itself — it decomposes the job and delegates to specialist subagents (planner, researcher, implementer, reviewer, scout), each starting with a fresh context and receiving only a structured handoff. Three modes: chains, batches of independent sections, and single lookups.
Verified Output
Generated documents aren't taken on faith: the report pipeline cross-checks every claim against sources before a page is ever written down.
"If the model fits on the card, the whole system fits in the room — sovereignty isn't a checkbox, it's the architecture."
Context Is a Budget 256K Is the Ceiling — 100K Is the Craft
A 256K-token window sounds like the problem is already solved. It isn't. Chroma Research tested eighteen frontier models and found every single one degrades as input grows — a 200,000-token window is already losing quality at 50,000. Liu et al. showed accuracy dropping 30% or more when key information moves from the edges of the context to its middle, and agents routinely fail to fetch the context they could retrieve on demand (Gao, Vercel, 2026). 2026 RULER-based measurements put the usable share of a headline context window at roughly half. The model is not running out of space — it is drowning in irrelevant context.
So on 32 GB of VRAM the window is managed the way memory is managed: quantized, compacted, pruned at the source, overflowed into the fast DDR5 right next door, and backed by memory that lives outside the window entirely. Every token in the context has to earn its place — that is the whole discipline of running a frontier workload on one consumer card.
Why Bigger Isn't Better:
18 / 18
frontier models degrade as input grows — even on simple tasks (Chroma Research, July 2025).
−30%+
accuracy when key information moves from the edges of the context to its middle (Liu et al., TACL 2024).
50–65%
of a model's advertised context window is what's effectively usable — 2026 RULER-based measurements (arXiv 2509.21361).
And on the hardware side the math is measured, not assumed: at q8_0 this hybrid-attention model holds only ~35 KB of KV per token — 16 of its 64 layers keep a KV cache at all — so the full 256K cache is ~9 GB and the whole 30 GiB stack fits the card with ~2 GB to spare. Naive FP16 math would have said "tens of gigabytes" and sent you to rent a datacenter.
The Five Levers:
8-Bit KV Cache
The KV cache is what remembering costs. It runs at q8_0 instead of FP16 — roughly half the memory for negligible quality impact — held in one contiguous region so it never fragments, with tiled flash attention minimizing bandwidth pressure at long context. That one decision is the difference between "256K fits" and "nothing fits."
DDR5 Overflow
When a session runs past the card's 32 GB, a 16 GB slice of the 64 GB DDR5 pool next door takes the spill. The 256K KV cache itself is only ~9 GB at q8_0 and stays on the card — DDR5 covers the working set and long sessions. A long session trades a little bandwidth for availability: it slows down instead of dying. Latency is the price of not restarting.
Structured Compaction
When the window fills, older history compresses into a compact guide — not a prose narrative, but a curated record of decisions, errors, plans, prompts, and outcomes. A 60K-token reserve keeps room for the actual work, the most recent 30K stays verbatim because recent tokens get the most attention, and KV checkpoints every 8,192 tokens let the system rewind without re-prefilling.
Pruning at the Source
Compaction fixes history that already accumulated; pruning stops noise before it arrives. When a tool returns more than ~2,000 lines or 50 KB, the output is truncated and summarized by content type — JSON by keys and counts, logs by error rates and patterns, listings by directory structure. Pruning is not about removing information — it's about removing noise.
Think-in-Code
The strongest move is not compressing context — it is never generating it. Data processing runs in a sandbox: the bytes your code touches never enter conversation memory, only what it explicitly prints. A 700 KB log becomes a 3 KB summary; a dozen documentation pages get fetched and indexed without a raw byte reaching the window. Every 3 KB return instead of a 70 KB dump buys ~20× more tool calls before the limit.
Memory Outside the Window:
The Content Index
Everything the agent fetches or processes — web pages, files, command output — is automatically indexed in a local SQLite FTS5 store with BM25 ranking, stemming, and phrase matching. A 100-page PDF is indexed once and searched as often as needed, without ever loading its full text into context.
Session Memory
Decisions made, errors hit, plans formulated, approaches rejected — all auto-captured, categorized, and indexed across the entire session. When the agent needs to remember why a design choice was made, it searches the knowledge base instead of scrolling the conversation.
For agent memory, exact matching beats vector similarity — a code symbol is a code symbol, and "useEff" has to be found inside "useEffect". FTS5 gives ranked results with snippet extraction and needs no external service, which matters in a box that runs everything itself. This is the mechanism behind long-horizon autonomy: the agent can work for hours without losing track of earlier decisions, because the decisions are indexed and retrievable.
"The solution is never a bigger window — it's aggressive context management, so every token in the context carries signal, not noise."
Long Horizons, One Machine
Sessions That Outlive the Window
A single agent handed a big job fails in a predictable way: it exhausts its context window, buries the early instructions, and degrades as the conversation grows — by page forty of a report it is writing against its own accumulated token debt. And a 170-page report is dozens of subtasks: topic research, section planning, content writing, citation management, formatting, quality review. No window I own is big enough to hold all of it at once.
The fix is architectural, not hardware: an orchestrator that does no work itself hands structured state to specialist subagents, each starting from a clean window. Context is never accumulated — it is passed. The machine holds the memory; the window only holds the current move.
Why Decomposition Wins:
90.2% better
Anthropic's 2025 multi-agent research system beat a single Claude Opus 4 agent on internal research evals — at ~15× a chat's token cost. The published win is parallel breadth, and this box can't buy it. What it keeps is the decomposition: clean windows, scoped tools, structured handoff — one agent at a time.
94% saved
from early exit: stop decoding the moment an early layer is confident (Gandhi et al., 2025). The same trick at role level — the scout and worker take the short jobs, so the 27B spends its budget on reasoning, not lookups.
10 vs 30–45 min
the same research task end-to-end: ~10 on this box vs 30–45 through a cloud API — dozens of sequential calls, each paying network and queue latency. Latency is the budget.
The Roster:
Planner
Designs architecture, schemas, and implementation plans. The bridge between investigation and execution — it turns what research found into something implementable.
Researcher
Investigates problems and compares approaches — reads documentation, benchmarks, and source code, then produces analysis the planner can build on.
Implementer
Writes code, creates files, does the mechanical building. It receives the specification and executes — it knows what to build, not how the team decided.
Reviewer
Audits results for correctness, security, and compliance. It knows what was built, not what alternatives were considered — and it has read and lint tools, not a write handle.
Scout
Quick lookups, searches, fact-checks. Narrow, fast queries — the questions that shouldn't pollute a bigger context window.
Worker
The general-purpose fallback for self-contained tasks — anything that fits in one clean window and doesn't need a specialist.
Each role's tools are scoped to the job: the implementer gets file-write and code execution, the reviewer gets file-read and lint checks, the scout gets search and fetch. Tool scoping cuts both the attack surface and the context overhead — an agent can only be distracted by what it can see.
Three Ways to Dispatch:
Chain
For dependent tasks: research → plan → implement → review. Each agent receives only its predecessor's output. The default for complex work that must proceed logically.
Batch
For independent tasks: report sections or components, dispatched as a batch instead of a queue. On one GPU they still run one at a time — the win is that they don't depend on each other, so no section waits on another.
Single
For one-off tasks: a quick fix, a documentation update, a fact check, a citation lookup. The fast path for work that doesn't benefit from decomposition.
The orchestrator picks the mode by analyzing the task's dependencies and constructing an execution graph. A real report uses all three at once: batch for independent section research, chain for the investigation–planning–writing– review cycles, single for fact-checks and citations.
What Six Months of Daily Use Says:
170+ pages
the longest report this stack has produced — verified citations and dense content across every section; the most recent was 100+ pages.
6 months
of daily production use, with multi-hour sessions that hold their quality. Frontier proprietary models still win the hardest reasoning — on most real work the output is functionally equivalent.
Electricity only
the marginal cost per session. No API fees, no data egress, no vendor lock-in — after the hardware pays for itself, every token costs a few cents of power.
The quality bar is self-imposed, not a leaderboard: for technical writing, code, and analysis the output holds up against Claude Opus and GPT-5 class in practical use — while frontier models still win the hardest reasoning. The test is whether the work stands up to scrutiny: every report went through the critic loop, and the citations are there to check.
"Context is not accumulated — it is passed. That one sentence is the whole design."
The Harness
Open-Source End to End — Necessity, Made Into Virtue
The agent layer is open source too: a minimal MIT-licensed terminal harness, extended with pieces that exist because a single 5090 left no room for the usual assumptions — no cloud fallback, no parallel workers thrashing the GPU, no API in the loop.
pi.dev — the open harness
The agent itself: a minimal, extensible terminal coding harness — ~6.4M monthly downloads in 2026 — that deliberately ships without built-in subagents. What's missing is exactly what this box needs, so we built it.
The subagent extension
Our own pi extension: a fully sequential orchestrator. Even when the model emits several subagent calls in one turn, they run one at a time — a single GPU can't afford parallel workers thrashing its KV cache. Structured handoffs, per-agent tool scoping, hard output caps.
NInfer — the 5090 engine
A custom inference stack tuned for one card: NVFP4 decode, MTP draft-head speculation (~71% acceptance), unified q8_0 KV, DDR5 overflow, and 32 context checkpoints every 8,192 tokens. 148 tok/s average, 203 peak.
Sovereign tooling
Self-hosted SearxNG metasearch across 250+ engines, local document parsing with page-level citations, session recall and persistent memory. The whole loop runs in the room — no cloud API in the middle.
"From necessity, virtue: every component here exists because a 5090 wouldn't let us do it the usual way."
The Model Qwen3.8-27B — Intelligence per Gigabyte
A dense 27B model with a hybrid linear-attention core — multimodal, thinking by default, 256K native context. In NVFP4 it weighs ~14 GB, so the entire model sits on one 32 GB card, and at 148 tokens per second (203 peak with MTP speculative decoding) it out-streams the cloud APIs it replaces; the Feb 2026 flagship Opus 4.6 streams around 39 (BenchLM).
The point is compression, not scale. 4-bit quantization costs under 2% of quality, an 8-bit KV cache halves the context footprint, and a compound system — orchestration, verification, external memory — wraps an already capable model in a toolset that maximises intelligence per GB. The hardware does the thinking; the system decides what it's worth thinking about.
On the card:
82.4%
MMLU, measured on this machine (NVFP4, 14K questions) — within 0.4 pt of Opus 4.6's MMLU-Pro 82. The knowledge gap is closed; the system decides what it does with it.
40–200×
per-token cost advantage of this box over a frontier cloud API (Claude Opus 4.6: $5 / $25 per 1M tokens) — the marginal cost of a session is electricity.
3–5×
faster than typical cloud API streaming (30–60 tok/s) — a 10K token section takes a minute here, minutes there.
Scoreboard — Qwen3.8-27B
Selected benchmarks vs the previous 27B generation (Qwen3.6-27B) and a frontier cloud model — Claude Opus 4.6 Max, Anthropic, Feb 2026. Vendor-reported (Qwen); Opus figures via the Qwen model card.
Gradient — Qwen3.8-27B · Accent — Claude Opus 4.6 Max · Grey — Qwen3.6-27B
SWE-bench Pro — 61.7 / 53.4 / 53.5
LiveCodeBench v6 — 90.3 / 88.8 / 83.9
GPQA Diamond — 89.2 / 91.3 / 87.8
Terminal Bench 2.1 — 73.0 / 78.2 / 63.4
IFBench — 79.5 / 62.5 / 69.1
OSWorld-Verified — 84.3 / 72.7 / 63.9
Why it matters
A 27B model that leads the previous generation across coding, agent, and reasoning benchmarks — running at 148 tokens per second on one 32 GB card, where a typical cloud API streams 30–60. The intelligence is compressed into the machine; the system around it is what multiplies it. Measured against the Feb 2026 flagship Claude Opus 4.6 Max — $5 / $25 per million tokens — the 27B held the line on four of these six suites. The cloud frontier has since moved on (Opus 5, Jul 2026; Fable 5 at ~80 on SWE-bench Pro) — the scoreboard keeps the Feb 2026 comparison because that is the workload this box was built to replace — and its entire marginal cost remains electricity.
Sources: Qwen3.8-27B model card · Anthropic system card · Artificial Analysis.
"Small models don't catch up by adding parameters — they catch up by adding a system: the right toolset, the right memory, the right verification."
Reports That Verify Themselves A 100-Page Pipeline, End to End
The system that produced my own reports, fully locally: a phased research workflow — scope, broad search, deep search, cross-referencing, synthesis — with a critic subagent reviewing every section for accuracy, completeness, and citation quality before the writer finalizes it.
Sovereign Research
Research runs on a self-hosted metasearch across 250+ engines — no cloud API, no account, and every fetched document is parsed on the machine.
Exact Citations
The document parser returns page and position hits, so every claim in a report points at a real page — not a URL I hope is right.
Critic Loop
A reviewer subagent rejects sections that fail accuracy or citation quality; the writer revises until they pass. No section ships on faith.
Markdown to PDF
Output lands as typed, versioned Markdown that exports to clean PDFs — deliverables, not drafts.
The result: 100+ page reports with verified citations, generated end to end on this machine. Self-hosted inference costs only the electricity.
Proof, Not Promises Because the System Ships
A machine is only as interesting as what it produces. This one produces complete, verified, multi-part deliverables — researched, written, cross-checked, cited — end to end, on one machine.
That is the exercise: value generated from scratch, the way the steam engine replaced the horse in the fields. This very page was written and deployed by the stack it describes — researched, drafted, and uploaded to the server by the same agent that drafted it. No cloud API was in the room; the marginal cost was a few cents of electricity.
The output is the point: it stands up to any scrutiny — every claim checked, every citation verified, ready to be shown to anyone.
Sovereign AI, Ready to Prove It
I design and build local, sovereign AI systems — the hardware, the model, and the orchestration — that do real professional work on one machine. If you want to see AI that never leaves the room it runs in, let's talk.